{"html_url": "https://github.com/simonw/datasette/issues/1699#issuecomment-1092321966", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1699", "id": 1092321966, "node_id": "IC_kwDOBm6k_c5BG4Ku", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-08T00:20:32Z", "updated_at": "2022-04-08T00:20:56Z", "author_association": "OWNER", "body": "If we do this I'm keen to have it be more than just an alternative to the existing `sqlite-utils` command - especially since if I add `sqlite-utils` as a dependency of Datasette in the future that command will be installed as part of `pip install datasette` anyway.\r\n\r\nMy best thought on how to differentiate them so far is plugins: if Datasette plugins that provide alternative outputs - like `.geojson` and `.yml` and suchlike - also work for the `datasette query` command that would make a lot of sense to me.\r\n\r\nOne way that could work: a `--fmt geojson` option to this command which uses the plugin that was registered for the specified extension.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1193090967, "label": "Proposal: datasette query"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1699#issuecomment-1092357672", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1699", "id": 1092357672, "node_id": "IC_kwDOBm6k_c5BHA4o", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-04-08T01:39:40Z", "updated_at": "2022-04-08T01:39:40Z", "author_association": "CONTRIBUTOR", "body": "> My best thought on how to differentiate them so far is plugins: if Datasette plugins that provide alternative outputs - like .geojson and .yml and suchlike - also work for the datasette query command that would make a lot of sense to me.\r\n\r\nThat's my thinking, too. It's really the thing I've been wanting since writing `datasette-geojson`, since I'm always exporting with `datasette --get`. The workflow I'm always looking for is something like this:\r\n\r\n```sh\r\ncd alltheplaces-datasette\r\ndatasette query dunkin_in_suffolk -f geojson -o dunkin_in_suffolk.geojson\r\n```\r\n\r\nI think this probably needs either a new plugin hook separate from `register_output_renderer` or a way to use that without going through the HTTP stack. Or maybe a render mode that writes to a stream instead of a response. Maybe there's a new key in the dictionary that `register_output_renderer` returns that handles CLI exports.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1193090967, "label": "Proposal: datasette query"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1699#issuecomment-1092361727", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1699", "id": 1092361727, "node_id": "IC_kwDOBm6k_c5BHB3_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-08T01:47:43Z", "updated_at": "2022-04-08T01:47:43Z", "author_association": "OWNER", "body": "A render mode for that plugin hook that writes to a stream is exactly what I have in mind:\r\n- #1062 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1193090967, "label": "Proposal: datasette query"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1699#issuecomment-1092370880", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1699", "id": 1092370880, "node_id": "IC_kwDOBm6k_c5BHEHA", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-04-08T02:07:40Z", "updated_at": "2022-04-08T02:07:40Z", "author_association": "CONTRIBUTOR", "body": "So maybe `render_output_render` returns something like this:\r\n\r\n```python\r\n@hookimpl\r\ndef register_output_renderer(datasette):\r\n    return {\r\n        \"extension\": \"geojson\",\r\n        \"render\": render_geojson,\r\n        \"stream\": stream_geojson,\r\n        \"can_render\": can_render_geojson,\r\n    }\r\n```\r\n\r\nAnd stream gets an iterator, instead of a list of rows, so it can efficiently handle large queries. Maybe it also gets passed a destination stream, or it returns an iterator. I'm not sure what makes more sense. Either way, that might cover both CLI exports and streaming responses.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1193090967, "label": "Proposal: datasette query"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1699#issuecomment-1092386254", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1699", "id": 1092386254, "node_id": "IC_kwDOBm6k_c5BHH3O", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-04-08T02:39:25Z", "updated_at": "2022-04-08T02:39:25Z", "author_association": "CONTRIBUTOR", "body": "And just to think this through a little more, here's what `stream_geojson` might look like:\r\n\r\n```python\r\nasync def stream_geojson(datasette, columns, rows, database, stream):\r\n    db = datasette.get_database(database)\r\n    for row in rows:\r\n        feature = await row_to_geojson(row, db)\r\n        stream.write(feature + \"\\n\") # just assuming newline mode for now\r\n```\r\n\r\nAlternately, that could be an async generator, like this:\r\n\r\n```python\r\nasync def stream_geojson(datasette, columns, rows, database):\r\n    db = datasette.get_database(database)\r\n    for row in rows:\r\n        feature = await row_to_geojson(row, db)\r\n        yield feature\r\n```\r\n\r\nNot sure which makes more sense, but I think this pattern would open up a lot of possibility. If you had your [stream_indented_json](https://til.simonwillison.net/python/output-json-array-streaming) function, you could do `yield from stream_indented_json(rows, 2)` and be one your way.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1193090967, "label": "Proposal: datasette query"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1703#issuecomment-1092850719", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1703", "id": 1092850719, "node_id": "IC_kwDOBm6k_c5BI5Qf", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-04-08T13:18:04Z", "updated_at": "2022-04-08T13:18:04Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#1703](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (73aabe6) into [main](https://codecov.io/gh/simonw/datasette/commit/90d1be9952db9aaddc21a536e4d00a8de44765d7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (90d1be9) will **not change** coverage.\n> The diff coverage is `n/a`.\n\n```diff\n@@           Coverage Diff           @@\n##             main    #1703   +/-   ##\n=======================================\n  Coverage   91.75%   91.75%           \n=======================================\n  Files          34       34           \n  Lines        4573     4573           \n=======================================\n  Hits         4196     4196           \n  Misses        377      377           \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute <relative> (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [90d1be9...73aabe6](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1197298420, "label": "Update beautifulsoup4 requirement from <4.11.0,>=4.8.1 to >=4.8.1,<4.12.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1693#issuecomment-1093454899", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1693", "id": 1093454899, "node_id": "IC_kwDOBm6k_c5BLMwz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-08T23:07:04Z", "updated_at": "2022-04-08T23:07:04Z", "author_association": "OWNER", "body": "Tests failed here due to this issue:\r\n- https://github.com/psf/black/pull/2987\r\n\r\nA future Black release should fix that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1184850337, "label": "Bump black from 22.1.0 to 22.3.0"}, "performed_via_github_app": null}