issue_comments
27 rows where issue = 323681589
This data as json, CSV (advanced)
Suggested facets: created_at (date), updated_at (date)
| id ▼ | html_url | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 389570841 | https://github.com/simonw/datasette/issues/266#issuecomment-389570841 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM4OTU3MDg0MQ== | simonw 9599 | 2018-05-16T15:54:49Z | 2018-06-15T07:41:09Z | OWNER | At the most basic level, this will work based on an extension. Most places you currently put a `.json` extension should also allow a `.csv` extension. By default this will return the exact results you see on the current page (default max will remain 1000). ## Streaming all records Where things get interested is *streaming mode*. This will be an option which returns ALL matching records as a streaming CSV file, even if that ends up being millions of records. I think the best way to build this will be on top of the existing mechanism used to efficiently implement keyset pagination via `_next=` tokens. ## Expanding foreign keys For tables with foreign key references it would be useful if the CSV format could expand those references to include the labels from `label_column` - maybe via an additional `?_expand=1` option. When expanding each foreign key column will be shown twice: rowid,city_id,city_id_label,state | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 389572201 | https://github.com/simonw/datasette/issues/266#issuecomment-389572201 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM4OTU3MjIwMQ== | simonw 9599 | 2018-05-16T15:58:43Z | 2018-05-16T16:00:47Z | OWNER | This will likely be implemented in the `BaseView` class, which needs to know how to spot the `.csv` extension, call the underlying JSON generating function and then return the `columns` and `rows` as correctly formatted CSV. https://github.com/simonw/datasette/blob/9959a9e4deec8e3e178f919e8b494214d5faa7fd/datasette/views/base.py#L201-L207 This means it will take ALL arguments that are available to the `.json` view. It may ignore some (e.g. `_facet=` makes no sense since CSV tables don't have space to show the facet results). In streaming mode, things will behave a little bit differently - in particular, if `_stream=1` then `_next=` will be forbidden. It can't include a length header because we don't know how many bytes it will be CSV output will throw an error if the endpoint doesn't have rows and columns keys eg `/-/inspect.json` So the implementation... - looks for the `.csv` extension - internally fetches the `.json` data instead - If no `_stream` it just transposes that JSON to CSV with the correct content type header - If `_stream=1` - checks for `_next=` and throws an error if it was provided - Otherwise... fetch first page and emit CSV header and first set of rows - Then start async looping, emitting more CSV rows and following the `_next=` internal reference until done I like that this takes advantage of efficient pagination. It may not work so well for views which use offset/limit though. It won't work at all for custom SQL because custom SQL doesn't support _next= pagination. That's fine. For views... easiest fix is to cut off after first X000 records. That seems OK. View JSON would need to include a property that the mechanism can identify. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 389579363 | https://github.com/simonw/datasette/issues/266#issuecomment-389579363 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM4OTU3OTM2Mw== | simonw 9599 | 2018-05-16T16:20:06Z | 2018-05-16T16:20:06Z | OWNER | I started a thread on Twitter discussing various CSV output dialects: https://twitter.com/simonw/status/996783395504979968 - I want to pick defaults which will work as well as possible for whatever tools people might be using to consume the data. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 389579762 | https://github.com/simonw/datasette/issues/266#issuecomment-389579762 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM4OTU3OTc2Mg== | simonw 9599 | 2018-05-16T16:21:12Z | 2018-05-16T16:21:12Z | OWNER | > I basically want someone to tell me which arguments I can pass to Python's csv.writer() function that will result in the least complaints from people who try to parse the results :) https://twitter.com/simonw/status/996786815938977792 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 389592566 | https://github.com/simonw/datasette/issues/266#issuecomment-389592566 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM4OTU5MjU2Ng== | simonw 9599 | 2018-05-16T17:01:29Z | 2018-05-16T17:02:21Z | OWNER | Let's provide a CSV Dialect definition too: https://frictionlessdata.io/specs/csv-dialect/ - via https://twitter.com/drewdaraabrams/status/996794915680997382 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 389608473 | https://github.com/simonw/datasette/issues/266#issuecomment-389608473 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM4OTYwODQ3Mw== | simonw 9599 | 2018-05-16T17:52:35Z | 2018-05-16T17:54:11Z | OWNER | There are some code examples in this issue which should help with the streaming part: https://github.com/channelcat/sanic/issues/1067 Also https://github.com/channelcat/sanic/blob/master/docs/sanic/streaming.md#response-streaming | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 389626715 | https://github.com/simonw/datasette/issues/266#issuecomment-389626715 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM4OTYyNjcxNQ== | simonw 9599 | 2018-05-16T18:50:46Z | 2018-05-16T18:50:46Z | OWNER | > I’d recommend using the Windows-1252 encoding for maximum compatibility, unless you have any characters not in that set, in which case use UTF8 with a byte order mark. Bit of a pain, but some progams (eg various versions of Excel) don’t read UTF8. **frankieroberto** https://twitter.com/frankieroberto/status/996823071947460616 > There is software that consumes CSV and doesn't speak UTF8!? Huh. Well I can't just use Windows-1252 because I need to support the full UTF8 range of potential data - maybe I should support an optional ?_encoding=windows-1252 argument **simonw** https://twitter.com/simonw/status/996824677245857793 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 389893810 | https://github.com/simonw/datasette/issues/266#issuecomment-389893810 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM4OTg5MzgxMA== | simonw 9599 | 2018-05-17T14:49:35Z | 2018-05-17T14:49:35Z | OWNER | Idea: add a `supports_csv = False` property to `BaseView` and over-ride it to `True` just on the view classes that should support CSV (Table and Row). Slight subtlety: the `DatabaseView` class only supports CSV in the `custom_sql()` path. Maybe that needs to be refactored a bit. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 389894382 | https://github.com/simonw/datasette/issues/266#issuecomment-389894382 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM4OTg5NDM4Mg== | simonw 9599 | 2018-05-17T14:51:13Z | 2018-05-17T14:53:23Z | OWNER | I should definitely sanity check if the `_next=` route really is the most efficient way to build this. It may turn out that iterating over a SQLite cursor with a million rows in it is super-efficient and would provide much more reliable performance (plus solve the problem for retrieving full custom SQL queries where we can't do keyset pagination). Problem here is that we run SQL queries in a thread pool. A query that returns millions of rows would presumably tie up a SQL thread until it has finished, which could block the server. This may be a reason to stick with `_next=` keyset pagination - since it ensures each SQL thread yields back again after each 1,000 rows. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 393020749 | https://github.com/simonw/datasette/issues/266#issuecomment-393020749 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5MzAyMDc0OQ== | simonw 9599 | 2018-05-30T03:42:54Z | 2018-05-30T03:42:54Z | OWNER | Challenge: how to deal with tables where the name ends in `.csv`? I actually have one of these in the test suite at the moment: https://github.com/simonw/datasette/blob/d69ebce53385b7c6fafb85fdab3b136dbf3f332c/tests/fixtures.py#L234-L237 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 394417567 | https://github.com/simonw/datasette/issues/266#issuecomment-394417567 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5NDQxNzU2Nw== | simonw 9599 | 2018-06-04T16:30:48Z | 2018-06-04T16:32:55Z | OWNER | When serving streaming responses, I need to check that a large CSV file doesn't completely max out the CPU in a way that is harmful to the rest of the instance. If it does, one option may be to insert an async sleep call in between each chunk that is streamed back. This could be controlled by a `csv_pause_ms` config setting, defaulting to maybe 5 but can be disabled entirely by setting to 0. That's only if testing proves that this is a necessary mechanism. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397534196 | https://github.com/simonw/datasette/issues/266#issuecomment-397534196 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5NzUzNDE5Ng== | simonw 9599 | 2018-06-15T07:12:16Z | 2018-06-15T07:12:16Z | OWNER | The first version of this is now shipped to master. I ended up rewriting most of the experimental branch to deal with the nasty issue described in #303 Demo is available on https://fivethirtyeight.datasettes.com/fivethirtyeight-ab24e01/most-common-name%2Fsurnames  Here's the CSV version of that page: https://fivethirtyeight.datasettes.com/fivethirtyeight-ab24e01/most-common-name%2Fsurnames.csv | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397534404 | https://github.com/simonw/datasette/issues/266#issuecomment-397534404 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5NzUzNDQwNA== | simonw 9599 | 2018-06-15T07:13:20Z | 2018-06-15T07:13:20Z | OWNER | Still to add: the streaming version that iterates through all of the pages, as seen in experimental commit https://github.com/simonw/datasette/commit/ced379ea325787b8c3bf0a614daba1fa4856a3bd | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397534498 | https://github.com/simonw/datasette/issues/266#issuecomment-397534498 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5NzUzNDQ5OA== | simonw 9599 | 2018-06-15T07:13:52Z | 2018-06-15T07:13:52Z | OWNER | Also needs documentation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397729945 | https://github.com/simonw/datasette/issues/266#issuecomment-397729945 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5NzcyOTk0NQ== | simonw 9599 | 2018-06-15T20:13:05Z | 2018-06-15T20:13:05Z | OWNER | The "This data as ..." area of the page is getting a bit untidy, especially if I'm going to add other download options in the future. I think I'll move the HTML to the page footer (less concerns about taking up lots of space there) and then have a bit of JavaScript that turns it into a show/hide menu of some sort in its current location. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397842246 | https://github.com/simonw/datasette/issues/266#issuecomment-397842246 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5Nzg0MjI0Ng== | simonw 9599 | 2018-06-16T22:27:59Z | 2018-06-16T22:27:59Z | OWNER | Two demos of the new functionality in #233 as it applies to CSV: * https://datasette-labels-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List.csv?_labels=on - CSV with all foreign key columns expanded * https://datasette-labels-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List.csv?_label=qSpecies - CSV with specific columns expanded | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397842667 | https://github.com/simonw/datasette/issues/266#issuecomment-397842667 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5Nzg0MjY2Nw== | simonw 9599 | 2018-06-16T22:38:15Z | 2018-06-18T05:55:11Z | OWNER | Still todo: - [x] Streaming version - [ ] Tidy up the "This data as ..." UI - [x] Default .csv (and .json) links to use `?_labels=on` (only if at least one foreign key detected) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397912840 | https://github.com/simonw/datasette/issues/266#issuecomment-397912840 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5NzkxMjg0MA== | simonw 9599 | 2018-06-17T23:13:35Z | 2018-06-17T23:16:42Z | OWNER | This worked! https://github.com/simonw/datasette/commit/5a0a82faf9cf9dd109d76181ed00eea19472087c - it spat out a 76MB CSV when I ran it against the sf-trees demo database. It was just a quick hack though - it currently ignores `_labels=` and `_dl=` which need to be supported. I'm going to add a config option for turning full CSV export off just in case any Datasette users are uncomfortable with URLs that churn out that much data in one go. ``` ConfigOption("allow_csv_stream", True, """ Allow .csv?_stream=1 to download all rows (ignoring max_returned_rows) """.strip()), ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397915258 | https://github.com/simonw/datasette/issues/266#issuecomment-397915258 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5NzkxNTI1OA== | simonw 9599 | 2018-06-18T00:01:05Z | 2018-06-18T00:01:05Z | OWNER | Someone malicious could use a UNION to generate an unpleasantly large CSV response. I'll add another config setting which limits the response size to 100MB but can be turned off by setting it to 0. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397915403 | https://github.com/simonw/datasette/issues/266#issuecomment-397915403 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5NzkxNTQwMw== | simonw 9599 | 2018-06-18T00:03:17Z | 2018-06-18T00:14:37Z | OWNER | Since CSV streaming export doesn't work for custom SQL queries (since they don't support `_next=` pagination) there's no need to provide a option that disables streams just for custom SQL. Related: the UI should not show the option to download everything on custom SQL pages. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397916091 | https://github.com/simonw/datasette/issues/266#issuecomment-397916091 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5NzkxNjA5MQ== | simonw 9599 | 2018-06-18T00:13:43Z | 2018-06-18T00:15:50Z | OWNER | I was also worried about the performance of pagination over custom `_sort` orders or views which use offset pagination - but Datasette's SQL time limits should prevent those from getting out of hand. This does mean that a streaming CSV file may be truncated with an error - if this happens we should ensure the error is written out as the last line of the CSV so anyone who tried to import it gets a relevant error message informing them that the export did not complete. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397916321 | https://github.com/simonw/datasette/issues/266#issuecomment-397916321 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5NzkxNjMyMQ== | simonw 9599 | 2018-06-18T00:17:44Z | 2018-06-18T00:18:05Z | OWNER | The export UI could be a GET form controlling various parameters. This would discourage crawlers from hitting the export links and would also allow us to express the full range of export options. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397918264 | https://github.com/simonw/datasette/issues/266#issuecomment-397918264 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5NzkxODI2NA== | simonw 9599 | 2018-06-18T00:49:35Z | 2018-06-18T00:49:35Z | OWNER | Simpler design: the top of the page will link to basic .json and .csv and "advanced" - which will fragment link to an advanced export format the bottom of the page. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397923253 | https://github.com/simonw/datasette/issues/266#issuecomment-397923253 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5NzkyMzI1Mw== | simonw 9599 | 2018-06-18T01:49:52Z | 2018-06-18T03:02:28Z | OWNER | Ideally the downloadable filenames of exported CSVs would differ across different querystring parameters. Maybe S`treet_Trees-56cbd54.csv` where `56cbd54` is a hash of the querystring? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397949002 | https://github.com/simonw/datasette/issues/266#issuecomment-397949002 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5Nzk0OTAwMg== | simonw 9599 | 2018-06-18T05:53:17Z | 2018-06-18T05:53:17Z | OWNER | Advanced export pane:  | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 397952129 | https://github.com/simonw/datasette/issues/266#issuecomment-397952129 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5Nzk1MjEyOQ== | simonw 9599 | 2018-06-18T06:15:36Z | 2018-06-18T06:15:51Z | OWNER | Advanced export pane demo: https://latest.datasette.io/fixtures-35b6eb6/facetable?_size=4 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 | |
| 398098582 | https://github.com/simonw/datasette/issues/266#issuecomment-398098582 | https://api.github.com/repos/simonw/datasette/issues/266 | MDEyOklzc3VlQ29tbWVudDM5ODA5ODU4Mg== | simonw 9599 | 2018-06-18T15:40:32Z | 2018-06-18T15:40:32Z | OWNER | This is now released in Datasette 0.23! http://datasette.readthedocs.io/en/latest/changelog.html#v0-23 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Export to CSV 323681589 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] (
[html_url] TEXT,
[issue_url] TEXT,
[id] INTEGER PRIMARY KEY,
[node_id] TEXT,
[user] INTEGER REFERENCES [users]([id]),
[created_at] TEXT,
[updated_at] TEXT,
[author_association] TEXT,
[body] TEXT,
[reactions] TEXT,
[issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
ON [issue_comments] ([user]);