html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/782#issuecomment-782708938,https://api.github.com/repos/simonw/datasette/issues/782,782708938,MDEyOklzc3VlQ29tbWVudDc4MjcwODkzOA==,9599,simonw,2021-02-20T16:22:14Z,2021-02-20T16:22:14Z,OWNER,"I'm leaning back in the direction of a flat JSON array of objects as the default - this:
`/fixtures/roadside_attractions.json`
Would return:
```json
[
{
""pk"": 1,
""name"": ""The Mystery Spot"",
""address"": ""465 Mystery Spot Road, Santa Cruz, CA 95065"",
""latitude"": 37.0167,
""longitude"": -122.0024
},
{
""pk"": 2,
""name"": ""Winchester Mystery House"",
""address"": ""525 South Winchester Boulevard, San Jose, CA 95128"",
""latitude"": 37.3184,
""longitude"": -121.9511
},
{
""pk"": 3,
""name"": ""Burlingame Museum of PEZ Memorabilia"",
""address"": ""214 California Drive, Burlingame, CA 94010"",
""latitude"": 37.5793,
""longitude"": -122.3442
},
{
""pk"": 4,
""name"": ""Bigfoot Discovery Museum"",
""address"": ""5497 Highway 9, Felton, CA 95018"",
""latitude"": 37.0414,
""longitude"": -122.0725
}
]
```
To get the version that includes pagination information you would use the `?_extra=` parameter. For example:
`/fixtures/roadside_attractions.json?_extra=total&_extra=next_url`
```json
{
""rows"": [
{
""pk"": 1,
""name"": ""The Mystery Spot"",
""address"": ""465 Mystery Spot Road, Santa Cruz, CA 95065"",
""latitude"": 37.0167,
""longitude"": -122.0024
},
{
""pk"": 2,
""name"": ""Winchester Mystery House"",
""address"": ""525 South Winchester Boulevard, San Jose, CA 95128"",
""latitude"": 37.3184,
""longitude"": -121.9511
},
{
""pk"": 3,
""name"": ""Burlingame Museum of PEZ Memorabilia"",
""address"": ""214 California Drive, Burlingame, CA 94010"",
""latitude"": 37.5793,
""longitude"": -122.3442
},
{
""pk"": 4,
""name"": ""Bigfoot Discovery Museum"",
""address"": ""5497 Highway 9, Felton, CA 95018"",
""latitude"": 37.0414,
""longitude"": -122.0725
}
],
""total"": 4,
""next_url"": null
}
```
ANY usage of the `?_extra=` parameter would turn the list into an object with a `""rows""` key.
Opting in to the `total` is nice because it's actually expensive to run a count, so only doing a count if the user requests it feels good.
But... having to add `?_extra=total&_extra=next_url` for the common case of wanting both the total count and the URL to get the next page of results is a bit verbose. So maybe support aliases, like `?_extra=paginated` which is a shortcut for `?_extra=total&_extra=next_url`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782709270,https://api.github.com/repos/simonw/datasette/issues/782,782709270,MDEyOklzc3VlQ29tbWVudDc4MjcwOTI3MA==,9599,simonw,2021-02-20T16:23:51Z,2021-02-20T16:24:11Z,OWNER,"Also how would you opt out of returning the `""rows""` key? I sometimes want to do this - if I want to get back just the count or just the facets for example.
Some options:
* `/fixtures/roadside_attractions.json?_extra=total&_extra=-rows`
* `/fixtures/roadside_attractions.json?_extra=total&_skip=rows`
* `/fixtures/roadside_attractions.json?_extra=total&_size=0`
I quite like that last one with `?_size=0`. I think it would still return `""rows"": []` but that's OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782709425,https://api.github.com/repos/simonw/datasette/issues/782,782709425,MDEyOklzc3VlQ29tbWVudDc4MjcwOTQyNQ==,9599,simonw,2021-02-20T16:24:54Z,2021-02-20T16:24:54Z,OWNER,Having shortcuts means I could support `?_extra=all` for returning ALL possible keys.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782739926,https://api.github.com/repos/simonw/datasette/issues/782,782739926,MDEyOklzc3VlQ29tbWVudDc4MjczOTkyNg==,9599,simonw,2021-02-20T19:51:30Z,2021-02-20T19:52:19Z,OWNER,"Demos:
- https://latest-with-plugins.datasette.io/github/commits.json-preview
- https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=next_url
- https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total
- https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=next_url&_extra=total
- https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total&_size=0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782740488,https://api.github.com/repos/simonw/datasette/issues/782,782740488,MDEyOklzc3VlQ29tbWVudDc4Mjc0MDQ4OA==,9599,simonw,2021-02-20T19:55:23Z,2021-02-20T19:55:23Z,OWNER,"Am I saying you won't get back a key in the response unless you explicitly request it, either by name or by specifying a bundle of extras (e.g. `all` or `paginated`)?
The `""truncated"": true` key that tells you that your arbitrary query returned more than X results but was truncated is pretty important, do I really want people to have to opt-in to that one?
Also: having bundles like `all` or `paginated` live in the same namespace as single keys like `next_url` or `total` is a little odd - you can't tell by looking at them if they'll add a key called `all` or if they'll add a bunch of other stuff.
Maybe bundles could be prefixed with something, perhaps an underscore? `?_extra=_all` and `?_extra=_paginated` for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782740604,https://api.github.com/repos/simonw/datasette/issues/782,782740604,MDEyOklzc3VlQ29tbWVudDc4Mjc0MDYwNA==,9599,simonw,2021-02-20T19:56:21Z,2021-02-20T19:56:33Z,OWNER,"I think I want to support `?_extra=next_url,total` in addition to `?_extra=next_url&_extra=total` - partly because it's less characters to type, and also because I know there exist URL handling library that don't know how to handle the same parameter multiple times (though they're going to break against Datasette already, so it's not a big deal).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782740985,https://api.github.com/repos/simonw/datasette/issues/782,782740985,MDEyOklzc3VlQ29tbWVudDc4Mjc0MDk4NQ==,9599,simonw,2021-02-20T19:59:21Z,2021-02-20T19:59:21Z,OWNER,"This design should be influenced by how it's implemented.
One implementation that could be nice is that each of the keys that can be requested - `next_url`, `total` etc - maps to an `async def` function which can do the work. So that expensive `count(*)` will only be executed by the `async def total` function if it is requested.
This raises more questions: Both `next` and `next_url` work off the same underlying data, so if they are both requested can we re-use the work that `next` does somehow? Maybe by letting these functions depend on each other (so `next_url()` knows to first call `next()`, but only if it hasn't been called already.
I think I need to flesh out the full default collection of `?_extra=` parameters in order to design how they will work under the hood.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782741107,https://api.github.com/repos/simonw/datasette/issues/782,782741107,MDEyOklzc3VlQ29tbWVudDc4Mjc0MTEwNw==,9599,simonw,2021-02-20T20:00:22Z,2021-02-20T20:00:22Z,OWNER,"A really exciting opportunity this opens up is for parallel execution - the `facets()` and `suggested_facets()` and `total()` async functions could be called in parallel, which could speed things up if I'm confident the SQLite thread pool can execute on multiple CPU cores (it should be able to because the Python `sqlite3` module releases the GIL while it's executing C code).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782741719,https://api.github.com/repos/simonw/datasette/issues/782,782741719,MDEyOklzc3VlQ29tbWVudDc4Mjc0MTcxOQ==,9599,simonw,2021-02-20T20:05:04Z,2021-02-20T20:05:04Z,OWNER,"> The only advantage of headers is that you don’t need to do .rows, but that’s actually good as a data validation step anyway—if .rows is missing assume there’s an error and do your error handling path instead of parsing the rest.
This is something I've not thought very hard about. If there's an error, I need to return a top-level object, not a top-level array, so I can provide details of the error.
But this means that client code will have to handle this difference - it will have to know that the returned data can be array-shaped if nothing went wrong, and object-shaped if there's an error.
The HTTP status code helps here - calling client code can know that a 200 status code means there will be an array, but an error status code means an object.
If developers really hate that the shape could be different, they can always use `?_extra=next` to ensure that the top level item is an object whether or not an error occurred. So I think this is OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782742233,https://api.github.com/repos/simonw/datasette/issues/782,782742233,MDEyOklzc3VlQ29tbWVudDc4Mjc0MjIzMw==,9599,simonw,2021-02-20T20:09:16Z,2021-02-20T20:09:16Z,OWNER,I just noticed that https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total&_size=0&_trace=1 executes 35 SQL queries at the moment! A great reminder that a big improvement from this change will be a reduction in queries through not calculating things like suggested facets unless they are explicitly requested.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782745199,https://api.github.com/repos/simonw/datasette/issues/782,782745199,MDEyOklzc3VlQ29tbWVudDc4Mjc0NTE5OQ==,30665,frankieroberto,2021-02-20T20:32:03Z,2021-02-20T20:32:03Z,NONE,"I think it’s a good idea if the top level item of the response JSON is always an object, rather than an array, at least as the default. Mainly because it allows you to add extra keys in a backwards-compatible way. Also just seems more expected somehow.
The API design guidance for the UK government also recommends this: https://www.gov.uk/guidance/gds-api-technical-and-data-standards#use-json
I also strongly dislike having versioned APIs (eg with a `/v1/` path prefix, as it invariably means that old versions stop working at some point, even though the bit of the API you’re using might not have changed at all.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782746633,https://api.github.com/repos/simonw/datasette/issues/782,782746633,MDEyOklzc3VlQ29tbWVudDc4Mjc0NjYzMw==,9599,simonw,2021-02-20T20:43:07Z,2021-02-20T20:43:07Z,OWNER,"Another option: `.json` always returns an object with a list of keys that gets increased through adding `?_extra=` parameters.
`.jsona` always returns a JSON array of objects
I had something similar to this in Datasette a few years ago - a `.jsono` extension, which still redirects to the `shape=array` version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782746755,https://api.github.com/repos/simonw/datasette/issues/782,782746755,MDEyOklzc3VlQ29tbWVudDc4Mjc0Njc1NQ==,30665,frankieroberto,2021-02-20T20:44:05Z,2021-02-20T20:44:05Z,NONE,"Minor suggestion: rename `size` query param to `limit`, to better reflect that it’s a maximum number of rows returned rather than a guarantee of getting that number, and also for consistency with the SQL keyword?
I like the idea of specifying a limit of 0 if you don’t want any rows data - and returning an empty array under the `rows` key seems fine.
Have you given any thought as to whether to pretty print (format with spaces) the output or not? Can be useful for debugging/exploring in a browser or other basic tools which don’t parse the JSON. Could be default (can’t be much bigger with gzip?) or opt-in.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782747164,https://api.github.com/repos/simonw/datasette/issues/782,782747164,MDEyOklzc3VlQ29tbWVudDc4Mjc0NzE2NA==,9599,simonw,2021-02-20T20:47:16Z,2021-02-20T20:47:16Z,OWNER,(I started a thread on Twitter about this: https://twitter.com/simonw/status/1363220355318358016),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782747743,https://api.github.com/repos/simonw/datasette/issues/782,782747743,MDEyOklzc3VlQ29tbWVudDc4Mjc0Nzc0Mw==,9599,simonw,2021-02-20T20:52:10Z,2021-02-20T20:52:10Z,OWNER,"> Minor suggestion: rename `size` query param to `limit`, to better reflect that it’s a maximum number of rows returned rather than a guarantee of getting that number, and also for consistency with the SQL keyword?
The problem there is that `?_size=x` isn't actually doing the same thing as the SQL `limit` keyword. Consider this query:
https://latest-with-plugins.datasette.io/github?sql=select+*+from+commits - `select * from commits`
Datasette returns 1,000 results, and shows a ""Custom SQL query returning more than 1,000 rows"" message at the top. That's the `size` kicking in - I only fetch the first 1,000 results from the cursor to avoid exhausting resources. In the JSON version of that at https://latest-with-plugins.datasette.io/github.json?sql=select+*+from+commits there's a `""truncated"": true` key to let you know what happened.
I find myself using `?_size=2` against Datasette occasionally if I know the rows being returned are really big and I don't want to load 10+MB of HTML.
This is only really a concern for arbitrary SQL queries though - for table pages such as https://latest-with-plugins.datasette.io/github/commits?_size=10 adding `?_size=10` actually puts a `limit 10` on the underlying SQL query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782747878,https://api.github.com/repos/simonw/datasette/issues/782,782747878,MDEyOklzc3VlQ29tbWVudDc4Mjc0Nzg3OA==,9599,simonw,2021-02-20T20:53:11Z,2021-02-20T20:53:11Z,OWNER,"... though thinking about this further, I could re-implement the `select * from commits` (but only return a max of 10 results) feature using a nested `select * from (select * from commits) limit 10` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782748093,https://api.github.com/repos/simonw/datasette/issues/782,782748093,MDEyOklzc3VlQ29tbWVudDc4Mjc0ODA5Mw==,9599,simonw,2021-02-20T20:54:52Z,2021-02-20T20:54:52Z,OWNER,"> Have you given any thought as to whether to pretty print (format with spaces) the output or not? Can be useful for debugging/exploring in a browser or other basic tools which don’t parse the JSON. Could be default (can’t be much bigger with gzip?) or opt-in.
Adding a `?_pretty=1` option that does that is a great idea, I'm filing a ticket for it: #1237","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782748501,https://api.github.com/repos/simonw/datasette/issues/782,782748501,MDEyOklzc3VlQ29tbWVudDc4Mjc0ODUwMQ==,9599,simonw,2021-02-20T20:58:18Z,2021-02-20T20:58:18Z,OWNER,"Yet another option: support a `?_path=x` option which returns a nested path from the result. So you could do this:
`/github/commits.json?_path=rows` - to get back a top-level array pulled from the `""rows""` key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782756398,https://api.github.com/repos/simonw/datasette/issues/782,782756398,MDEyOklzc3VlQ29tbWVudDc4Mjc1NjM5OA==,601316,simonrjones,2021-02-20T22:05:48Z,2021-02-20T22:05:48Z,NONE,"> I think it’s a good idea if the top level item of the response JSON is always an object, rather than an array, at least as the default.
I agree it is more predictable if the top level item is an object with a rows or data object that contains an array of data, which then allows for other top-level meta data.
I can see the argument for removing this and just using an array for convenience - but I think that's OK as an option (as you have now).
Rather than have lots of top-level keys you could have a ""meta"" object to contain non-data stuff. You could use something like ""links"" for API endpoint URLs (or use a standard like HAL). Which would then leave the top level a bit cleaner - if that's what you what.
Have you had much feedback from users who use the Datasette API a lot?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782765665,https://api.github.com/repos/simonw/datasette/issues/782,782765665,MDEyOklzc3VlQ29tbWVudDc4Mjc2NTY2NQ==,9599,simonw,2021-02-20T23:34:41Z,2021-02-20T23:34:41Z,OWNER,"OK, I'm back to the ""top level object as the default"" side of things now - it's pretty much unanimous at this point, and it's certainly true that it's not a decision you'll even regret.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,