html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/datasette/issues/1014#issuecomment-706631006,https://api.github.com/repos/simonw/datasette/issues/1014,706631006,MDEyOklzc3VlQ29tbWVudDcwNjYzMTAwNg==,9599,simonw,2020-10-11T00:36:43Z,2020-10-11T00:36:43Z,OWNER,"Demo using [paginate-json](https://github.com/simonw/paginate-json): ``` % paginate-json 'https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array' | jq '. | length' https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cd%2Cv http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ch%2Cr http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cl%2Cn http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cp%2Cj http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ct%2Cf http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cx%2Cb http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ca%2Cx http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ce%2Ct http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ci%2Cp http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Cm%2Cl 1001 ``` New documentation: https://docs.datasette.io/en/latest/json_api.html#pagination","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers, https://github.com/simonw/datasette/issues/782#issuecomment-706740250,https://api.github.com/repos/simonw/datasette/issues/782,706740250,MDEyOklzc3VlQ29tbWVudDcwNjc0MDI1MA==,9599,simonw,2020-10-11T17:40:48Z,2020-10-11T17:43:07Z,OWNER,"Building this plugin reminded me of an oddity of the `register_output_renderer()` plugin hook: one of the arguments that can be passed to it is `data`, which is the default internal data structure created by Datasette - but I deliberately avoided documenting that on https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette because it's not a stable interface. That's not ideal. I'd like custom renderers to be able to access this data to get at things like suggested facets, on an opt-in basis. So maybe that kind of stuff is re-implemented as ""extras"" which are awaitable callables - then renderer plugins can call the extras that they need to as part of their execution. To illustrate the problem (in this case the need to access `next_url`) here's my first prototype of the plugin: ```python from datasette import hookimpl from datasette.utils.asgi import Response @hookimpl def register_output_renderer(datasette): return { ""extension"": ""json-preview"", ""render"": json_preview, } def json_preview(data, columns, rows): next_url = data.get(""next_url"") headers = {} if next_url: headers[""link""] = '<{}>; rel=""next""'.format(next_url) return Response.json([dict(zip(columns, row)) for row in rows], headers=headers) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/782#issuecomment-706745236,https://api.github.com/repos/simonw/datasette/issues/782,706745236,MDEyOklzc3VlQ29tbWVudDcwNjc0NTIzNg==,9599,simonw,2020-10-11T18:16:05Z,2020-10-11T18:16:05Z,OWNER,Here's the `datasette-json-preview` plugin I'll be using to experiment with different formats: https://github.com/simonw/datasette-json-preview,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/1015#issuecomment-706756879,https://api.github.com/repos/simonw/datasette/issues/1015,706756879,MDEyOklzc3VlQ29tbWVudDcwNjc1Njg3OQ==,9599,simonw,2020-10-11T19:35:03Z,2020-10-11T19:35:03Z,OWNER,"Since plugins are installed via pip this would require Datasette to be restarted. This StackOverflow thread looks relevant to that: https://stackoverflow.com/questions/11329917/restart-python-script-from-within-itself This recipe looks promising: ```python import os import sys import psutil import logging def restart_program(): """"""Restarts the current program, with file objects and descriptors cleanup """""" try: p = psutil.Process(os.getpid()) for handler in p.get_open_files() + p.connections(): os.close(handler.fd) except Exception, e: logging.error(e) python = sys.executable os.execl(python, python, *sys.argv) ``` https://docs.python.org/3/library/os.html#os.execl says about `os.execl`: > These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718910318,Research: could Datasette install its own plugins?, https://github.com/dogsheep/evernote-to-sqlite/issues/1#issuecomment-706775706,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/1,706775706,MDEyOklzc3VlQ29tbWVudDcwNjc3NTcwNg==,9599,simonw,2020-10-11T22:14:00Z,2020-10-11T22:14:00Z,MEMBER,A live demo would be good too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718934942,Documentation on how to use this with Datasette, https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776180,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5,706776180,MDEyOklzc3VlQ29tbWVudDcwNjc3NjE4MA==,9599,simonw,2020-10-11T22:17:55Z,2020-10-11T22:17:55Z,MEMBER,"We could even do server-side thumbnailing for some of these images, but I'm inclined to serve up the full size ones and set a width on the image element based on the `width` attribute on ``.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from tags inline in Datasette, https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776242,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5,706776242,MDEyOklzc3VlQ29tbWVudDcwNjc3NjI0Mg==,9599,simonw,2020-10-11T22:18:30Z,2020-10-11T22:19:48Z,MEMBER,"Alternatively, rather than relying on `datasette-media` this could base64-embed the images. `evernote-to-sqlite` could register itself as a Datasette plugin that knows how to do this. Maybe rename the column to `evernote_content` and register a render cell hook that knows how to rewrite those note bodies so that they are visible? Might need to feed them through Bleach too, just in case any nasty code can get into them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from tags inline in Datasette, https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776447,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5,706776447,MDEyOklzc3VlQ29tbWVudDcwNjc3NjQ0Nw==,9599,simonw,2020-10-11T22:20:32Z,2020-10-11T22:20:32Z,MEMBER,Or... I could do this client-side. JavaScript that looks for `` tags and fetches the data using `fetch()` wouldn't be too hard to write.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from tags inline in Datasette, https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776680,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5,706776680,MDEyOklzc3VlQ29tbWVudDcwNjc3NjY4MA==,9599,simonw,2020-10-11T22:22:16Z,2020-10-11T22:22:16Z,MEMBER,"Maybe the best way do this is with a custom route, `/-/evernote/note-id` - that way I can clean the HTML and resolve the other things in the `` structure without using `render_cell()` and the like. My concern about using `render_cell()` is that it could lead to weird security problems when combined with `?sql=` queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from tags inline in Datasette, https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776808,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5,706776808,MDEyOklzc3VlQ29tbWVudDcwNjc3NjgwOA==,9599,simonw,2020-10-11T22:23:14Z,2020-10-11T22:23:14Z,MEMBER,... but it's still important to be able to get to the rendered note directly from the browse notes `/evernote/notes` page. Maybe use a simple `render_cell()` hook that just knows how to generate the link to the rendered note page?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from tags inline in Datasette, https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706784028,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4,706784028,MDEyOklzc3VlQ29tbWVudDcwNjc4NDAyOA==,9599,simonw,2020-10-11T23:20:32Z,2020-10-11T23:20:32Z,MEMBER,I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938508,Configure FTS + add an index on the date columns, https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785086,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6,706785086,MDEyOklzc3VlQ29tbWVudDcwNjc4NTA4Ng==,9599,simonw,2020-10-11T23:28:50Z,2020-10-11T23:28:50Z,MEMBER,"The XML for the OCR stuff is a bit weird. Currently I'm doing this to it: https://github.com/dogsheep/evernote-to-sqlite/blob/c33d7b043a45eb3e88676e5fa3ce31755199d9f8/evernote_to_sqlite/utils.py#L70-L78 This can produce some odd results, for example: > Sure 'Sure, 'Sure. Sure, Sure. sure sure. sure ? If you If Yau [you live jive In m 1n an area devoid of natural wonders, wanders, wonders ? wonders wonders. your mind will be blown, blown' blown. blown ? -e i ? ,1 IL it ? at ? KY ? fl ft bat at Which came from this image: ![image](https://user-images.githubusercontent.com/9599/95692952-5dd7c880-0bde-11eb-939a-d10b800a4105.png) The XML for that is: ```xml Sure 'Sure, 'Sure. Sure, Sure. sure sure. sure ? If you If Yau [you live jive In m 1n an area devoid of natural wonders, wanders, wonders ? wonders wonders. your mind will be blown, blown' blown. blown ? -e i ? ,1 IL it ? at ? KY ? fl ft bat at ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718949182,Better handling of OCR data, https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785201,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6,706785201,MDEyOklzc3VlQ29tbWVudDcwNjc4NTIwMQ==,9599,simonw,2020-10-11T23:29:39Z,2020-10-11T23:29:39Z,MEMBER,"It looks to me like each of those `` blocks has a number of guesses in order of confidence: ```xml wonders, wanders, wonders ? wonders wonders. ``` So maybe the best approach here is to just take the first `t` element within each `item`. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718949182,Better handling of OCR data, https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706786548,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4,706786548,MDEyOklzc3VlQ29tbWVudDcwNjc4NjU0OA==,9599,simonw,2020-10-11T23:39:46Z,2020-10-11T23:39:46Z,MEMBER,Should have used porter stemming for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938508,Configure FTS + add an index on the date columns, https://github.com/simonw/datasette/issues/1016#issuecomment-706788010,https://api.github.com/repos/simonw/datasette/issues/1016,706788010,MDEyOklzc3VlQ29tbWVudDcwNjc4ODAxMA==,9599,simonw,2020-10-11T23:50:39Z,2020-10-11T23:50:39Z,OWNER,"For consistency can reuse the icon used on selected facets: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718953669,"Add a ""delete"" icon next to filters (in addition to ""remove filter"")", https://github.com/simonw/datasette/issues/782#issuecomment-706735280,https://api.github.com/repos/simonw/datasette/issues/782,706735280,MDEyOklzc3VlQ29tbWVudDcwNjczNTI4MA==,9599,simonw,2020-10-11T17:03:01Z,2020-10-11T17:03:01Z,OWNER,"Should that default also include `""columns""` as a list of strings? That would be duplicate data of the keys in the `""rows""` list of objects, and I've never found myself wanting it in my own code - so I'm going to say no.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/782#issuecomment-706735200,https://api.github.com/repos/simonw/datasette/issues/782,706735200,MDEyOklzc3VlQ29tbWVudDcwNjczNTIwMA==,9599,simonw,2020-10-11T17:02:11Z,2020-10-11T17:14:51Z,OWNER,"Since the total count can be expensive to calculate, I'm inclined to make that an opt-in extra - maybe `?_extra=count`. Based on that, the default JSON shape could look something like this: ```json { ""rows"": [{""id"": 1}, {""id"": 2}], ""next"": ""2"", ""next_url"": ""/db/table?_next=2"" } ``` And with `?_extra=count`: ```json { ""rows"": [{""id"": 1}, {""id"": 2}], ""next"": ""2"", ""next_url"": ""/db/table?_next=2"", ""count"": 31 } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/782#issuecomment-706735341,https://api.github.com/repos/simonw/datasette/issues/782,706735341,MDEyOklzc3VlQ29tbWVudDcwNjczNTM0MQ==,9599,simonw,2020-10-11T17:03:29Z,2020-10-11T17:15:34Z,OWNER,"Maybe `.jsonfull` becomes a new renderer that returns ALL of the defined `?_extra=` blocks. Or... `?_extra=all` turns on ALL of the available information blocks (some of which can come from plugins).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/782#issuecomment-706736541,https://api.github.com/repos/simonw/datasette/issues/782,706736541,MDEyOklzc3VlQ29tbWVudDcwNjczNjU0MQ==,9599,simonw,2020-10-11T17:12:27Z,2020-10-11T17:12:27Z,OWNER,"The core issue that I keep reconsidering is whether the default `.json` representation should be an object or a list. Arguments in favour of a list: - It's what I always want. Almost all of the code that I've written against the API myself uses `?_shape=array`. - It's really easy to use. You can pipe it to e.g. `sqlite-utils insert`, you can load it into JavaScript without thinking about it. Arguments against: - Nowhere to put pagination or total counts. I added pagination to the `link:` HTTP header in #1014 (inspired by the WordPress and GitHub APIs) but I haven't solved for total count, and there's other stuff that's useful like `""truncated"": true` to indicate that more than 1000 results were returned and they were truncated. - An array is inherently non-extensible: if the root item is an object it's easy to add new features to it in a backwards-compatible way in the future. An array is a fixed format. But maybe that last point is a positive? It ensures the default `.json` format remains completely predictable forever. If `.json` DID default to an array of objects, the `?_shape=` argument could still be used to get back alternative formats. Maybe `.json?_extra=total` changes the shape of that default to be this instead: ```json { ""rows"": [{""id"": 1}, {""id"": 2}], ""total"": 104 } ``` The thing I care about most though is `next_url`. That could be provided like so: `.json?_extra=total&_extra=next` - alternative syntax `.json?_extra=total,next`: ```json { ""rows"": [{""id"": 1}, {""id"": 2}], ""total"": 104, ""next"": ""2"", ""next_url"": ""/db/table.json?_extra=total&_extra=next&_next=2"" } ``` This is feeling a bit verbose for a common combination though. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/782#issuecomment-706738020,https://api.github.com/repos/simonw/datasette/issues/782,706738020,MDEyOklzc3VlQ29tbWVudDcwNjczODAyMA==,9599,simonw,2020-10-11T17:23:18Z,2020-10-11T17:23:48Z,OWNER,"I'm going to prototype what it would look like if the default shape was a list of objects and `?_extra=` turns that into an object with a `rows` key, in a plugin. As a separate extension (maybe `.json-preview`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,