WorkingDirectory=/path/to/data
ExecStart=/path/to/datasette serve -h 0.0.0.0 ./my.db
Restart=on-failure
[Install]
WantedBy=multi-user.target
```
Activate it with:
```bash
$ sudo systemctl daemon-reload
$ sudo systemctl enable datasette
$ sudo systemctl start datasette
```
Logs are best viewed using `journalctl -u datasette -f`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459397625,Documentation with recommendations on running Datasette in production without using Docker,
https://github.com/simonw/datasette/pull/450#issuecomment-489342728,https://api.github.com/repos/simonw/datasette/issues/450,489342728,MDEyOklzc3VlQ29tbWVudDQ4OTM0MjcyOA==,45057,russss,2019-05-04T16:37:35Z,2019-05-04T16:37:35Z,CONTRIBUTOR,For a bit more context: this fixes a crash with `unsupported operand type(s) for +: 'int' and 'NoneType'` on the index page for me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440304714,Coalesce hidden table count to 0,
https://github.com/simonw/datasette/issues/446#issuecomment-489222223,https://api.github.com/repos/simonw/datasette/issues/446,489222223,MDEyOklzc3VlQ29tbWVudDQ4OTIyMjIyMw==,45057,russss,2019-05-03T20:01:19Z,2019-05-03T20:01:29Z,CONTRIBUTOR,"Also I have a slight preference against (ab)using `__slots__` to enforce fields, although I have done it myself in the past. It would be possible to do this with `__setattr__` instead, although that's an implementation detail and I'm not too fussed about it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440134714,Define mechanism for plugins to return structured data,
https://github.com/simonw/datasette/issues/446#issuecomment-489221481,https://api.github.com/repos/simonw/datasette/issues/446,489221481,MDEyOklzc3VlQ29tbWVudDQ4OTIyMTQ4MQ==,45057,russss,2019-05-03T19:58:31Z,2019-05-03T19:58:31Z,CONTRIBUTOR,"In this particular case I don't think there's an issue making all those required. However, I suspect we might have to allow optional values at some point - my preferred solution to russss/datasette-geo#2 would need one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440134714,Define mechanism for plugins to return structured data,
https://github.com/simonw/datasette/issues/419#issuecomment-489060765,https://api.github.com/repos/simonw/datasette/issues/419,489060765,MDEyOklzc3VlQ29tbWVudDQ4OTA2MDc2NQ==,45057,russss,2019-05-03T11:07:42Z,2019-05-03T11:07:42Z,CONTRIBUTOR,"Are you planning on removing inspect entirely?
I didn't spot this work before I started on datasette-geo, but ironically I think it has a use case which really needs the inspect functionality (or some replacement).
Datasette-geo uses it to store the bounding box of all the geographic features in the table. This is needed when rendering the map because it avoids having to send loads of tile requests for areas which are empty.
Even with relatively small datasets, calculating the bounding box seems to take around 5 seconds, so I don't think it's really feasible to do this on page load.
One possible fix would be to do this on startup, and then in a thread which watches the database for changes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",421551434,"Default to opening files in mutable mode, special option for immutable files",
https://github.com/simonw/datasette/pull/432#issuecomment-488595724,https://api.github.com/repos/simonw/datasette/issues/432,488595724,MDEyOklzc3VlQ29tbWVudDQ4ODU5NTcyNA==,45057,russss,2019-05-02T08:50:53Z,2019-05-02T08:50:53Z,CONTRIBUTOR,"> Can I pull those needs out of the Facet class somehow?
I was thinking that it might be handy for datasette to have a request object which wraps the Sanic Request. This could include the datasette-specific querystring decoding and the `special_args` parsing from TableView.data.
This would mean that we could expose the request object to plugin hooks without coupling them to Sanic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",432893491,"Refactor facets to a class and new plugin, refs #427",
https://github.com/simonw/datasette/pull/441#issuecomment-488247617,https://api.github.com/repos/simonw/datasette/issues/441,488247617,MDEyOklzc3VlQ29tbWVudDQ4ODI0NzYxNw==,45057,russss,2019-05-01T09:57:50Z,2019-05-01T09:57:50Z,CONTRIBUTOR,"Just for the record, this PR is now finished and ready to merge from my perspective.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",438437973,Add register_output_renderer hook,
https://github.com/simonw/datasette/pull/439#issuecomment-487859345,https://api.github.com/repos/simonw/datasette/issues/439,487859345,MDEyOklzc3VlQ29tbWVudDQ4Nzg1OTM0NQ==,45057,russss,2019-04-30T08:21:19Z,2019-04-30T08:21:19Z,CONTRIBUTOR,I think the best approach to this is to pass through the `view_name` parameter I added in #441. It's then simple enough for me to add `.geojson` to the URL in JS - I don't need the pkey.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",438240541,[WIP] Add primary key to the extra_body_script hook arguments,
https://github.com/simonw/datasette/pull/441#issuecomment-487748271,https://api.github.com/repos/simonw/datasette/issues/441,487748271,MDEyOklzc3VlQ29tbWVudDQ4Nzc0ODI3MQ==,45057,russss,2019-04-29T21:20:17Z,2019-04-29T21:20:17Z,CONTRIBUTOR,"Also I just pushed a change to add registered output renderers to the templates:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",438437973,Add register_output_renderer hook,
https://github.com/simonw/datasette/pull/441#issuecomment-487735247,https://api.github.com/repos/simonw/datasette/issues/441,487735247,MDEyOklzc3VlQ29tbWVudDQ4NzczNTI0Nw==,45057,russss,2019-04-29T20:39:43Z,2019-04-29T20:39:43Z,CONTRIBUTOR,"I updated the hook to pass the datasette object through now.
You can see the working [GeoJSON render function here](https://github.com/russss/datasette-geo/blob/master/datasette_plugin_geo/geojson.py) - the [hook function is here](https://github.com/russss/datasette-geo/blob/master/datasette_plugin_geo/__init__.py#L65-L70).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",438437973,Add register_output_renderer hook,
https://github.com/simonw/datasette/pull/441#issuecomment-487724539,https://api.github.com/repos/simonw/datasette/issues/441,487724539,MDEyOklzc3VlQ29tbWVudDQ4NzcyNDUzOQ==,45057,russss,2019-04-29T20:08:32Z,2019-04-29T20:08:32Z,CONTRIBUTOR,I also just realised that I should be passing the datasette object into the hook function...as I just found I need it. So hold off merging until I've fixed that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",438437973,Add register_output_renderer hook,
https://github.com/simonw/datasette/pull/441#issuecomment-487723476,https://api.github.com/repos/simonw/datasette/issues/441,487723476,MDEyOklzc3VlQ29tbWVudDQ4NzcyMzQ3Ng==,45057,russss,2019-04-29T20:05:23Z,2019-04-29T20:05:23Z,CONTRIBUTOR,"This is the minimal example (I also included it in the docs):
```python
from datasette import hookimpl
def render_test(args, data, view_name):
return {
'body': 'Hello World',
'content_type': 'text/plain'
}
@hookimpl
def register_output_renderer():
return {
'extension': 'test',
'callback': render_test
}
```
I'm working on the GeoJSON one now and it should be ready soon. (I forgot I was going to run into the same problem as before - that Spatialite's stupid binary format isn't WKB and I have no way of altering the query to change that - but I've just managed to write some code to rearrange the bytes from Spatialite blob-geometry into WKB...)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",438437973,Add register_output_renderer hook,
https://github.com/simonw/datasette/pull/424#issuecomment-487692377,https://api.github.com/repos/simonw/datasette/issues/424,487692377,MDEyOklzc3VlQ29tbWVudDQ4NzY5MjM3Nw==,45057,russss,2019-04-29T18:30:46Z,2019-04-29T18:30:46Z,CONTRIBUTOR,"Actually no, I ended up not using the inspected column types in my plugin, and the binary column issue can be solved a lot more simply, so I'll close this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",427429265,Column types in inspected metadata,
https://github.com/simonw/datasette/pull/424#issuecomment-487689477,https://api.github.com/repos/simonw/datasette/issues/424,487689477,MDEyOklzc3VlQ29tbWVudDQ4NzY4OTQ3Nw==,45057,russss,2019-04-29T18:22:40Z,2019-04-29T18:22:40Z,CONTRIBUTOR,This is pretty conflicty because I forgot how to use git fetch. If you're interested in merging this I'll rewrite it against an actual modern checkout...,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",427429265,Column types in inspected metadata,
https://github.com/simonw/datasette/pull/441#issuecomment-487686655,https://api.github.com/repos/simonw/datasette/issues/441,487686655,MDEyOklzc3VlQ29tbWVudDQ4NzY4NjY1NQ==,45057,russss,2019-04-29T18:14:25Z,2019-04-29T18:14:25Z,CONTRIBUTOR,"Subsidiary note which I forgot in the commit message:
I've decided to give each view a short string name to aid in differentiating which view a hook is being called from. Since hooks are functions and not subclasses, and can get called from different places in the URL hierarchy, it's sometimes difficult to distinguish what data you're actually operating on. I think this will come in handy for other hooks as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",438437973,Add register_output_renderer hook,
https://github.com/simonw/datasette/pull/439#issuecomment-487542486,https://api.github.com/repos/simonw/datasette/issues/439,487542486,MDEyOklzc3VlQ29tbWVudDQ4NzU0MjQ4Ng==,45057,russss,2019-04-29T11:20:30Z,2019-04-29T11:20:30Z,CONTRIBUTOR,Actually I think this is not the whole story because of the rowid issue. I'm going to think about this one a bit more.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",438240541,[WIP] Add primary key to the extra_body_script hook arguments,
https://github.com/simonw/datasette/pull/437#issuecomment-487537452,https://api.github.com/repos/simonw/datasette/issues/437,487537452,MDEyOklzc3VlQ29tbWVudDQ4NzUzNzQ1Mg==,45057,russss,2019-04-29T10:58:49Z,2019-04-29T10:58:49Z,CONTRIBUTOR,I've just spotted that this implements #215.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",438048318,Add inspect and prepare_sanic hooks,
https://github.com/simonw/datasette/issues/294#issuecomment-405026800,https://api.github.com/repos/simonw/datasette/issues/294,405026800,MDEyOklzc3VlQ29tbWVudDQwNTAyNjgwMA==,45057,russss,2018-07-14T14:24:31Z,2018-07-14T14:24:31Z,CONTRIBUTOR,"I had a quick look at this in relation to #343 and I feel like it might be worth modelling the inspected table metadata internally as an object rather than a dict. (We'd still have to serialise it back to JSON.)
There are a few places where we rely on the structure of this metadata dict for various reasons, including in templates (and potentially also in user templates). It would be nice to have a reasonably well defined API for accessing metadata internally so that it's clearer what we're breaking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",327365110,inspect should record column types,
https://github.com/simonw/datasette/issues/343#issuecomment-405026441,https://api.github.com/repos/simonw/datasette/issues/343,405026441,MDEyOklzc3VlQ29tbWVudDQwNTAyNjQ0MQ==,45057,russss,2018-07-14T14:17:14Z,2018-07-14T14:17:14Z,CONTRIBUTOR,This probably depends on #294.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",341228846,Render boolean fields better by default,
https://github.com/simonw/datasette/issues/344#issuecomment-405022335,https://api.github.com/repos/simonw/datasette/issues/344,405022335,MDEyOklzc3VlQ29tbWVudDQwNTAyMjMzNQ==,45057,russss,2018-07-14T13:00:48Z,2018-07-14T13:00:48Z,CONTRIBUTOR,"Looks like this was a red herring actually, and heroku had a blip when I was testing it...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",341229113,datasette publish heroku fails without name provided,
https://github.com/simonw/datasette/issues/276#issuecomment-401312981,https://api.github.com/repos/simonw/datasette/issues/276,401312981,MDEyOklzc3VlQ29tbWVudDQwMTMxMjk4MQ==,45057,russss,2018-06-29T10:14:54Z,2018-06-29T10:14:54Z,CONTRIBUTOR,"> @RusSs Different map projections can presumably be handled on the client side using a leaflet plugin to transform the geometry (eg kartena/Proj4Leaflet) although the leaflet side would need to detect or be informed of the original projection?
Well, as @simonw mentioned, GeoJSON only supports WGS84, and GeoJSON (and/or TopoJSON) is the standard we probably want to aim for. On-the-fly reprojection in spatialite is not an issue anyway, and in general I think you want to be serving stuff to web maps in WGS84 or Web Mercator.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,Handle spatialite geometry columns better,
https://github.com/simonw/datasette/issues/276#issuecomment-393106520,https://api.github.com/repos/simonw/datasette/issues/276,393106520,MDEyOklzc3VlQ29tbWVudDM5MzEwNjUyMA==,45057,russss,2018-05-30T10:09:25Z,2018-05-30T10:09:25Z,CONTRIBUTOR,"I don't think it's unreasonable to only support spatialite geometries in a coordinate reference system which is at least transformable to WGS84. It would be nice to support different CRSes in the database so conversion to spatialite from the source data is lossless.
I think the working CRS for datasette should be WGS84 though (leaflet requires it, for example) - it's just a case of calling `ST_Transform(geom, 4326)` on the column while we're loading the data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,Handle spatialite geometry columns better,
https://github.com/simonw/datasette/issues/276#issuecomment-392825746,https://api.github.com/repos/simonw/datasette/issues/276,392825746,MDEyOklzc3VlQ29tbWVudDM5MjgyNTc0Ng==,45057,russss,2018-05-29T15:42:53Z,2018-05-29T15:42:53Z,CONTRIBUTOR,"I haven't had time to look further into this, but if doing this as a plugin results in useful hooks then I think we should do it that way. We could always require the plugin as a standard dependency.
I think this is going to result in quite a bit of refactoring anyway so it's a good time to add hooks regardless.
On the other hand, if we have to add lots of specialist hooks for it then maybe it's worth integrating into the core.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,Handle spatialite geometry columns better,
https://github.com/simonw/datasette/issues/276#issuecomment-391505930,https://api.github.com/repos/simonw/datasette/issues/276,391505930,MDEyOklzc3VlQ29tbWVudDM5MTUwNTkzMA==,45057,russss,2018-05-23T21:41:37Z,2018-05-23T21:41:37Z,CONTRIBUTOR,"> I'm not keen on anything that modifies the SQLite file itself on startup
Ah I didn't mean that - I meant altering the SELECT query to fetch the data so that it ran a spatialite function to transform that specific column.
I think that's less useful as a general-purpose plugin hook though, and it's not that hard to parse the WKB in Python (my default approach would be to use [shapely](https://github.com/Toblerity/Shapely), which is great, but geomet looks like an interesting pure-python alternative).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,Handle spatialite geometry columns better,
https://github.com/simonw/datasette/issues/276#issuecomment-391050113,https://api.github.com/repos/simonw/datasette/issues/276,391050113,MDEyOklzc3VlQ29tbWVudDM5MTA1MDExMw==,45057,russss,2018-05-22T16:13:00Z,2018-05-22T16:13:00Z,CONTRIBUTOR,"Yup, I'll have a think about it. My current thoughts are for spatialite we'll need to hook into the following places:
* Inspection, so we can detect which columns are geometry columns. (We also currently ignore spatialite tables during inspection, it may be worth moving that to the plugin as well.)
* After data load, so we can convert WKB into the correct intermediate format for display. The alternative here is to alter the select SQL itself and get spatialite to do this conversion, but that strikes me as a bit more complex and possibly not as useful.
* HTML rendering.
* Querying?
The rendering and querying hooks could also potentially be used to move the units support into a plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,Handle spatialite geometry columns better,
https://github.com/simonw/datasette/issues/276#issuecomment-390795067,https://api.github.com/repos/simonw/datasette/issues/276,390795067,MDEyOklzc3VlQ29tbWVudDM5MDc5NTA2Nw==,45057,russss,2018-05-21T21:55:57Z,2018-05-21T21:55:57Z,CONTRIBUTOR,"Well, we do have the capability to detect spatialite so my intention certainly wasn't to require it.
I can see the advantage of having it as a plugin but it does touch a number of points in the code. I think I'm going to attack this by refactoring the necessary bits and seeing where that leads (which was my plan anyway).
I think my main concern is - if I add certain plugin hooks for this, is anything else ever going to use them? I'm not sure I have an answer to that question yet, either way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,Handle spatialite geometry columns better,
https://github.com/simonw/datasette/pull/209#issuecomment-381905593,https://api.github.com/repos/simonw/datasette/issues/209,381905593,MDEyOklzc3VlQ29tbWVudDM4MTkwNTU5Mw==,45057,russss,2018-04-17T08:50:28Z,2018-04-17T08:50:28Z,CONTRIBUTOR,"I've added another commit which puts classes a class on each `` by default with its column name, and I've also made the PK column bold.
Unfortunately the tests are still failing on 3.6, which is weird. I can't reproduce locally...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314455877, Don't duplicate simple primary keys in the link column,
https://github.com/simonw/datasette/issues/203#issuecomment-381763651,https://api.github.com/repos/simonw/datasette/issues/203,381763651,MDEyOklzc3VlQ29tbWVudDM4MTc2MzY1MQ==,45057,russss,2018-04-16T21:59:17Z,2018-04-16T21:59:17Z,CONTRIBUTOR,"Ah, I had no idea you could bind python functions into sqlite!
I think the primary purpose of this issue has been served now - I'm going to close this and create a new issue for the only bit of this that hasn't been touched yet, which is (optionally) exposing units in the JSON API.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",313837303,Support for units,
https://github.com/simonw/datasette/pull/209#issuecomment-381738137,https://api.github.com/repos/simonw/datasette/issues/209,381738137,MDEyOklzc3VlQ29tbWVudDM4MTczODEzNw==,45057,russss,2018-04-16T20:27:43Z,2018-04-16T20:27:43Z,CONTRIBUTOR,"Tests now fixed, honest. The failing test on Travis looks like an intermittent sqlite failure which should resolve itself on a retry...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314455877, Don't duplicate simple primary keys in the link column,
https://github.com/simonw/datasette/pull/209#issuecomment-381441392,https://api.github.com/repos/simonw/datasette/issues/209,381441392,MDEyOklzc3VlQ29tbWVudDM4MTQ0MTM5Mg==,45057,russss,2018-04-15T21:59:15Z,2018-04-15T21:59:15Z,CONTRIBUTOR,"I suspected this would cause some test failures, but I'll wait for opinions before attempting to fix them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314455877, Don't duplicate simple primary keys in the link column,
https://github.com/simonw/datasette/issues/125#issuecomment-381361734,https://api.github.com/repos/simonw/datasette/issues/125,381361734,MDEyOklzc3VlQ29tbWVudDM4MTM2MTczNA==,45057,russss,2018-04-14T21:26:30Z,2018-04-14T21:26:30Z,CONTRIBUTOR,"FWIW I am now doing this on my WTR app (instead of silently limiting maps to 1000).
[Telefonica](https://wtr-api.herokuapp.com/wtr-663ea99/licensee/18325) now has about 4000 markers and good old [BT](https://wtr-api.herokuapp.com/wtr-663ea99/licensee/8412) has 22,000 or so.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275135393,Plot rows on a map with Leaflet and Leaflet.markercluster,
https://github.com/simonw/datasette/pull/205#issuecomment-381332222,https://api.github.com/repos/simonw/datasette/issues/205,381332222,MDEyOklzc3VlQ29tbWVudDM4MTMzMjIyMg==,45057,russss,2018-04-14T14:16:35Z,2018-04-14T14:16:35Z,CONTRIBUTOR,I've added some tests and that docs link.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314319372,Support filtering with units and more,
https://github.com/simonw/datasette/issues/203#issuecomment-381315675,https://api.github.com/repos/simonw/datasette/issues/203,381315675,MDEyOklzc3VlQ29tbWVudDM4MTMxNTY3NQ==,45057,russss,2018-04-14T09:14:45Z,2018-04-14T09:27:30Z,CONTRIBUTOR,"> I'd like to figure out a sensible opt-in way to expose this in the JSON output as well. Maybe with a &_units=true parameter?
From a machine-readable perspective I'm not sure why it would be useful to decorate the values with units. Edit: Should have had some coffee first. It's clearly useful for stuff like map rendering!
I agree that the unit metadata should definitely be exposed in the JSON.
> In #204 you said ""I'd like to add support for using units when querying but this is PR is pretty usable as-is."" - I'm fascinated to hear more about how this could work.
I'm thinking about a couple of approaches here. I think the simplest one is: if the column has a unit attached, optionally accept units in query fields:
```python
column_units = ureg(""Hz"") # Create a unit object for the column's unit
query_variable = ureg(""4 GHz"") # Supplied query variable
# Now we can convert the query units into column units before querying
supplied_value.to(column_units).magnitude
> 4000000000.0
# If the user doesn't supply units, pint just returns the plain
# number and we can query as usual assuming it's the base unit
query_variable = ureg(""50"")
query_variable
> 50
isinstance(query_variable, numbers.Number)
> True
```
This also lets us do some nice unit conversion on querying:
```python
column_units = ureg(""m"")
query_variable = ureg(""50 ft"")
supplied_value.to(column_units)
>
```
The alternative would be to provide a dropdown of units next to the query field (so a ""Hz"" field would give you ""kHz"", ""MHz"", ""GHz""). Although this would be clearer to the user, it isn't so easy - we'd need to know more about the context of the field to give you sensible SI prefixes (I'm not so interested in nanoHertz, for example).
You also lose the bonus of being able to convert - although pint will happily show you all the compatible units, it again suffers from a lack of context:
```python
ureg(""m"").compatible_units()
> frozenset({,
,
,
,
,
,
,
,
,
,
,
})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",313837303,Support for units,
https://github.com/simonw/datasette/pull/202#issuecomment-381237440,https://api.github.com/repos/simonw/datasette/issues/202,381237440,MDEyOklzc3VlQ29tbWVudDM4MTIzNzQ0MA==,45057,russss,2018-04-13T19:22:53Z,2018-04-13T19:22:53Z,CONTRIBUTOR,I spotted you'd mentioned that in #184 but only after I'd written the patch!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",313785206,Raise 404 on nonexistent table URLs,
https://github.com/simonw/datasette/issues/203#issuecomment-380966565,https://api.github.com/repos/simonw/datasette/issues/203,380966565,MDEyOklzc3VlQ29tbWVudDM4MDk2NjU2NQ==,45057,russss,2018-04-12T22:43:08Z,2018-04-12T22:43:08Z,CONTRIBUTOR,"Looks like [pint](https://pint.readthedocs.io/en/latest/tutorial.html) is pretty good at this.
```python
In [1]: import pint
In [2]: ureg = pint.UnitRegistry()
In [3]: q = 3e6 * ureg('Hz')
In [4]: '{:~P}'.format(q.to_compact())
Out[4]: '3.0 MHz'
In [5]: q = 0.3 * ureg('m')
In [5]: '{:~P}'.format(q.to_compact())
Out[5]: '300.0 mm'
In [6]: q = 5 * ureg('')
In [7]: '{:~P}'.format(q.to_compact())
Out[7]: '5'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",313837303,Support for units,
https://github.com/simonw/datasette/pull/200#issuecomment-380608372,https://api.github.com/repos/simonw/datasette/issues/200,380608372,MDEyOklzc3VlQ29tbWVudDM4MDYwODM3Mg==,45057,russss,2018-04-11T21:55:46Z,2018-04-11T21:55:46Z,CONTRIBUTOR,"> I think the most reliable way to detect spatialite is to run `SELECT AddGeometryColumn(1, 2, 3, 4, 5);` against a `:memory:` database and see if it throws an exception
Or just see if there's a `geometry_columns` table? I think that's quite unlikely to be added by accident (and it's an OGC standard). It also tells you if Spatialite is installed in the database rather than just loaded.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",313494458,Hide Spatialite system tables,
https://github.com/dogsheep/twitter-to-sqlite/issues/58#issuecomment-910121331,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/58,910121331,IC_kwDODEm0Qs42P1lz,42904,rubenv,2021-09-01T09:49:33Z,2021-09-01T09:49:33Z,CONTRIBUTOR,"Found the cause, it's the other commands. PR #59 submitted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",984939366,"Error: Use either --since or --since_id, not both - still broken",
https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717359145,https://api.github.com/repos/simonw/sqlite-utils/issues/189,717359145,MDEyOklzc3VlQ29tbWVudDcxNzM1OTE0NQ==,35681,adamwolf,2020-10-27T16:20:32Z,2020-10-27T16:20:32Z,CONTRIBUTOR,"No problem. I added a test. Let me know if it looks sufficient or if you want me to to tweak something!
If you don't mind, would you tag this PR as ""hacktoberfest-accepted""? If you do mind, no problem and I'm sorry for asking :) My kiddos like the shirts.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records,
https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1501017004,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1501017004,IC_kwDOCGYnMM5Zd7Os,25778,eyeseast,2023-04-09T01:49:43Z,2023-04-09T01:49:43Z,CONTRIBUTOR,I'm going to close this in favor of #536. Will try a cleaner approach to custom paths once that one is merge.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon,
https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465315726,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1465315726,IC_kwDOCGYnMM5XVvGO,25778,eyeseast,2023-03-12T22:21:56Z,2023-03-12T22:21:56Z,CONTRIBUTOR,"Exactly, that's what I was running into. On my M2 MacBook, SpatiaLite ends up in what is -- for the moment -- a non-standard location, so even when I passed in the location with `--load-extension`, I still hit an error on `create-spatial-index`.
What I learned doing this originally is that SQLite needs to load the extension for each connection, even if all the SpatiaLite stuff is already in the database. So that's why `init_spatialite()` gets called again.
Here's the code where I hit the error: https://github.com/eyeseast/boston-parcels/blob/main/Makefile#L30 It works using this branch.
I'm not attached to this solution if you can think of something better. And I'm not sure, TBH, my test would actually catch what I'm after here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon,
https://github.com/simonw/datasette/issues/2033#issuecomment-1457172180,https://api.github.com/repos/simonw/datasette/issues/2033,1457172180,IC_kwDOBm6k_c5W2q7U,25778,eyeseast,2023-03-06T22:54:52Z,2023-03-06T22:54:52Z,CONTRIBUTOR,This would be a nice feature to have with `datasette publish` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1612296210,`datasette install -r requirements.txt`,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419357290,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1419357290,IC_kwDOCGYnMM5Umaxq,25778,eyeseast,2023-02-06T16:21:44Z,2023-02-06T16:21:44Z,CONTRIBUTOR,SQLite doesn't have a native `DATETIME` type. It stores dates internally as strings and then has [functions](https://www.sqlite.org/lang_datefunc.html) to work with date-like strings. Yes it's weird.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,
https://github.com/simonw/datasette/issues/1983#issuecomment-1375810027,https://api.github.com/repos/simonw/datasette/issues/1983,1375810027,IC_kwDOBm6k_c5SATHr,25778,eyeseast,2023-01-09T15:35:58Z,2023-01-09T15:35:58Z,CONTRIBUTOR,"Yes please, and thank you. I realized I was maybe getting myself in trouble using that, but I think it's a good way to standardize JSON handling.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1525815985,Make CustomJSONEncoder a documented public API,
https://github.com/simonw/datasette/issues/1978#issuecomment-1375708725,https://api.github.com/repos/simonw/datasette/issues/1978,1375708725,IC_kwDOBm6k_c5R_6Y1,25778,eyeseast,2023-01-09T14:30:00Z,2023-01-09T14:30:00Z,CONTRIBUTOR,Totally missed that issue. I can close this as a duplicate.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1522778923,Document datasette.urls.row and row_blob,
https://github.com/simonw/datasette/issues/1605#issuecomment-1332310772,https://api.github.com/repos/simonw/datasette/issues/1605,1332310772,IC_kwDOBm6k_c5PaXL0,25778,eyeseast,2022-11-30T15:06:37Z,2022-11-30T15:06:37Z,CONTRIBUTOR,I'll add issues for both and do a documentation PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports,
https://github.com/simonw/datasette/issues/1605#issuecomment-1331187551,https://api.github.com/repos/simonw/datasette/issues/1605,1331187551,IC_kwDOBm6k_c5PWE9f,25778,eyeseast,2022-11-29T19:29:42Z,2022-11-29T19:29:42Z,CONTRIBUTOR,"Interesting. I started a version using metadata like I outlined up top, but I realized that there's no documented way for a plugin to access either metadata or canned queries. Or at least, I couldn't find a way.
There is this method: https://github.com/simonw/datasette/blob/main/datasette/app.py#L472 but I don't want to rely on it if it's not documented. Same with this: https://github.com/simonw/datasette/blob/main/datasette/app.py#L544
If those are safe, I'll build on them. I'm also happy to document them, if that greases the wheels.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports,
https://github.com/simonw/datasette/issues/1886#issuecomment-1314241058,https://api.github.com/repos/simonw/datasette/issues/1886,1314241058,IC_kwDOBm6k_c5OVboi,25778,eyeseast,2022-11-14T19:06:35Z,2022-11-14T19:06:35Z,CONTRIBUTOR,"This probably counts as a case study: https://github.com/eyeseast/spatial-data-cooking-show. Even has video.
Seriously, though, this workflow has become integral to my work with reporters and editors across USA TODAY Network. Very often, I get sent a folder of data in mixed formats, with a vague ask of how we should communicate some part of it to users. Datasette and its constellation of tools makes it easy to get a quick look at that data, run exploratory queries, map it and ask questions to figure out what's important to show. And then I export a version of the data that's exactly what I need for display.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here",
https://github.com/simonw/datasette/issues/1884#issuecomment-1314066229,https://api.github.com/repos/simonw/datasette/issues/1884,1314066229,IC_kwDOBm6k_c5OUw81,25778,eyeseast,2022-11-14T16:48:35Z,2022-11-14T16:48:35Z,CONTRIBUTOR,"I'm realizing I don't know if a virtual table will ever return a count. Maybe it depends on the implementation. For these three, just checking now, it'll always return zero.
That said, I'm not sure there's any downside to having them return zero and caching that. (They're hidden, too.) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect,
https://github.com/simonw/datasette/issues/1884#issuecomment-1313962183,https://api.github.com/repos/simonw/datasette/issues/1884,1313962183,IC_kwDOBm6k_c5OUXjH,25778,eyeseast,2022-11-14T15:46:32Z,2022-11-14T15:46:32Z,CONTRIBUTOR,"It does work, though I think it's probably still worth excluding virtual tables that will always be zero. Here's the same inspection as before, now with `--load-extension spatialite`:
```json
{
""alltheplaces"": {
""hash"": ""0843cfe414439ab903c22d1121b7ddbc643418c35c7f0edbcec82ef1452411df"",
""size"": 963375104,
""file"": ""alltheplaces.db"",
""tables"": {
""spatial_ref_sys"": {
""count"": 6215
},
""spatialite_history"": {
""count"": 18
},
""sqlite_sequence"": {
""count"": 2
},
""geometry_columns"": {
""count"": 3
},
""spatial_ref_sys_aux"": {
""count"": 6164
},
""views_geometry_columns"": {
""count"": 0
},
""virts_geometry_columns"": {
""count"": 0
},
""geometry_columns_statistics"": {
""count"": 3
},
""views_geometry_columns_statistics"": {
""count"": 0
},
""virts_geometry_columns_statistics"": {
""count"": 0
},
""geometry_columns_field_infos"": {
""count"": 0
},
""views_geometry_columns_field_infos"": {
""count"": 0
},
""virts_geometry_columns_field_infos"": {
""count"": 0
},
""geometry_columns_time"": {
""count"": 3
},
""geometry_columns_auth"": {
""count"": 3
},
""views_geometry_columns_auth"": {
""count"": 0
},
""virts_geometry_columns_auth"": {
""count"": 0
},
""data_licenses"": {
""count"": 10
},
""sql_statements_log"": {
""count"": 0
},
""states"": {
""count"": 56
},
""counties"": {
""count"": 3234
},
""idx_states_geometry_rowid"": {
""count"": 56
},
""idx_states_geometry_node"": {
""count"": 3
},
""idx_states_geometry_parent"": {
""count"": 2
},
""idx_counties_geometry_rowid"": {
""count"": 3234
},
""idx_counties_geometry_node"": {
""count"": 98
},
""idx_counties_geometry_parent"": {
""count"": 97
},
""idx_places_geometry_rowid"": {
""count"": 1236796
},
""idx_places_geometry_node"": {
""count"": 38163
},
""idx_places_geometry_parent"": {
""count"": 38162
},
""places"": {
""count"": 1332609
},
""SpatialIndex"": {
""count"": 0
},
""ElementaryGeometries"": {
""count"": 0
},
""KNN"": {
""count"": 0
},
""idx_states_geometry"": {
""count"": 56
},
""idx_counties_geometry"": {
""count"": 3234
},
""idx_places_geometry"": {
""count"": 1236796
}
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect,
https://github.com/simonw/datasette/issues/1884#issuecomment-1309735529,https://api.github.com/repos/simonw/datasette/issues/1884,1309735529,IC_kwDOBm6k_c5OEPpp,25778,eyeseast,2022-11-10T03:57:23Z,2022-11-10T03:57:23Z,CONTRIBUTOR,Here's how to get a list of virtual tables: https://stackoverflow.com/questions/46617118/how-to-fetch-names-of-virtual-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect,
https://github.com/simonw/datasette/issues/1851#issuecomment-1292592210,https://api.github.com/repos/simonw/datasette/issues/1851,1292592210,IC_kwDOBm6k_c5NC2RS,25778,eyeseast,2022-10-26T20:03:46Z,2022-10-26T20:03:46Z,CONTRIBUTOR,"Yeah, every time I see something cool done with triggers, I remember that I need to start using triggers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table,
https://github.com/simonw/datasette/issues/1851#issuecomment-1291228502,https://api.github.com/repos/simonw/datasette/issues/1851,1291228502,IC_kwDOBm6k_c5M9pVW,25778,eyeseast,2022-10-25T23:02:10Z,2022-10-25T23:02:10Z,CONTRIBUTOR,That's reasonable. Canned queries and custom endpoints are certainly going to give more room for specific needs. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table,
https://github.com/simonw/datasette/issues/1851#issuecomment-1290615599,https://api.github.com/repos/simonw/datasette/issues/1851,1290615599,IC_kwDOBm6k_c5M7Tsv,25778,eyeseast,2022-10-25T14:05:12Z,2022-10-25T14:05:12Z,CONTRIBUTOR,"This could use a new plugin hook, too. I don't want to complicate your life too much, but for things like GIS, I'd want a way to turn regular JSON into SpatiaLite geometries or combine X/Y coordinates into point geometries and such. Happy to help however I can.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table,
https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258712931,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1258712931,IC_kwDOCGYnMM5LBm9j,25778,eyeseast,2022-09-26T22:31:58Z,2022-09-26T22:31:58Z,CONTRIBUTOR,"Right. The backup command will copy tables completely, but in the case of conflicting table names, the destination gets overwritten silently. That might not be what you want here. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables,
https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258508215,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1258508215,IC_kwDOCGYnMM5LA0-3,25778,eyeseast,2022-09-26T19:22:14Z,2022-09-26T19:22:14Z,CONTRIBUTOR,"This might be fairly straightforward using SQLite's backup utility: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.backup
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables,
https://github.com/simonw/datasette/issues/1528#issuecomment-1151887842,https://api.github.com/repos/simonw/datasette/issues/1528,1151887842,IC_kwDOBm6k_c5EqGni,25778,eyeseast,2022-06-10T03:23:08Z,2022-06-10T03:23:08Z,CONTRIBUTOR,I just put together a version of this in a plugin: https://github.com/eyeseast/datasette-query-files. Happy to have any feedback.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1060631257,"Add new `""sql_file""` key to Canned Queries in metadata?",
https://github.com/simonw/datasette/issues/1742#issuecomment-1128064864,https://api.github.com/repos/simonw/datasette/issues/1742,1128064864,IC_kwDOBm6k_c5DPOdg,25778,eyeseast,2022-05-16T19:42:13Z,2022-05-16T19:42:13Z,CONTRIBUTOR,"Just to add a wrinkle here, this loads fine: https://alltheplaces-datasette.fly.dev/alltheplaces/places.geojson?_trace=1
But also, this doesn't add any trace data: https://alltheplaces-datasette.fly.dev/alltheplaces/places.json?_trace=1
What am I missing?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1237586379,?_trace=1 fails with datasette-geojson for some reason,
https://github.com/simonw/datasette/issues/1742#issuecomment-1128049716,https://api.github.com/repos/simonw/datasette/issues/1742,1128049716,IC_kwDOBm6k_c5DPKw0,25778,eyeseast,2022-05-16T19:24:44Z,2022-05-16T19:24:44Z,CONTRIBUTOR,"Where is `_trace` getting injected? And is it something a plugin should be able to handle? (If it is, I guess I should handle it in this case.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1237586379,?_trace=1 fails with datasette-geojson for some reason,
https://github.com/simonw/datasette/issues/741#issuecomment-1125342229,https://api.github.com/repos/simonw/datasette/issues/741,1125342229,IC_kwDOBm6k_c5DE1wV,25778,eyeseast,2022-05-12T19:21:16Z,2022-05-12T19:21:16Z,CONTRIBUTOR,"Came here to check if this had been flagged already. Was helping a colleague get something on Cloud Run and had to dig to find `--extra-options=""--setting sql_time_limit_ms 2500""`.
If I get some time next week, maybe I'll try to tackle it. Would definitely make things easier to be able to do something like this:
```sh
datasette publish cloudrun something.db --setting sql_time_limit_ms 2500
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/1101#issuecomment-1105642187,https://api.github.com/repos/simonw/datasette/issues/1101,1105642187,IC_kwDOBm6k_c5B5sLL,25778,eyeseast,2022-04-21T18:59:08Z,2022-04-21T18:59:08Z,CONTRIBUTOR,"Ha! That was your idea (and a good one).
But it's probably worth measuring to see what overhead it adds. It did require both passing in the database and making the whole thing `async`.
Just timing the queries themselves:
1. [Using `AsGeoJSON(geometry) as geometry`](https://alltheplaces-datasette.fly.dev/alltheplaces?sql=select%0D%0A++id%2C%0D%0A++properties%2C%0D%0A++AsGeoJSON%28geometry%29+as+geometry%2C%0D%0A++spider%0D%0Afrom%0D%0A++places%0D%0Aorder+by%0D%0A++id%0D%0Alimit%0D%0A++1000) takes 10.235 ms
2. [Leaving as binary](https://alltheplaces-datasette.fly.dev/alltheplaces?sql=select%0D%0A++id%2C%0D%0A++properties%2C%0D%0A++geometry%2C%0D%0A++spider%0D%0Afrom%0D%0A++places%0D%0Aorder+by%0D%0A++id%0D%0Alimit%0D%0A++1000) takes 8.63 ms
Looking at the network panel:
1. Takes about 200 ms for the `fetch` request
2. Takes about 300 ms
I'm not sure how best to time the GeoJSON generation, but it would be interesting to check. Maybe I'll write a plugin to add query times to response headers.
The other thing to consider with async streaming is that it might be well-suited for a slower response. When I have to get the whole result and send a response in a fixed amount of time, I need the most efficient query possible. If I can hang onto a connection and get things one chunk at a time, maybe it's ok if there's some overhead.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1101#issuecomment-1105588651,https://api.github.com/repos/simonw/datasette/issues/1101,1105588651,IC_kwDOBm6k_c5B5fGr,25778,eyeseast,2022-04-21T18:15:39Z,2022-04-21T18:15:39Z,CONTRIBUTOR,"What if you split rendering and streaming into two things:
- `render` is a function that returns a response
- `stream` is a function that sends chunks, or yields chunks passed to an ASGI `send` callback
That way current plugins still work, and streaming is purely additive. A `stream` function could get a cursor or iterator of rows, instead of a list, so it could more efficiently handle large queries.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1713#issuecomment-1099540225,https://api.github.com/repos/simonw/datasette/issues/1713,1099540225,IC_kwDOBm6k_c5BiacB,25778,eyeseast,2022-04-14T19:09:57Z,2022-04-14T19:09:57Z,CONTRIBUTOR,"I wonder if this overlaps with what I outlined in #1605. You could run something like this:
```sh
datasette freeze -d exports/
aws s3 cp exports/ s3://my-export-bucket/$(date)
```
And maybe that does what you need. Of course, that plugin isn't built yet. But that's the idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203943272,Datasette feature for publishing snapshots of query results,
https://github.com/simonw/datasette/issues/1699#issuecomment-1094453751,https://api.github.com/repos/simonw/datasette/issues/1699,1094453751,IC_kwDOBm6k_c5BPAn3,25778,eyeseast,2022-04-11T01:32:12Z,2022-04-11T01:32:12Z,CONTRIBUTOR,"Was looking through old issues and realized a bunch of this got discussed in #1101 (including by me!), so sorry to rehash all this. Happy to help with whatever piece of it I can. Would be very excited to be able to use format plugins with exports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,Proposal: datasette query,
https://github.com/simonw/datasette/issues/1699#issuecomment-1092386254,https://api.github.com/repos/simonw/datasette/issues/1699,1092386254,IC_kwDOBm6k_c5BHH3O,25778,eyeseast,2022-04-08T02:39:25Z,2022-04-08T02:39:25Z,CONTRIBUTOR,"And just to think this through a little more, here's what `stream_geojson` might look like:
```python
async def stream_geojson(datasette, columns, rows, database, stream):
db = datasette.get_database(database)
for row in rows:
feature = await row_to_geojson(row, db)
stream.write(feature + ""\n"") # just assuming newline mode for now
```
Alternately, that could be an async generator, like this:
```python
async def stream_geojson(datasette, columns, rows, database):
db = datasette.get_database(database)
for row in rows:
feature = await row_to_geojson(row, db)
yield feature
```
Not sure which makes more sense, but I think this pattern would open up a lot of possibility. If you had your [stream_indented_json](https://til.simonwillison.net/python/output-json-array-streaming) function, you could do `yield from stream_indented_json(rows, 2)` and be one your way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,Proposal: datasette query,
https://github.com/simonw/datasette/issues/1699#issuecomment-1092370880,https://api.github.com/repos/simonw/datasette/issues/1699,1092370880,IC_kwDOBm6k_c5BHEHA,25778,eyeseast,2022-04-08T02:07:40Z,2022-04-08T02:07:40Z,CONTRIBUTOR,"So maybe `render_output_render` returns something like this:
```python
@hookimpl
def register_output_renderer(datasette):
return {
""extension"": ""geojson"",
""render"": render_geojson,
""stream"": stream_geojson,
""can_render"": can_render_geojson,
}
```
And stream gets an iterator, instead of a list of rows, so it can efficiently handle large queries. Maybe it also gets passed a destination stream, or it returns an iterator. I'm not sure what makes more sense. Either way, that might cover both CLI exports and streaming responses.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,Proposal: datasette query,
https://github.com/simonw/datasette/issues/1699#issuecomment-1092357672,https://api.github.com/repos/simonw/datasette/issues/1699,1092357672,IC_kwDOBm6k_c5BHA4o,25778,eyeseast,2022-04-08T01:39:40Z,2022-04-08T01:39:40Z,CONTRIBUTOR,"> My best thought on how to differentiate them so far is plugins: if Datasette plugins that provide alternative outputs - like .geojson and .yml and suchlike - also work for the datasette query command that would make a lot of sense to me.
That's my thinking, too. It's really the thing I've been wanting since writing `datasette-geojson`, since I'm always exporting with `datasette --get`. The workflow I'm always looking for is something like this:
```sh
cd alltheplaces-datasette
datasette query dunkin_in_suffolk -f geojson -o dunkin_in_suffolk.geojson
```
I think this probably needs either a new plugin hook separate from `register_output_renderer` or a way to use that without going through the HTTP stack. Or maybe a render mode that writes to a stream instead of a response. Maybe there's a new key in the dictionary that `register_output_renderer` returns that handles CLI exports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,Proposal: datasette query,
https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1077671779,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1077671779,IC_kwDOCGYnMM5AO_dj,25778,eyeseast,2022-03-24T14:11:33Z,2022-03-24T14:11:43Z,CONTRIBUTOR,"Coming back to this. I was about to add a utility function to [datasette-geojson]() to convert lat/lng columns to geometries. Thankfully I googled first. There's a SpatiaLite function for this: [MakePoint](https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html#p0).
```sql
select MakePoint(longitude, latitude) as geometry from places;
```
I'm not sure if that would work with `conversions`, since it needs two columns, but it's an option for tables that already have latitude, longitude columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code",
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-1067981656,https://api.github.com/repos/simonw/sqlite-utils/issues/131,1067981656,IC_kwDOCGYnMM4_qBtY,25778,eyeseast,2022-03-15T13:21:42Z,2022-03-15T13:21:42Z,CONTRIBUTOR,"Just ran into this issue last night. I have a big table that's _mostly_ numbers, but also a zip code column in a state where ZIP codes start with 0. Would be great to run something like this:
```sh
sqlite-utils insert data.db places file.csv --csv --detect-types --type zipcode text
```
Maybe I'll take a crack at this one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042,sqlite-utils insert: options for column types,
https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065477258,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065477258,IC_kwDOCGYnMM4_geSK,25778,eyeseast,2022-03-11T20:14:59Z,2022-03-11T20:14:59Z,CONTRIBUTOR,"Good call on adding this to `create-table`, especially for stored columns. Having the stored/virtual split might make this tricky to implement, but I haven't gone any farther than thinking about what the CLI looks like. I'm going to try making the SQL side work first and figure that'll tell me more about what it needs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns,
https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059647114,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059647114,IC_kwDOCGYnMM4_KO6K,25778,eyeseast,2022-03-05T01:54:24Z,2022-03-05T01:54:24Z,CONTRIBUTOR,"I haven't tried this, but it looks like Pandas has a method for this: https://pandas.pydata.org/docs/reference/api/pandas.read_sql_query.html
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration,
https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040998433,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040998433,IC_kwDOCGYnMM4-DGAh,25778,eyeseast,2022-02-16T01:29:39Z,2022-02-16T01:29:39Z,CONTRIBUTOR,Happy to do it and have it in the library. Going to use it a bunch. This whole SpatiaLite toolchain become a huge part of my work in the past year.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI,
https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040580250,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040580250,IC_kwDOCGYnMM4-Bf6a,25778,eyeseast,2022-02-15T17:40:00Z,2022-02-15T17:40:00Z,CONTRIBUTOR,@simonw I think this is ready for a look.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI,
https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1038336591,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1038336591,IC_kwDOCGYnMM4948JP,25778,eyeseast,2022-02-13T18:48:21Z,2022-02-13T18:49:49Z,CONTRIBUTOR,"Been chipping away at this between other things and realized `sqlite-utils init-spatialite` is probably unnecessary. Any of the other commands requires running `db.init_spatialite` to have the extension functions available, and that will do everything `init-spatialite` would do.
I think it's probably worth keeping a SpatiaLite flag on `create-database` in case you wanted to create all the spatial metadata up front. Otherwise, it's going to get added the first time you run `add-geometry-column` or `create-spatial-index`, which is probably fine in most cases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI,
https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1035057014,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1035057014,IC_kwDOCGYnMM49sbd2,25778,eyeseast,2022-02-10T15:30:28Z,2022-02-10T15:30:40Z,CONTRIBUTOR,"Yeah, the CLI experience is probably where any kind of multi-column, configured setup is going to fall apart. Sticking with GIS examples, one way I might think about this is using the [fiona CLI](https://fiona.readthedocs.io/en/latest/cli.html):
```sh
# assuming a database is already created and has SpatiaLite
fio cat boundary.shp | sqlite-utils insert boundaries --conversion geometry GeometryGeoJSON -
```
Anyway, very interested to see where you land here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism,
https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032732242,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1032732242,IC_kwDOCGYnMM49jj5S,25778,eyeseast,2022-02-08T15:26:59Z,2022-02-08T15:26:59Z,CONTRIBUTOR,"What if you did something like this:
```python
class Conversion:
def __init__(self, *args, **kwargs):
""Put whatever settings you need here""
def python(self, row, column, value): # not sure on args here
""Python step to transform value""
return value
def sql(self, row, column, value):
""Return the actual sql that goes in the insert/update step, and maybe params""
# value is the return of self.python()
return value, []
```
This way, you're always passing an instance, which has methods that do the conversion. (Or you're passing a SQL string, as you would now.) The `__init__` could take column names, or SRID, or whatever other setup state you need per row, but the row is getting processed with the `python` and `sql` methods (or whatever you want to call them). This is pretty rough, so do what you will with names and args and such.
You'd then use it like this:
```python
# subclass might be unneeded here, if methods are present
class LngLatConversion(Conversion):
def __init__(self, x=""longitude"", y=""latitude""):
self.x = x
self.y = y
def python(self, row, column, value):
x = row[self.x]
y = row[self.y]
return x, y
def sql(self, row, column, value):
# value is now a tuple, returned above
s = ""GeomFromText(POINT(? ?))""
return s, value
table.insert_all(rows, conversions={""point"": LngLatConversion(""lng"", ""lat""))}
```
I haven't thought through all the implementation details here, and it'll probably break in ways I haven't foreseen, but wanted to get this idea out of my head. Hope it helps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism,
https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031791783,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1031791783,IC_kwDOCGYnMM49f-Sn,25778,eyeseast,2022-02-07T18:37:40Z,2022-02-07T18:37:40Z,CONTRIBUTOR,"I've never used it either, but it's interesting, right? Feel like I should try it for something.
I'm trying to get my head around how this conversions feature might work, because I really like the idea of it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism,
https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031779460,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1031779460,IC_kwDOCGYnMM49f7SE,25778,eyeseast,2022-02-07T18:24:56Z,2022-02-07T18:24:56Z,CONTRIBUTOR,"I wonder if there's any overlap with the goals here and the `sqlite3` module's concept of adapters and converters: https://docs.python.org/3/library/sqlite3.html#sqlite-and-python-types
I'm not sure that's _exactly_ what we're talking about here, but it might be a parallel with some useful ideas to borrow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism,
https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030741289,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030741289,IC_kwDOCGYnMM49b90p,25778,eyeseast,2022-02-06T03:03:43Z,2022-02-06T03:03:43Z,CONTRIBUTOR,"> I wonder if there are any interesting non-geospatial canned conversions that it would be worth including?
Off the top of my head:
- Un-nesting JSON objects into columns
- Splitting arrays
- Normalizing dates and times
- URL munging with `urlparse`
- Converting strings to numbers
Some of this is easy enough with SQL functions, some is easier in Python. Maybe that's where having pre-built classes gets really handy, because it saves you from thinking about which way it's implemented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code",
https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740826,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740826,IC_kwDOCGYnMM49b9ta,25778,eyeseast,2022-02-06T02:59:10Z,2022-02-06T02:59:10Z,CONTRIBUTOR,"All this said, I don't think it's unreasonable to point people to dedicated tools like `geojson-to-sqlite`. If I'm dealing with a bunch of GeoJSON or Shapefiles, I need to something to read those anyway (or I need to figure out virtual tables). But something like this might make it easier to build those libraries, or standardize the underlying parts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code",
https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740653,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740653,IC_kwDOCGYnMM49b9qt,25778,eyeseast,2022-02-06T02:57:17Z,2022-02-06T02:57:17Z,CONTRIBUTOR,"I like the idea of having stock conversions you could import. I'd actually move them to a dedicated module (call it `sqlite_utils.conversions` or something), because it's different from other utilities. Maybe they even take configuration, or they're composable.
```python
from sqlite_utils.conversions import LongitudeLatitude
db[""places""].insert(
{
""name"": ""London"",
""lng"": -0.118092,
""lat"": 51.509865,
},
conversions={""point"": LongitudeLatitude(""lng"", ""lat"")},
)
```
I would definitely use that for every CSV I get with lat/lng columns where I actually need GeoJSON.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code",
https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030629879,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030629879,IC_kwDOCGYnMM49bin3,25778,eyeseast,2022-02-05T13:57:33Z,2022-02-05T19:49:38Z,CONTRIBUTOR,"I'm mostly using [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite) at the moment. Even with shapefiles, I'm usually converting to GeoJSON and projecting to EPSG:4326 (with [ogr2ogr](https://gdal.org/programs/ogr2ogr.html)) first.
I think an open question here is how much you want to leave to external libraries and how much you want here. My thinking has been that adding Spatialite helpers here would make external stuff easier, but it would be nice to have some standard way to insert geometries.
I'm in the middle of adding GeoJSON and Spatialite support to [geocode-sqlite](https://github.com/eyeseast/geocode-sqlite), and that will probably use WKT. Since that's all points, I think I can just make the string inline. But for polygons, I'd generally use Shapely, which probably isn't a dependency you want to add to sqlite-utils.
I've also been trying to get some of the approaches [here](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=Supporting+GeoJSON) to work, but haven't had any success so far.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1030002502,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1030002502,IC_kwDOCGYnMM49ZJdG,25778,eyeseast,2022-02-04T13:50:19Z,2022-02-04T13:50:19Z,CONTRIBUTOR,Awesome. Thanks for your help getting it in. Will now look at adding CLI versions of this. It's going to be super helpful on a bunch of my projects.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029370537,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029370537,IC_kwDOCGYnMM49WvKp,25778,eyeseast,2022-02-03T20:25:58Z,2022-02-03T20:25:58Z,CONTRIBUTOR,"OK, I moved all the GIS helpers into `db.py` as methods on `Database` and `Table`, and I put `find_spatialite` back in `utils.py`. I deleted `gis.py`, since there's nothing left it. Docs and tests are updated and passing.
I think this is better.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029338360,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029338360,IC_kwDOCGYnMM49WnT4,25778,eyeseast,2022-02-03T19:43:56Z,2022-02-03T19:43:56Z,CONTRIBUTOR,"Works for me. I was just looking at how the FTS extensions work and they're just methods, too. So this can be consistent with that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029326568,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029326568,IC_kwDOCGYnMM49Wkbo,25778,eyeseast,2022-02-03T19:28:26Z,2022-02-03T19:28:26Z,CONTRIBUTOR,"> `from sqlite_utils.utils import find_spatialite` is part of the documented API already:
>
> https://sqlite-utils.datasette.io/en/3.22.1/python-api.html#finding-spatialite
>
> To avoid needing to bump the major version number to 4 to indicate a backwards incompatible change, we should keep a `from .gis import find_spatialite` line at the top of `utils.py` such that any existing code with that documented import continues to work.
This is fixed now. I had to take out the type annotations for `Database` and `Table` to avoid a circular import, but that's fine and may be moot if these become class methods.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029317527,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029317527,IC_kwDOCGYnMM49WiOX,25778,eyeseast,2022-02-03T19:18:02Z,2022-02-03T19:18:02Z,CONTRIBUTOR,"Taking part of the conversation from #385 here.
> Would sqlite-utils add-geometry-column ... be a good CLI enhancement. for example?
Yes. And also `sqlite-utils create-spatial-index` would be great to have. My plan would be to add those once the Python API is settled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029306428,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029306428,IC_kwDOCGYnMM49Wfg8,25778,eyeseast,2022-02-03T19:03:43Z,2022-02-03T19:03:43Z,CONTRIBUTOR,"I thought about adding these as methods on `Database` and `Table`, and I'm back and forth on it for the same reasons you are. It's certainly cleaner, and it's clearer what you're operating on. I could go either way.
I do sort of like having all the Spatialite stuff in its own module, just because it's built around an extension you might not have or want, but I don't know if that's a good reason to have a different API.
You could have `init_spatialite` add methods to `Database` and `Table`, so they're only there if you have Spatialite set up. Is that too clever? It feels too clever.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029180984,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029180984,IC_kwDOCGYnMM49WA44,25778,eyeseast,2022-02-03T16:42:04Z,2022-02-03T16:42:04Z,CONTRIBUTOR,Fixed my spelling. That's a useful thing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029175907,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029175907,IC_kwDOCGYnMM49V_pj,25778,eyeseast,2022-02-03T16:36:54Z,2022-02-03T16:36:54Z,CONTRIBUTOR,"@simonw Not sure if you've seen this, but any chance you can run the tests?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/datasette/issues/1605#issuecomment-1018778667,https://api.github.com/repos/simonw/datasette/issues/1605,1018778667,IC_kwDOBm6k_c48uVQr,25778,eyeseast,2022-01-21T19:00:01Z,2022-01-21T19:00:01Z,CONTRIBUTOR,"Let me know if you want help prototyping any of this, because I'm thinking about it and trying stuff out. Happy to be a sounding board, if it helps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports,
https://github.com/simonw/datasette/issues/1605#issuecomment-1018741262,https://api.github.com/repos/simonw/datasette/issues/1605,1018741262,IC_kwDOBm6k_c48uMIO,25778,eyeseast,2022-01-21T18:05:09Z,2022-01-21T18:05:09Z,CONTRIBUTOR,"Thinking about this more, as well as #1356 and various other tickets related to output formats, I think there's a missing plugin hook for formatting results, separate from `register_output_renderer` (or maybe part of it, depending on #1101).
Right now, as I understand it, getting output in any format goes through the normal view stack -- a table, a row or a query -- and so by the time `register_output_renderer` gets it, the results have already been truncated or paginated. What I'd want, I think, is to be able to register ways to format results independent of where those results are sent.
It's possible this could be done using [`conn.row_factory`](https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.row_factory) (maybe in the `prepare_connection` hook), but I'm not sure that's where it belongs.
Another option is some kind of registry of serializers, which `register_output_renderer` and other plugin hooks could use. What I'm trying to avoid here is writing a plugin that also needs plugins for formats I haven't thought of yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports,
https://github.com/simonw/datasette/issues/1605#issuecomment-1016994329,https://api.github.com/repos/simonw/datasette/issues/1605,1016994329,IC_kwDOBm6k_c48nhoZ,25778,eyeseast,2022-01-20T00:27:17Z,2022-01-20T00:27:17Z,CONTRIBUTOR,"Right now, I usually have a line in a Makefile like this:
```make
combined.geojson: project.db
pipenv run datasette project.db --get /project/combined.geojson \
--load-extension spatialite \
--setting sql_time_limit_ms 5000 \
--setting max_returned_rows 20000 \
-m metadata.yml > $@
```
That all assumes I've loaded whatever I need into `project.db` and created a canned query called `combined` (and then uses `datasette-geojson` for geojson output).
It works, but as you can see, it's a lot to manage, a lot of boilerplate, and it wasn't obvious how to get there. If there's an error in the canned query, I get an HTML error page, so that's hard to debug. And it's only one query, so each output needs a line like this. Make isn't ideal, either, for that reason.
The thing I really liked with `datafreeze` was doing templated filenames. I have a project now where I need to export a bunch of litttle geojson files, based on queries, and it would be awesome to be able to do something like this:
```yml
databases:
project:
queries:
boundaries:
sql: ""SELECT * FROM boundaries""
filename: ""boundaries/{id}.geojson""
mode: ""item""
format: geojson
```
And then do:
```sh
datasette freeze -m metadata.yml project.db
```
For HTML export, maybe there's a `template` argument, or `format: template` or something. And that gets you a static site generator, kinda for free.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports,
https://github.com/simonw/datasette/issues/1601#issuecomment-1016651485,https://api.github.com/repos/simonw/datasette/issues/1601,1016651485,IC_kwDOBm6k_c48mN7d,25778,eyeseast,2022-01-19T16:39:03Z,2022-01-19T16:39:03Z,CONTRIBUTOR,"I think both of these are Spatialite specific. They get generated when you first initialize the extension. KNN is actually deprecated in favor of [KNN2](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN2), as I understand it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list,
https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1013698557,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1013698557,IC_kwDOCGYnMM48a8_9,25778,eyeseast,2022-01-15T15:15:22Z,2022-01-15T15:15:22Z,CONTRIBUTOR,@simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012413729,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012413729,IC_kwDOCGYnMM48WDUh,25778,eyeseast,2022-01-13T18:50:00Z,2022-01-13T18:50:00Z,CONTRIBUTOR,"One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012253198,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012253198,IC_kwDOCGYnMM48VcIO,25778,eyeseast,2022-01-13T15:39:14Z,2022-01-13T15:39:14Z,CONTRIBUTOR,"Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (`gis.py` or something) just so it's easier to find later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012230212,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012230212,IC_kwDOCGYnMM48VWhE,25778,eyeseast,2022-01-13T15:15:13Z,2022-01-13T15:15:13Z,CONTRIBUTOR,"Some proposals I'd add to sqlite-utils:
Some version of this, from [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite/blob/main/geojson_to_sqlite/utils.py#L124-L130):
```python
def init_spatialite(db, lib):
db.conn.enable_load_extension(True)
db.conn.load_extension(lib)
# Initialize SpatiaLite if not yet initialized
if ""spatial_ref_sys"" in db.table_names():
return
db.conn.execute(""select InitSpatialMetadata(1)"")
```
Also a function for creating a spatial index:
```python
db.conn.execute(""select CreateSpatialIndex(?, ?)"", [table, ""geometry""])
```
I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like:
```sh
sqlite-utils spatial-index spatial.db table-name column-name
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012158895,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012158895,IC_kwDOCGYnMM48VFGv,25778,eyeseast,2022-01-13T13:55:59Z,2022-01-13T13:55:59Z,CONTRIBUTOR,"Came here to add this. I might pick it up.
Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983155079,https://api.github.com/repos/simonw/sqlite-utils/issues/348,983155079,IC_kwDOCGYnMM46mcGH,25778,eyeseast,2021-12-01T00:28:40Z,2021-12-01T00:28:40Z,CONTRIBUTOR,"I'd use this. Right now, I tend to do `touch my.db` and then `enable-wal` or whatever else, but I'm never sure if that's a bad idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-953911245,https://api.github.com/repos/simonw/sqlite-utils/issues/242,953911245,IC_kwDOCGYnMM4424fN,25778,eyeseast,2021-10-28T14:37:55Z,2021-10-28T14:37:55Z,CONTRIBUTOR,"I've been thinking about this a bit lately, doing a project that involves moving a lot of data in and out of SQLite files, datasette and GeoJSON. This has me leaning toward the idea that something like [`datasette query`](https://github.com/simonw/datasette/issues/1356) would be a better place to do async queries.
I know there's a lot of overlap in sqlite-utils and datasette, and maybe keeping sqlite-utils synchronous would let datasette be entirely async and give a cleaner separation of implementations.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/datasette/issues/1101#issuecomment-869191854,https://api.github.com/repos/simonw/datasette/issues/1101,869191854,MDEyOklzc3VlQ29tbWVudDg2OTE5MTg1NA==,25778,eyeseast,2021-06-27T16:42:14Z,2021-06-27T16:42:14Z,CONTRIBUTOR,This would really help with this issue: https://github.com/eyeseast/datasette-geojson/issues/7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861944202,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861944202,MDEyOklzc3VlQ29tbWVudDg2MTk0NDIwMg==,25778,eyeseast,2021-06-16T01:41:03Z,2021-06-16T01:41:03Z,CONTRIBUTOR,"So, I do things like this a lot, too. I like the idea of piping in from stdin. Something like this would be nice to do in a makefile:
```sh
cat file.csv | sqlite-utils --csv --table data - 'SELECT * FROM data WHERE col=""whatever""' > filtered.csv
```
If you assumed that you're always piping out the same format you're piping in, the option names don't have to change. Depends how much you want to change formats.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/datasette/pull/1370#issuecomment-857298526,https://api.github.com/repos/simonw/datasette/issues/1370,857298526,MDEyOklzc3VlQ29tbWVudDg1NzI5ODUyNg==,25778,eyeseast,2021-06-09T01:18:59Z,2021-06-09T01:18:59Z,CONTRIBUTOR,"I'm happy to grab some or all of these in this PR, if you want. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",914130834,Ensure db.path is a string before trying to insert into internal database,
https://github.com/simonw/datasette/issues/1356#issuecomment-853895159,https://api.github.com/repos/simonw/datasette/issues/1356,853895159,MDEyOklzc3VlQ29tbWVudDg1Mzg5NTE1OQ==,25778,eyeseast,2021-06-03T14:03:59Z,2021-06-03T14:03:59Z,CONTRIBUTOR,"(Putting thoughts here to keep the conversation in one place.)
I think using datasette for this use-case is the right approach. I usually have both datasette and sqlite-utils installed in the same project, and that's where I'm trying out queries, so it probably makes the most sense to have datasette also manage the output (and maybe the input, too).
It seems like both `--get` and `--query` could work better as subcommands, rather than options, if you're looking at building out a full CLI experience in datasette. It would give a cleaner separation in what you're trying to do and let each have its own dedicated options. So something like this:
```sh
# run an arbitrary query
datasette query covid.db ""select * from ny_times_us_counties limit 1"" --format yaml
# run a canned query
datasette get covid.db some-canned-query --format yaml
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787121933,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787121933,MDEyOklzc3VlQ29tbWVudDc4NzEyMTkzMw==,25778,eyeseast,2021-02-27T19:18:57Z,2021-02-27T19:18:57Z,CONTRIBUTOR,"I think HTTPX gets it exactly right, with a clear separation between sync and async clients, each with a basically identical API. (I'm about to switch [feed-to-sqlite](https://github.com/eyeseast/feed-to-sqlite) over to it, from Requests, to eventually make way for async support.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/datasette/issues/731#issuecomment-618758326,https://api.github.com/repos/simonw/datasette/issues/731,618758326,MDEyOklzc3VlQ29tbWVudDYxODc1ODMyNg==,25778,eyeseast,2020-04-24T01:55:00Z,2020-04-24T01:55:00Z,CONTRIBUTOR,Mounting `./static` at `/static` seems the simplest way. Saves you the trouble of deciding what else (`img` for example) gets special treatment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605110015,Option to automatically configure based on directory layout,
https://github.com/simonw/datasette/issues/731#issuecomment-618126449,https://api.github.com/repos/simonw/datasette/issues/731,618126449,MDEyOklzc3VlQ29tbWVudDYxODEyNjQ0OQ==,25778,eyeseast,2020-04-23T01:38:55Z,2020-04-23T01:38:55Z,CONTRIBUTOR,"I've almost suggested this same thing a couple times. I tend to have Makefile (because I'm doing other `make` stuff anyway to get data prepped), and I end up putting all those CLI options in something like `make run`. But it would be way easier to just have all those typical options -- plugins, templates, metadata -- be defaults.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605110015,Option to automatically configure based on directory layout,
https://github.com/simonw/datasette/pull/434#issuecomment-489105665,https://api.github.com/repos/simonw/datasette/issues/434,489105665,MDEyOklzc3VlQ29tbWVudDQ4OTEwNTY2NQ==,25778,eyeseast,2019-05-03T14:01:30Z,2019-05-03T14:01:30Z,CONTRIBUTOR,This is exactly what I needed. Thank you.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",434321685,"""datasette publish cloudrun"" command to publish to Google Cloud Run",
https://github.com/simonw/datasette/issues/236#issuecomment-799003172,https://api.github.com/repos/simonw/datasette/issues/236,799003172,MDEyOklzc3VlQ29tbWVudDc5OTAwMzE3Mg==,21148,jacobian,2021-03-14T23:42:57Z,2021-03-14T23:42:57Z,CONTRIBUTOR,"Oh, and the container image can be up to 10GB, so the EFS step might not be needed except for pretty big stuff.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/simonw/datasette/issues/236#issuecomment-799002993,https://api.github.com/repos/simonw/datasette/issues/236,799002993,MDEyOklzc3VlQ29tbWVudDc5OTAwMjk5Mw==,21148,jacobian,2021-03-14T23:41:51Z,2021-03-14T23:41:51Z,CONTRIBUTOR,"Now that [Lambda supports Docker](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/), this probably is a bit easier and may be able to build on top of the existing package command.
There are weirdnesses in how the command actually gets invoked; the [aws-lambda-python image](https://hub.docker.com/r/amazon/aws-lambda-python) shows a bit of that. So Datasette would probably need some sort of Lambda-specific entry point to make this work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/dogsheep/twitter-to-sqlite/pull/55#issuecomment-760950128,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55,760950128,MDEyOklzc3VlQ29tbWVudDc2MDk1MDEyOA==,21148,jacobian,2021-01-15T13:44:52Z,2021-01-15T13:44:52Z,CONTRIBUTOR,"I found and fixed another bug, this one around importing the tweets table. @simonw let me know if you'd prefer this broken out into multiple PRs, happy to do that if it makes review/merging easier.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779211940,Fix archive imports,
https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-754729035,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54,754729035,MDEyOklzc3VlQ29tbWVudDc1NDcyOTAzNQ==,21148,jacobian,2021-01-05T16:03:29Z,2021-01-05T16:03:29Z,CONTRIBUTOR,"I was able to fix this, at least enough to get _my_ archive to import. Not sure if there's more work to be done here or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779088071,Archive import appears to be broken on recent exports,
https://github.com/dogsheep/twitter-to-sqlite/pull/55#issuecomment-754728696,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55,754728696,MDEyOklzc3VlQ29tbWVudDc1NDcyODY5Ng==,21148,jacobian,2021-01-05T16:02:55Z,2021-01-05T16:02:55Z,CONTRIBUTOR,"This now works for me, though I'm entirely ensure if it's a just-my-export thing or a wider issue. Also, this doesn't contain any tests. So I'm not sure if there's more work to be done here, or if this is good enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779211940,Fix archive imports,
https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-754721153,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54,754721153,MDEyOklzc3VlQ29tbWVudDc1NDcyMTE1Mw==,21148,jacobian,2021-01-05T15:51:09Z,2021-01-05T15:51:09Z,CONTRIBUTOR,"Correction: the failure is on `lists-member.js` (I was thrown by the `block` variable name, but that's just a coincidence)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779088071,Archive import appears to be broken on recent exports,
https://github.com/simonw/datasette/issues/639#issuecomment-558687342,https://api.github.com/repos/simonw/datasette/issues/639,558687342,MDEyOklzc3VlQ29tbWVudDU1ODY4NzM0Mg==,21148,jacobian,2019-11-26T15:40:00Z,2019-11-26T15:40:00Z,CONTRIBUTOR,"A bit of background: the reason `heroku git:clone` brings down an empty directory is because `datasette publish heroku` uses the [builds API](https://devcenter.heroku.com/articles/build-and-release-using-the-api), rather than a `git push`, to release the app. I originally did this because it seemed like a lower bar than having a working `git`, but the downside is, as you found out, that tweaking the created app is hard.
So there's one option -- change `datasette publish heroku` to use `git push` instead of `heroku builds:create`.
@pkoppstein - what you suggested seems like it ought to work (you don't need maintenance mode, though). I'm not sure why it doesn't.
You could also look into using the [slugs API](https://devcenter.heroku.com/articles/platform-api-deploying-slugs) to download the slug, change `metadata.json`, re-pack and re-upload the slug.
Ultimately though I think I think @simonw's idea of reading `metadata.json` from an external source might be better (#357). Reading from an alternate URL would be fine, or you could also just stuff the whole `metadata.json` into a Heroku config var, and write a plugin to read it from there.
Hope this helps a bit!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",527670799,updating metadata.json without recreating the app,
https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552134876,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29,552134876,MDEyOklzc3VlQ29tbWVudDU1MjEzNDg3Ng==,21148,jacobian,2019-11-09T20:33:38Z,2019-11-09T20:33:38Z,CONTRIBUTOR,❤️ thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518725064,`import` command fails on empty files,
https://github.com/simonw/datasette/issues/14#issuecomment-346244871,https://api.github.com/repos/simonw/datasette/issues/14,346244871,MDEyOklzc3VlQ29tbWVudDM0NjI0NDg3MQ==,21148,jacobian,2017-11-22T05:06:30Z,2017-11-22T05:06:30Z,CONTRIBUTOR,"I'd also suggest taking a look at [stevedore](https://docs.openstack.org/stevedore/latest/), which has a ton of tools for doing plugin stuff. I've had good luck with it in the past.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267707940,Datasette Plugins,
https://github.com/simonw/datasette/pull/104#issuecomment-346124764,https://api.github.com/repos/simonw/datasette/issues/104,346124764,MDEyOklzc3VlQ29tbWVudDM0NjEyNDc2NA==,21148,jacobian,2017-11-21T18:52:14Z,2017-11-21T18:52:14Z,CONTRIBUTOR,"OK, now this should work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274284246,[WIP] Add publish to heroku support,
https://github.com/simonw/datasette/pull/104#issuecomment-346124073,https://api.github.com/repos/simonw/datasette/issues/104,346124073,MDEyOklzc3VlQ29tbWVudDM0NjEyNDA3Mw==,21148,jacobian,2017-11-21T18:49:55Z,2017-11-21T18:49:55Z,CONTRIBUTOR,"Actually hang on, don't merge - there are some bugs that #141 masked when I tested this out elsewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274284246,[WIP] Add publish to heroku support,
https://github.com/simonw/datasette/pull/104#issuecomment-346116745,https://api.github.com/repos/simonw/datasette/issues/104,346116745,MDEyOklzc3VlQ29tbWVudDM0NjExNjc0NQ==,21148,jacobian,2017-11-21T18:23:25Z,2017-11-21T18:23:25Z,CONTRIBUTOR,"@simonw ready for a review and merge if you want.
There's still some nasty duplicated code in cli.py and utils.py, which is just going to get worse if/when we start adding any other deploy targets (and I want to do one for cloud.gov, at least). I think there's an opportunity for some refactoring here. I'm happy to do that now as part of this PR, or if you merge this first I'll do it in a different one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274284246,[WIP] Add publish to heroku support,
https://github.com/simonw/datasette/pull/104#issuecomment-345452669,https://api.github.com/repos/simonw/datasette/issues/104,345452669,MDEyOklzc3VlQ29tbWVudDM0NTQ1MjY2OQ==,21148,jacobian,2017-11-18T16:18:45Z,2017-11-18T16:18:45Z,CONTRIBUTOR,"I'd like to do a bit of cleanup, and some error checking in case heroku/heroku-builds isn't installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274284246,[WIP] Add publish to heroku support,
https://github.com/simonw/datasette/pull/104#issuecomment-344710204,https://api.github.com/repos/simonw/datasette/issues/104,344710204,MDEyOklzc3VlQ29tbWVudDM0NDcxMDIwNA==,21148,jacobian,2017-11-15T19:57:50Z,2017-11-15T19:57:50Z,CONTRIBUTOR,"A first basic stab at making this work, just to prove the approach. Right now this requires [a Heroku CLI plugin](https://github.com/heroku/heroku-builds), which seems pretty unreasonable. I think this can be replaced with direct API calls, which could clean up a lot of things. But I wanted to prove it worked first, and it does.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274284246,[WIP] Add publish to heroku support,
https://github.com/simonw/datasette/issues/88#issuecomment-344430689,https://api.github.com/repos/simonw/datasette/issues/88,344430689,MDEyOklzc3VlQ29tbWVudDM0NDQzMDY4OQ==,15543,tomdyson,2017-11-14T23:08:22Z,2017-11-14T23:08:22Z,CONTRIBUTOR,"> I'm getting an internal server error on http://run.plnkr.co/preview/cj9zlf1qc0003414y90ajkwpk/ at the moment
Sorry about that - here's a working version on Netlify:
https://nhs-england-map.netlify.com","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273775212,Add NHS England Hospitals example to wiki,
https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573389669,https://api.github.com/repos/simonw/sqlite-utils/issues/74,573389669,MDEyOklzc3VlQ29tbWVudDU3MzM4OTY2OQ==,15092,jayvdb,2020-01-12T07:21:17Z,2020-01-12T07:21:17Z,CONTRIBUTOR,"I guess there is some extra flag for ` CliRunner.invoke` to check exitcode and raise the exception, or that should be an extra assert added.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546073980,Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column,
https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573388052,https://api.github.com/repos/simonw/sqlite-utils/issues/74,573388052,MDEyOklzc3VlQ29tbWVudDU3MzM4ODA1Mg==,15092,jayvdb,2020-01-12T06:51:30Z,2020-01-12T06:51:30Z,CONTRIBUTOR,"Thanks. That showed me that there was a click cli runner error, and setting `export LANG=en_US.UTF-8` fixed it.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546073980,Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column,
https://github.com/simonw/datasette/issues/1167#issuecomment-754619930,https://api.github.com/repos/simonw/datasette/issues/1167,754619930,MDEyOklzc3VlQ29tbWVudDc1NDYxOTkzMA==,3637,benpickles,2021-01-05T12:57:57Z,2021-01-05T12:57:57Z,CONTRIBUTOR,"Not sure where exactly to put the actual docs (presumably somewhere in [docs/contributing.rst](https://github.com/simonw/datasette/blob/main/docs/contributing.rst)) but I've made a slight change to make it easier to run locally (copying [the approach in excalidraw](https://github.com/excalidraw/excalidraw/blob/ade2565f497243a5e428f4906d8ed80c872fd981/package.json#L90-L94)): https://github.com/simonw/datasette/compare/main...benpickles:prettier-docs
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777145954,Add Prettier to contributing documentation,
https://github.com/simonw/datasette/issues/1169#issuecomment-754007242,https://api.github.com/repos/simonw/datasette/issues/1169,754007242,MDEyOklzc3VlQ29tbWVudDc1NDAwNzI0Mg==,3637,benpickles,2021-01-04T14:29:57Z,2021-01-04T14:29:57Z,CONTRIBUTOR,I somewhat share your reluctance to add a package.json to seemingly every project out there but ultimately if they're project dependencies it's important they're managed within the codebase.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777677671,Prettier package not actually being cached,
https://github.com/simonw/datasette/pull/1170#issuecomment-754004715,https://api.github.com/repos/simonw/datasette/issues/1170,754004715,MDEyOklzc3VlQ29tbWVudDc1NDAwNDcxNQ==,3637,benpickles,2021-01-04T14:25:44Z,2021-01-04T14:25:44Z,CONTRIBUTOR,I was going to re-add the filter to only run Prettier when there have been changes in `datasette/static` but that would mean it wouldn't run when the package is updated. That plus the fact that [the last run of the job took only 8 seconds](https://github.com/benpickles/datasette/runs/1640121514) is why I decided not to re-add the filter.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778126516,Install Prettier via package.json,
https://github.com/simonw/datasette/pull/1931#issuecomment-1339916064,https://api.github.com/repos/simonw/datasette/issues/1931,1339916064,IC_kwDOBm6k_c5P3X8g,3556,davidbgk,2022-12-06T19:42:45Z,2022-12-06T19:42:45Z,CONTRIBUTOR,"The `""return"": true` option is really nice!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert,
https://github.com/simonw/datasette/issues/1929#issuecomment-1339906969,https://api.github.com/repos/simonw/datasette/issues/1929,1339906969,IC_kwDOBm6k_c5P3VuZ,3556,davidbgk,2022-12-06T19:34:20Z,2022-12-06T19:34:20Z,CONTRIBUTOR,I confirm that it works 👍 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473659191,Incorrect link from the API explorer to the JSON API documentation,
https://github.com/simonw/datasette/issues/1871#issuecomment-1309650806,https://api.github.com/repos/simonw/datasette/issues/1871,1309650806,IC_kwDOBm6k_c5OD692,3556,davidbgk,2022-11-10T01:38:58Z,2022-11-10T01:38:58Z,CONTRIBUTOR,"> Realized the API explorer doesn't need the API key piece at all - it can work with standard cookie-based auth.
>
> This also reflects how most plugins are likely to use this API, where they'll be adding JavaScript that uses `fetch()` to call the write API directly.
I agree (that's what I did with the previous insert plugin), maybe a complete example using `fetch()` in the documentation would be valuable as a “Getting started with the API” or similar?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool,
https://github.com/simonw/datasette/issues/1552#issuecomment-996229007,https://api.github.com/repos/simonw/datasette/issues/1552,996229007,IC_kwDOBm6k_c47YT-P,3556,davidbgk,2021-12-16T22:04:39Z,2021-12-16T22:04:39Z,CONTRIBUTOR,"Wow, that was fast, thank you so much @simonw !
> I'm also not convinced that this configuration syntax is right. It's a bit weird having a `""facets""` list that can either by column-name-strings or `{""type-of-facet"": ""column-name""}` objects. Maybe there's a better design for this?
I agree that it's not ideal, my initial naive approach was to detect if it's an array, like what is done here:
https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/facets.py#L312-L313
But it requires an extra query to determine the type, which is a bit problematic, especially for big tables I guess.
Taking a look at #510, I wonder if a `facet_delimiter` should be defined for that kind of columns (that would help our team not to have an intermediary conversion step from `foo|bar` to `[""foo"",""bar""]` for instance).
To be consistent with the `--extract-column` parameter, maybe an explicit casting/delimiter would be useful: `--set-column 'Foo:Array:|'`.
Throwing a lot of ideas without knowing the big picture… but sometimes newcomers have superpowers :).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/1552#issuecomment-995296725,https://api.github.com/repos/simonw/datasette/issues/1552,995296725,IC_kwDOBm6k_c47UwXV,3556,davidbgk,2021-12-15T23:29:32Z,2021-12-15T23:29:32Z,CONTRIBUTOR,"@simonw thank you for your fast answer and your guidance!
While digging into the code, I found an undocumented way of doing it:
```yaml
facets: [""Facet for a column"", {""array"": ""Facet for an array""}]
```
The only remaining problem with that solution is here: https://github.com/simonw/datasette/blob/250db8192cb8aba5eb8cd301ccc2a49525bc3d24/datasette/facets.py#L33
We have:
```python
type, metadata_config = metadata_config.items()[0]
```
But it requires to cast the `dict_items` as a list prior to access the first element:
```python
type, metadata_config = list(metadata_config.items())[0]
```
I guess it's an unspotted bug? (I mean, independently of the facets-with-arrays issue.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/1114#issuecomment-735436014,https://api.github.com/repos/simonw/datasette/issues/1114,735436014,MDEyOklzc3VlQ29tbWVudDczNTQzNjAxNA==,2182,danp,2020-11-29T18:33:30Z,2020-11-29T18:33:30Z,CONTRIBUTOR,Thank you!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752966476,--load-extension=spatialite not working with datasetteproject/datasette docker image,
|