.html file that hardcodes a default param in the link, but I'd rather not customize the template just for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421,Allow canned query params to specify default values,
https://github.com/simonw/datasette/issues/741#issuecomment-806166575,https://api.github.com/repos/simonw/datasette/issues/741,806166575,MDEyOklzc3VlQ29tbWVudDgwNjE2NjU3NQ==,9599,simonw,2021-03-24T20:30:33Z,2021-03-24T20:30:33Z,OWNER,"`datasette package` is a mostly unmaintained feature at this point - it has a bit of test coverage but I've not made any improvements to it in a few years, and I don't use it for my own projects.
I'll make this change to `package` at the same time as I land it for `publish` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/741#issuecomment-806010960,https://api.github.com/repos/simonw/datasette/issues/741,806010960,MDEyOklzc3VlQ29tbWVudDgwNjAxMDk2MA==,596279,zaneselvans,2021-03-24T17:19:42Z,2021-03-24T17:19:42Z,NONE,"Ah, okay so `--extra-options` applies to both `datasette publish` and `datasette package`? There wren't any examples of it being used with `publish` in the docs, so this tripped me up for a bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/1274#issuecomment-805216038,https://api.github.com/repos/simonw/datasette/issues/1274,805216038,MDEyOklzc3VlQ29tbWVudDgwNTIxNjAzOA==,9599,simonw,2021-03-23T20:14:53Z,2021-03-23T20:14:53Z,OWNER,"Yes this is one of the main reasons I'm planning to switch to encouraging YAML be default instead of JSON (while still supporting JSON) - YAML supports comments and multi-line strings.
See #1153 for YAML by default in the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",839008371,Might there be some way to comment metadata.json?,
https://github.com/simonw/datasette/issues/1274#issuecomment-805214307,https://api.github.com/repos/simonw/datasette/issues/1274,805214307,MDEyOklzc3VlQ29tbWVudDgwNTIxNDMwNw==,7476523,bobwhitelock,2021-03-23T20:12:29Z,2021-03-23T20:12:29Z,CONTRIBUTOR,"One issue I could see with adding first class support for metadata in hjson format is that this would require adding an additional dependency to handle this, for a feature that would be unused by many users. I wonder if this could fit in as a plugin instead; if a hook existed for loading metadata (maybe as part of https://github.com/simonw/datasette/issues/860) the metadata could then come from any source, as specified by plugins, e.g. hjson, toml, XML, a database table etc.
Until/unless this exists, a few ideas for how you could add comments:
- Using YAML as you suggest.
- A common pattern is adding a `""comment""` key for comments to any object in JSON - I don't think including an unnecessary key like this would break anything in Datasette, but not certain.
- You could use another tool as a preprocessor for your JSON metadata - e.g. hjson or Jsonnet. You'd write the metadata in that format, and then convert that into JSON to actually use as your final metadata.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",839008371,Might there be some way to comment metadata.json?,
https://github.com/simonw/datasette/issues/1153#issuecomment-805109341,https://api.github.com/repos/simonw/datasette/issues/1153,805109341,MDEyOklzc3VlQ29tbWVudDgwNTEwOTM0MQ==,9599,simonw,2021-03-23T17:55:48Z,2021-03-23T18:41:57Z,OWNER,"Beginnings of a UI element for switching between them:
```html
```
![]()
That `` has a padding of 12px, so using 12px padding on the tab links should get them to line up better.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1249#issuecomment-805033155,https://api.github.com/repos/simonw/datasette/issues/1249,805033155,MDEyOklzc3VlQ29tbWVudDgwNTAzMzE1NQ==,9599,simonw,2021-03-23T16:12:13Z,2021-03-23T16:12:13Z,OWNER,"Don't forget to update this bit of the docs: https://docs.datasette.io/en/0.55/spatialite.html#building-spatialite-from-source
> The packaged versions of SpatiaLite usually provide SpatiaLite 4.3.0a. For an example of how to build the most recent unstable version, 4.4.0-RC0 (which includes the powerful [VirtualKNN module](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN)), take a look at the [Datasette Dockerfile](https://github.com/simonw/datasette/blob/master/Dockerfile).
See also #1273","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1270#issuecomment-805058241,https://api.github.com/repos/simonw/datasette/issues/1270,805058241,MDEyOklzc3VlQ29tbWVudDgwNTA1ODI0MQ==,9599,simonw,2021-03-23T16:45:39Z,2021-03-23T16:45:39Z,OWNER,"I managed to build SpatiaLite such that this isn't necessary any more. I'm still interested in pursuing this further though - it feels like it could be a more robust way of implementing timeouts, but I need to prove to myself that it's better (maybe better performance, or handles more edge-cases?). Not sure how to prove that yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),
https://github.com/simonw/datasette/issues/1153#issuecomment-805056806,https://api.github.com/repos/simonw/datasette/issues/1153,805056806,MDEyOklzc3VlQ29tbWVudDgwNTA1NjgwNg==,9599,simonw,2021-03-23T16:43:38Z,2021-03-23T16:43:38Z,OWNER,"I used this code to get that:
```javascript
var jsonVersion = JSON.stringify(window.jsyaml.load(document.querySelector('.highlight-yaml').textContent), null, 4);
div.querySelector('.highlight pre').innerText = jsonVersion;
div.querySelector('.highlight pre').style.whiteSpace = 'pre-wrap'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805055291,https://api.github.com/repos/simonw/datasette/issues/1153,805055291,MDEyOklzc3VlQ29tbWVudDgwNTA1NTI5MQ==,9599,simonw,2021-03-23T16:41:31Z,2021-03-23T16:41:31Z,OWNER,"One downside of doing this conversion in JavaScript: it's much harder to get the same JSON syntax highlighting as that provided by Sphinx:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805050163,https://api.github.com/repos/simonw/datasette/issues/1153,805050163,MDEyOklzc3VlQ29tbWVudDgwNTA1MDE2Mw==,9599,simonw,2021-03-23T16:34:35Z,2021-03-23T16:35:32Z,OWNER,"https://docs.datasette.io/en/stable/metadata.html has this example:
```yaml
title: Demonstrating Metadata from YAML
description_html: |-
This description includes a long HTML string
- YAML is better for embedding HTML strings than JSON!
license: ODbL
license_url: https://opendatacommons.org/licenses/odbl/
databases:
fixtures:
tables:
no_primary_key:
hidden: true
queries:
neighborhood_search:
sql: |-
select neighborhood, facet_cities.name, state
from facetable join facet_cities on facetable.city_id = facet_cities.id
where neighborhood like '%' || :text || '%' order by neighborhood;
title: Search neighborhoods
description_html: |-
This demonstrates basic LIKE search
```
I ran this in the browser dev tools:
```javascript
var s = document.createElement('script')
s.src = 'https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js'
document.head.appendChild(s)
var yamlExample = document.querySelector('.highlight-yaml').textContent);
console.log(JSON.stringify(window.jsyaml.load(yamlExample), null, 4))
```
And got:
```json
{
""title"": ""Demonstrating Metadata from YAML"",
""description_html"": ""
This description includes a long HTML string
\n\n - YAML is better for embedding HTML strings than JSON!
\n
"",
""license"": ""ODbL"",
""license_url"": ""https://opendatacommons.org/licenses/odbl/"",
""databases"": {
""fixtures"": {
""tables"": {
""no_primary_key"": {
""hidden"": true
}
},
""queries"": {
""neighborhood_search"": {
""sql"": ""select neighborhood, facet_cities.name, state\nfrom facetable join facet_cities on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%' order by neighborhood;"",
""title"": ""Search neighborhoods"",
""description_html"": ""This demonstrates basic LIKE search""
}
}
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805047117,https://api.github.com/repos/simonw/datasette/issues/1153,805047117,MDEyOklzc3VlQ29tbWVudDgwNTA0NzExNw==,9599,simonw,2021-03-23T16:30:15Z,2021-03-23T16:46:06Z,OWNER,"https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js is only 12.5KB zipped, 38KB total - so that's not a bad option.
https://github.com/nodeca/js-yaml","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805042880,https://api.github.com/repos/simonw/datasette/issues/1153,805042880,MDEyOklzc3VlQ29tbWVudDgwNTA0Mjg4MA==,9599,simonw,2021-03-23T16:24:32Z,2021-03-23T16:24:32Z,OWNER,... actually I think I would do that conversion in Python. The client-side YAML parsers all look a little bit heavy to me in terms of additional page weight.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805041522,https://api.github.com/repos/simonw/datasette/issues/1153,805041522,MDEyOklzc3VlQ29tbWVudDgwNTA0MTUyMg==,9599,simonw,2021-03-23T16:22:46Z,2021-03-23T16:22:46Z,OWNER,"That's a good idea. I could do that with JavaScript - loading YAML and converting it to JSON in JavaScript shouldn't be hard, and it's better than JSON-to-YAML because there's only one correct JSON representation of a YAML file whereas you can represent a JSON document in YAML in a bunch of different ways.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/pull/1159#issuecomment-804639427,https://api.github.com/repos/simonw/datasette/issues/1159,804639427,MDEyOklzc3VlQ29tbWVudDgwNDYzOTQyNw==,192568,mroswell,2021-03-23T05:56:02Z,2021-03-23T05:56:02Z,CONTRIBUTOR,"With just three facets, I like it, but it does take more horizontal space. Would be nice to have a switch somewhere, enabling either original compact option or this proposed more-readable option. Also some control over word wrap (width setting) and facet spacing. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",774332247,Improve the display of facets information,
https://github.com/simonw/datasette/pull/1159#issuecomment-804698315,https://api.github.com/repos/simonw/datasette/issues/1159,804698315,MDEyOklzc3VlQ29tbWVudDgwNDY5ODMxNQ==,552629,lovasoa,2021-03-23T07:58:28Z,2021-03-23T07:58:38Z,NONE,@mroswell Did you try it with more columns ? The display is flexible and columns get closer as new ones are added.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",774332247,Improve the display of facets information,
https://github.com/simonw/datasette/issues/1153#issuecomment-804640440,https://api.github.com/repos/simonw/datasette/issues/1153,804640440,MDEyOklzc3VlQ29tbWVudDgwNDY0MDQ0MA==,192568,mroswell,2021-03-23T05:58:20Z,2021-03-23T05:58:20Z,CONTRIBUTOR,Could there be a little widget that offers conversion from one to the other? ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/163#issuecomment-804539729,https://api.github.com/repos/simonw/datasette/issues/163,804539729,MDEyOklzc3VlQ29tbWVudDgwNDUzOTcyOQ==,192568,mroswell,2021-03-23T02:41:14Z,2021-03-23T02:41:14Z,CONTRIBUTOR,"I'm visiting old issues for context while learning datasette. Let me know if okay to make the occasional comment like this one.
querystring argument now located at:
https://docs.datasette.io/en/latest/settings.html#sql-time-limit-ms","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",279547886,Document the querystring argument for setting a different time limit,
https://github.com/simonw/datasette/issues/164#issuecomment-804541064,https://api.github.com/repos/simonw/datasette/issues/164,804541064,MDEyOklzc3VlQ29tbWVudDgwNDU0MTA2NA==,192568,mroswell,2021-03-23T02:45:12Z,2021-03-23T02:45:12Z,CONTRIBUTOR,"""datasette skeleton"" feature removed #476","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",280013907,datasette skeleton command for kick-starting database and table metadata,
https://github.com/simonw/datasette/issues/163#issuecomment-804540869,https://api.github.com/repos/simonw/datasette/issues/163,804540869,MDEyOklzc3VlQ29tbWVudDgwNDU0MDg2OQ==,9599,simonw,2021-03-23T02:44:33Z,2021-03-23T02:44:33Z,OWNER,Comments welcome!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",279547886,Document the querystring argument for setting a different time limit,
https://github.com/simonw/datasette/issues/1149#issuecomment-804415619,https://api.github.com/repos/simonw/datasette/issues/1149,804415619,MDEyOklzc3VlQ29tbWVudDgwNDQxNTYxOQ==,192568,mroswell,2021-03-22T21:43:16Z,2021-03-22T21:43:16Z,CONTRIBUTOR,Sounds like a good idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769520939,Make it easier to theme Datasette with CSS,
https://github.com/simonw/datasette/issues/1249#issuecomment-804406675,https://api.github.com/repos/simonw/datasette/issues/1249,804406675,MDEyOklzc3VlQ29tbWVudDgwNDQwNjY3NQ==,9599,simonw,2021-03-22T21:26:27Z,2021-03-22T21:26:27Z,OWNER,(Without the `apt-get update ...` SpatiaLite line it's 125MB),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804404544,https://api.github.com/repos/simonw/datasette/issues/1249,804404544,MDEyOklzc3VlQ29tbWVudDgwNDQwNDU0NA==,9599,simonw,2021-03-22T21:22:56Z,2021-03-22T21:24:24Z,OWNER,"Final version of Dockerfile which installs the specified version from GitHub:
docker build . -t datasette-spatialite --build-arg VERSION=0.55
```dockerfile
FROM python:3.9.2-slim-buster as build
# Version of Datasette to install, e.g. 0.55
# docker build . -t datasette --build-arg VERSION=0.55
ARG VERSION
# software-properties-common provides add-apt-repository
# which we need in order to install a more recent release
# of libsqlite3-mod-spatialite from the sid distribution
RUN apt-get update && \
apt-get -y --no-install-recommends install software-properties-common && \
add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
apt-get update && \
apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
apt-get remove -y software-properties-common && \
apt clean && \
rm -rf /var/lib/apt && \
rm -rf /var/lib/dpkg
RUN pip install https://github.com/simonw/datasette/archive/refs/tags/${VERSION}.zip && \
find /usr/local/lib -name '__pycache__' | xargs rm -r && \
rm -rf /root/.cache/pip
EXPOSE 8001
CMD [""datasette""]
```
Run against 0.55 this produces an image of 262MB","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/88#issuecomment-804471733,https://api.github.com/repos/simonw/datasette/issues/88,804471733,MDEyOklzc3VlQ29tbWVudDgwNDQ3MTczMw==,192568,mroswell,2021-03-22T23:46:36Z,2021-03-22T23:46:36Z,CONTRIBUTOR,Google Map API limits seem to prevent https://nhs-england-map.netlify.com from being a working demo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273775212,Add NHS England Hospitals example to wiki,
https://github.com/simonw/datasette/issues/1249#issuecomment-804338678,https://api.github.com/repos/simonw/datasette/issues/1249,804338678,MDEyOklzc3VlQ29tbWVudDgwNDMzODY3OA==,9599,simonw,2021-03-22T19:33:43Z,2021-03-22T19:33:43Z,OWNER,"Replacing `rm -rf /var/lib/{apt,dpkg,cache,log}/` with
```
rm -rf /var/lib/apt && \
rm -rf /var/lib/dpkg
```
Got the size down to 305MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804318314,https://api.github.com/repos/simonw/datasette/issues/1249,804318314,MDEyOklzc3VlQ29tbWVudDgwNDMxODMxNA==,9599,simonw,2021-03-22T19:04:30Z,2021-03-22T19:04:30Z,OWNER,Considering the image on Docker Hub right now is `383MB` this is actually an improvement.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804317545,https://api.github.com/repos/simonw/datasette/issues/1249,804317545,MDEyOklzc3VlQ29tbWVudDgwNDMxNzU0NQ==,9599,simonw,2021-03-22T19:03:22Z,2021-03-22T19:03:22Z,OWNER,"This Dockerfile:
```dockerfile
FROM python:3.9.2-slim-buster as build
# software-properties-common provides add-apt-repository
RUN apt-get update && \
apt-get -y install software-properties-common && \
add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
apt-get update && \
apt-get -t sid install -y libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
RUN pip install datasette
EXPOSE 8001
CMD [""datasette""]
```
Produces a 344MB image that includes a working SpatiaLite 5.0 module. And weirdly... it doesn't exhibit the hanging bug!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804310353,https://api.github.com/repos/simonw/datasette/issues/1249,804310353,MDEyOklzc3VlQ29tbWVudDgwNDMxMDM1Mw==,9599,simonw,2021-03-22T18:52:12Z,2021-03-22T18:52:12Z,OWNER,"This Dockerfile:
```dockerfile
FROM python:3.9.2-slim-buster as build
# Setup build dependencies
RUN apt update \
&& apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
&& apt clean
RUN wget ""https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz"" && tar xzf sqlite-autoconf-3340100.tar.gz \
&& cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
CFLAGS=""-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1"" \
&& make && make install
RUN wget ""http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz"" && tar zxf freexl-1.0.6.tar.gz \
&& cd freexl-1.0.6 && ./configure && make && make install
RUN wget ""http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz"" && tar zxf libspatialite-5.0.1.tar.gz \
&& cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install
RUN wget ""http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz"" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install
RUN wget ""http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz"" && tar zxf spatialite-tools-5.0.0.tar.gz \
&& cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install
# Add local code to the image instead of fetching from pypi.
#COPY . /datasette
#RUN pip install /datasette
RUN pip install datasette
FROM python:3.9.2-slim-buster
# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=build /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu
ENV LD_LIBRARY_PATH=/usr/local/lib
EXPOSE 8001
CMD [""datasette""]
```
Produced a 448MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804309510,https://api.github.com/repos/simonw/datasette/issues/1249,804309510,MDEyOklzc3VlQ29tbWVudDgwNDMwOTUxMA==,9599,simonw,2021-03-22T18:50:50Z,2021-03-22T18:50:50Z,OWNER,"Ideally I'd like to use the Debian stable `python:3.9.2-slim-buster` base image but install SpatiaLite from Debian unstable here: https://packages.debian.org/sid/libspatialite7
This pattern might let me do that: https://github.com/helmesjo/cpp_bash_utils/blob/f031e926249f8e2d7f260f22dc8974c6d5be11fe/docker/images/linux-gcc.dockerfile#L20-L24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804384196,https://api.github.com/repos/simonw/datasette/issues/1249,804384196,MDEyOklzc3VlQ29tbWVudDgwNDM4NDE5Ng==,9599,simonw,2021-03-22T20:48:46Z,2021-03-22T20:48:46Z,OWNER,I think part of the reason it's smaller is that I ran `pip install datasette` instead of using `COPY . /datasette` followed by `pip install /datasette`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804380181,https://api.github.com/repos/simonw/datasette/issues/1249,804380181,MDEyOklzc3VlQ29tbWVudDgwNDM4MDE4MQ==,9599,simonw,2021-03-22T20:42:16Z,2021-03-22T20:42:16Z,OWNER,"Considering the image on Docker Hub is 383MB, I'm happy with getting that down to 262MB. I'm going to stop looking for new optimizations here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804379644,https://api.github.com/repos/simonw/datasette/issues/1249,804379644,MDEyOklzc3VlQ29tbWVudDgwNDM3OTY0NA==,9599,simonw,2021-03-22T20:41:23Z,2021-03-22T20:41:23Z,OWNER,I tried adding `apt-get remove -y software-properties-common &&` to remove `software-properties-common` but it made no difference to the image size.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804372977,https://api.github.com/repos/simonw/datasette/issues/1249,804372977,MDEyOklzc3VlQ29tbWVudDgwNDM3Mjk3Nw==,9599,simonw,2021-03-22T20:30:37Z,2021-03-22T20:30:37Z,OWNER,"I tried copying just the `mod_spatialite.so` file into a second stage build but it failed. So I ran `bash` in a working image and used `ldd` to figure out what it was linked to:
```
root@39683f91e588:/usr/lib/x86_64-linux-gnu# ldd mod_spatialite.so
linux-vdso.so.1 (0x00007ffd021f4000)
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f5c75412000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5c753f0000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c752ac000)
libminizip.so.1 => /usr/lib/x86_64-linux-gnu/libminizip.so.1 (0x00007f5c750a0000)
librttopo.so.1 => /usr/lib/x86_64-linux-gnu/librttopo.so.1 (0x00007f5c75028000)
libfreexl.so.1 => /usr/lib/x86_64-linux-gnu/libfreexl.so.1 (0x00007f5c7501c000)
libproj.so.19 => /usr/lib/x86_64-linux-gnu/libproj.so.19 (0x00007f5c74ca7000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c74a89000)
libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f5c74967000)
libgeos_c.so.1 => /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007f5c7492b000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c74766000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c74760000)
libicuuc.so.67 => /usr/lib/x86_64-linux-gnu/libicuuc.so.67 (0x00007f5c74575000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5c7454d000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5c75d49000)
libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f5c744c7000)
libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f5c74439000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5c7426c000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5c74250000)
libgeos-3.9.0.so => /usr/lib/x86_64-linux-gnu/libgeos-3.9.0.so (0x00007f5c74040000)
libicudata.so.67 => /usr/lib/x86_64-linux-gnu/libicudata.so.67 (0x00007f5c72527000)
libwebp.so.6 => /usr/lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f5c724bc000)
libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f5c7241c000)
libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f5c7220e000)
libjpeg.so.62 => /usr/lib/x86_64-linux-gnu/libjpeg.so.62 (0x00007f5c72188000)
libdeflate.so.0 => /usr/lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007f5c7216c000)
libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f5c72144000)
libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f5c72125000)
librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f5c71f08000)
libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f5c71eda000)
libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f5c71ec5000)
libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f5c71e8d000)
libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f5c71ce0000)
libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f5c71c93000)
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f5c71bb3000)
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f5c71b7f000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f5c71b77000)
libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f5c71b23000)
liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f5c71b12000)
libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f5c7198e000)
libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f5c71955000)
libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f5c718d0000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f5c717b2000)
libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f5c71683000)
libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f5c71470000)
libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f5c71461000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f5c71458000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5c7143e000)
libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f5c71421000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f5c713fe000)
libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f5c713f4000)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804368372,https://api.github.com/repos/simonw/datasette/issues/1249,804368372,MDEyOklzc3VlQ29tbWVudDgwNDM2ODM3Mg==,9599,simonw,2021-03-22T20:22:43Z,2021-03-22T20:22:43Z,OWNER,"```dockerfile
FROM python:3.9.2-slim-buster as build
# software-properties-common provides add-apt-repository
RUN apt-get update && \
apt-get -y --no-install-recommends install software-properties-common && \
add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
apt-get update && \
apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/apt && \
rm -rf /var/lib/dpkg
RUN pip install datasette && \
find /usr/local/lib -name '__pycache__' | xargs rm -r && \
rm -rf /root/.cache/pip
EXPOSE 8001
CMD [""datasette""]
```
262 MB","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804363687,https://api.github.com/repos/simonw/datasette/issues/1249,804363687,MDEyOklzc3VlQ29tbWVudDgwNDM2MzY4Nw==,9599,simonw,2021-03-22T20:15:00Z,2021-03-22T20:15:00Z,OWNER,"```
RUN pip install datasette && \
find /usr/local/lib -name '__pycache__' | xargs rm -r
```
That dropped it to 265MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804360701,https://api.github.com/repos/simonw/datasette/issues/1249,804360701,MDEyOklzc3VlQ29tbWVudDgwNDM2MDcwMQ==,9599,simonw,2021-03-22T20:10:07Z,2021-03-22T20:10:07Z,OWNER,Adding `--no-install-recommends` dropped it to 275MB,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804347152,https://api.github.com/repos/simonw/datasette/issues/1249,804347152,MDEyOklzc3VlQ29tbWVudDgwNDM0NzE1Mg==,9599,simonw,2021-03-22T19:47:56Z,2021-03-22T19:48:03Z,OWNER,I wrote a bunch of tips on creating smaller Docker images here: https://simonwillison.net/2018/Nov/19/smaller-python-docker-images/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804344553,https://api.github.com/repos/simonw/datasette/issues/1249,804344553,MDEyOklzc3VlQ29tbWVudDgwNDM0NDU1Mw==,9599,simonw,2021-03-22T19:43:25Z,2021-03-22T19:43:25Z,OWNER,Does `--no-install-recommends` make a difference?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/pull/1271#issuecomment-804299406,https://api.github.com/repos/simonw/datasette/issues/1271,804299406,MDEyOklzc3VlQ29tbWVudDgwNDI5OTQwNg==,9599,simonw,2021-03-22T18:36:14Z,2021-03-22T21:49:27Z,OWNER,"This isn't actually working - the outer code attempts to send an `.interrupt()` call to the connection object via the `connections` thread-local, which doesn't work because it's a thread-local so the connection isn't visible to that code.
Need to figure out how to communicate with that thread properly.
Also a test that fails in this particular case would be a good idea!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,Use SQLite conn.interrupt() instead of sqlite_timelimit(),
https://github.com/simonw/datasette/pull/1271#issuecomment-804265042,https://api.github.com/repos/simonw/datasette/issues/1271,804265042,MDEyOklzc3VlQ29tbWVudDgwNDI2NTA0Mg==,9599,simonw,2021-03-22T17:45:45Z,2021-03-22T17:45:45Z,OWNER,"I can remove this code too:
https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L190-L192","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,Use SQLite conn.interrupt() instead of sqlite_timelimit(),
https://github.com/simonw/datasette/issues/1249#issuecomment-804263434,https://api.github.com/repos/simonw/datasette/issues/1249,804263434,MDEyOklzc3VlQ29tbWVudDgwNDI2MzQzNA==,9599,simonw,2021-03-22T17:43:25Z,2021-03-22T17:43:25Z,OWNER,I figured out the cause of the hang in #1268 - it was caused by `select count(*) from SpatialIndex` interacting badly with the `set_progress_handler()` mechanism I was using to implement query time limits. #1271 has a replacement for that using `asyncio.wait_for()` and `conn.interrupt()` which should resolve the SpatiaLite issue too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1268#issuecomment-804261915,https://api.github.com/repos/simonw/datasette/issues/1268,804261915,MDEyOklzc3VlQ29tbWVudDgwNDI2MTkxNQ==,9599,simonw,2021-03-22T17:41:12Z,2021-03-22T17:41:12Z,OWNER,"Closing this because I've figured out the root of the problem now, and I have a potential solution.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1269#issuecomment-804261610,https://api.github.com/repos/simonw/datasette/issues/1269,804261610,MDEyOklzc3VlQ29tbWVudDgwNDI2MTYxMA==,9599,simonw,2021-03-22T17:40:41Z,2021-03-22T17:40:41Z,OWNER,"#1270 looks promising, and I don't want to leave open a security hole where someone could potentially hang Datasette with a nasty `count(*)` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837348479,Don't attempt to run count(*) against virtual tables,
https://github.com/simonw/datasette/pull/1271#issuecomment-804261103,https://api.github.com/repos/simonw/datasette/issues/1271,804261103,MDEyOklzc3VlQ29tbWVudDgwNDI2MTEwMw==,22429695,codecov[bot],2021-03-22T17:39:57Z,2021-03-22T17:39:57Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=h1) Report
> Merging [#1271](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=desc) (fb2ad7a) into [main](https://codecov.io/gh/simonw/datasette/commit/c4f1ec7f33fd7d5b93f0f895dafb5351cc3bfc5b?el=desc) (c4f1ec7) will **decrease** coverage by `0.28%`.
> The diff coverage is `94.28%`.
[](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #1271 +/- ##
==========================================
- Coverage 91.51% 91.22% -0.29%
==========================================
Files 34 34
Lines 4255 4263 +8
==========================================
- Hits 3894 3889 -5
- Misses 361 374 +13
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1271/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.41% <94.28%> (-0.52%)` | :arrow_down: |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1271/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `92.24% <0.00%> (-1.90%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=footer). Last update [c4f1ec7...fb2ad7a](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,Use SQLite conn.interrupt() instead of sqlite_timelimit(),
https://github.com/simonw/datasette/issues/1270#issuecomment-804255633,https://api.github.com/repos/simonw/datasette/issues/1270,804255633,MDEyOklzc3VlQ29tbWVudDgwNDI1NTYzMw==,9599,simonw,2021-03-22T17:32:02Z,2021-03-22T17:32:08Z,OWNER,Confirmed that the `interrupt()` based cancellation mechanism fixes the SpatiaLite issue in #1268!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),
https://github.com/simonw/datasette/issues/1270#issuecomment-803834784,https://api.github.com/repos/simonw/datasette/issues/1270,803834784,MDEyOklzc3VlQ29tbWVudDgwMzgzNDc4NA==,9599,simonw,2021-03-22T07:31:57Z,2021-03-22T16:22:19Z,OWNER,"I think the implementation for this goes here: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L146-L157
I figured out a similar pattern in `datasette-ripgrep` here: https://github.com/simonw/datasette-ripgrep/blob/0.7/datasette_ripgrep/__init__.py#L63-L71","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),
https://github.com/simonw/datasette/issues/1268#issuecomment-803802957,https://api.github.com/repos/simonw/datasette/issues/1268,803802957,MDEyOklzc3VlQ29tbWVudDgwMzgwMjk1Nw==,9599,simonw,2021-03-22T06:38:14Z,2021-03-22T06:38:14Z,OWNER,"Also worth trying is to change this code:
```python
n = 1000
if ms < 50:
n = 1
```
What happens with `n = 10` instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1249#issuecomment-803700940,https://api.github.com/repos/simonw/datasette/issues/1249,803700940,MDEyOklzc3VlQ29tbWVudDgwMzcwMDk0MA==,9599,simonw,2021-03-22T01:14:24Z,2021-03-22T01:14:24Z,OWNER,I tried that with just `python3-pip` (removing `libsqlite3-mod-spatialite`) and got 435MB.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803700626,https://api.github.com/repos/simonw/datasette/issues/1249,803700626,MDEyOklzc3VlQ29tbWVudDgwMzcwMDYyNg==,9599,simonw,2021-03-22T01:13:04Z,2021-03-22T01:13:04Z,OWNER,"Building a Dockerfile containing just `FROM ubuntu:20.10` gave me `79.5MB`.
Building this one:
```dockerfile
FROM ubuntu:20.10
# Setup build dependencies
RUN apt update && \
apt install -y python3-pip libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
```
Resulted in a 515MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1269#issuecomment-803785808,https://api.github.com/repos/simonw/datasette/issues/1269,803785808,MDEyOklzc3VlQ29tbWVudDgwMzc4NTgwOA==,9599,simonw,2021-03-22T06:00:53Z,2021-03-22T06:00:53Z,OWNER,This may not be necessary if using `.interrupt() for SQLite timeouts in #1270 works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837348479,Don't attempt to run count(*) against virtual tables,
https://github.com/simonw/datasette/issues/1268#issuecomment-803784902,https://api.github.com/repos/simonw/datasette/issues/1268,803784902,MDEyOklzc3VlQ29tbWVudDgwMzc4NDkwMg==,9599,simonw,2021-03-22T05:59:06Z,2021-03-22T05:59:06Z,OWNER,"Even if I implement that workaround in #1269 I'm concerned that this could still allow users to deliberately crash Datasette (if it's running SpatiaLite 5.0) by executing `select count(*) from SpatialIndex`.
That `interrupt` timeout mechanism is worth digging into further.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803782705,https://api.github.com/repos/simonw/datasette/issues/1268,803782705,MDEyOklzc3VlQ29tbWVudDgwMzc4MjcwNQ==,9599,simonw,2021-03-22T05:54:19Z,2021-03-22T05:54:19Z,OWNER,"Got two new TILs out of this:
* [Tracing every executed Python statement](https://til.simonwillison.net/python/tracing-every-statement)
* [Running gdb against a Python process in a running Docker container](https://til.simonwillison.net/docker/gdb-python-docker)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803777724,https://api.github.com/repos/simonw/datasette/issues/1268,803777724,MDEyOklzc3VlQ29tbWVudDgwMzc3NzcyNA==,9599,simonw,2021-03-22T05:42:50Z,2021-03-22T05:43:23Z,OWNER,"![]()
If I want to avoid counting virtual tables, I need to detect which tables are virtual tables.
The safest way to do this is probably to pull the `sql` for every table and then, in Python, check for values that start with `create virtual table` after converting to lower case, using any number of spaces.
This would catch things like ` CREATE virtual TABLE` which might be missed by a SQL `like` query. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803775121,https://api.github.com/repos/simonw/datasette/issues/1268,803775121,MDEyOklzc3VlQ29tbWVudDgwMzc3NTEyMQ==,9599,simonw,2021-03-22T05:36:26Z,2021-03-22T05:36:26Z,OWNER,So one fix could be to avoid running counts for anything that turns out to be a virtual table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803774926,https://api.github.com/repos/simonw/datasette/issues/1268,803774926,MDEyOklzc3VlQ29tbWVudDgwMzc3NDkyNg==,9599,simonw,2021-03-22T05:35:56Z,2021-03-22T05:35:56Z,OWNER,That's in this code here: https://github.com/simonw/datasette/blob/c4f1ec7f33fd7d5b93f0f895dafb5351cc3bfc5b/datasette/database.py#L221-L241,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803774518,https://api.github.com/repos/simonw/datasette/issues/1268,803774518,MDEyOklzc3VlQ29tbWVudDgwMzc3NDUxOA==,9599,simonw,2021-03-22T05:34:57Z,2021-03-22T05:34:57Z,OWNER,"... and sure enough, adding this code fixed the problem:
```diff
diff --git a/datasette/database.py b/datasette/database.py
index 3579cce..b466b12 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -224,6 +226,9 @@ class Database:
# Try to get counts for each table, $limit timeout for each count
counts = {}
for table in await self.table_names():
+ if table == ""SpatialIndex"":
+ counts[table] = 0
+ continue
try:
table_count = (
await self.execute(
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803773484,https://api.github.com/repos/simonw/datasette/issues/1268,803773484,MDEyOklzc3VlQ29tbWVudDgwMzc3MzQ4NA==,9599,simonw,2021-03-22T05:32:29Z,2021-03-22T05:32:29Z,OWNER,"To figure out which SQL query triggers the problem I added this code to write to a log file:
```python
with sqlite_timelimit(conn, time_limit_ms):
try:
cursor = conn.cursor()
with open(""/tmp/sql.log"", ""ab"", buffering=0) as fp:
fp.write((""{}: {}\n"".format(sql, params)).encode(""utf-8""))
cursor.execute(sql, params if params is not None else {})
```
I had to use `ab` binary mode because Python doesn't allow `buffering=0` for non-binary file operations.
With the log enabled, I used `docker exec -it 589ae68de943 bash` to attach to the running container and `tail -f /tmp/sql.log` to see the logs. Here's where it broke:
```
select count(*) from [idx_civici_geom_parent]: None
select count(*) from [sqlite_stat1]: None
select count(*) from [sqlite_stat3]: None
select count(*) from [SpatialIndex]: None
```
So attempting to run a `count(*)` against the `SpatialIndex` virtual table is the thing that triggers the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803764919,https://api.github.com/repos/simonw/datasette/issues/1268,803764919,MDEyOklzc3VlQ29tbWVudDgwMzc2NDkxOQ==,9599,simonw,2021-03-22T05:11:11Z,2021-03-22T05:11:11Z,OWNER,"Maybe I could implement SQLite query timeouts using the `interrupt()` method instead of the progress handler hack I'm currently using?
https://stackoverflow.com/questions/43240496/python-sqlite3-how-to-quickly-and-cleanly-interrupt-long-running-query-with-e has some tips.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803764200,https://api.github.com/repos/simonw/datasette/issues/1268,803764200,MDEyOklzc3VlQ29tbWVudDgwMzc2NDIwMA==,9599,simonw,2021-03-22T05:09:13Z,2021-03-22T05:09:13Z,OWNER,"I tried building a container where the `conn.set_progress_handler(handler, n)` line was commented out... and it fixed the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803762969,https://api.github.com/repos/simonw/datasette/issues/1268,803762969,MDEyOklzc3VlQ29tbWVudDgwMzc2Mjk2OQ==,9599,simonw,2021-03-22T05:05:51Z,2021-03-22T05:05:51Z,OWNER,I had to run `docker kill 16197781a7b5` to kill the broken container - Ctrl+C in the Datasette console window didn't do anything.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803762609,https://api.github.com/repos/simonw/datasette/issues/1268,803762609,MDEyOklzc3VlQ29tbWVudDgwMzc2MjYwOQ==,9599,simonw,2021-03-22T05:05:00Z,2021-03-22T05:05:00Z,OWNER,"Using https://til.simonwillison.net/docker/attach-bash-to-running-container - I figured out how to run `gdb`. I had to use `--privileged` here because otherwise `gdb` showed a ""Could not attach to process"" error.
```
docker exec --privileged -it 16197781a7b5 bash
# apt-get install gdb python3-dbg
# gdb /usr/bin/python3 -p 20
```
This paused the process. I tried running this:
```
(gdb) py-bt
Traceback (most recent call first):
File ""/usr/lib/python3.8/asyncio/base_events.py"", line 1845, in _run_once
if handle._cancelled:
File ""/usr/lib/python3.8/asyncio/base_events.py"", line 570, in run_forever
self._run_once()
File ""/usr/lib/python3.8/asyncio/base_events.py"", line 603, in run_until_complete
self.run_forever()
File ""/usr/local/lib/python3.8/dist-packages/uvicorn/server.py"", line 49, in run
loop.run_until_complete(self.serve(sockets=sockets))
File ""/usr/local/lib/python3.8/dist-packages/uvicorn/main.py"", line 386, in run
server.run()
File ""/usr/local/lib/python3.8/dist-packages/datasette/cli.py"", line 575, in serve
uvicorn.run(ds.app(), **uvicorn_kwargs)
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/usr/local/bin/datasette"", line 8, in
sys.exit(cli())
File ""/usr/lib/python3.8/trace.py"", line 450, in runctx
exec(cmd, globals, locals)
File ""/usr/lib/python3.8/trace.py"", line 6632, in main
File ""/usr/lib/python3.8/trace.py"", line 756, in
main()
File ""/usr/lib/python3.8/runpy.py"", line 343, in _run_code
File ""/usr/lib/python3.8/runpy.py"", line 450, in _run_module_as_main
```
Not sure if that's useful or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803759051,https://api.github.com/repos/simonw/datasette/issues/1268,803759051,MDEyOklzc3VlQ29tbWVudDgwMzc1OTA1MQ==,9599,simonw,2021-03-22T04:55:22Z,2021-03-22T04:55:22Z,OWNER,So I think there's a bug in the way the `set_progress_handler()` mechanism works when used in conjunction with SpatiaLite 5.0 on Linux.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803758793,https://api.github.com/repos/simonw/datasette/issues/1268,803758793,MDEyOklzc3VlQ29tbWVudDgwMzc1ODc5Mw==,9599,simonw,2021-03-22T04:54:32Z,2021-03-22T04:54:32Z,OWNER,"Hitting http://localhost:8001/tuscany_housenumbers triggers the bug. It gets stuck in a loop that looks like this:
![]()
Which looks to me like this code: https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L139-L158","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803758182,https://api.github.com/repos/simonw/datasette/issues/1268,803758182,MDEyOklzc3VlQ29tbWVudDgwMzc1ODE4Mg==,9599,simonw,2021-03-22T04:52:15Z,2021-03-22T04:52:15Z,OWNER,Hitting http://localhost:8001/ successfully shows the homepage (after a lot more scrolling).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803757746,https://api.github.com/repos/simonw/datasette/issues/1268,803757746,MDEyOklzc3VlQ29tbWVudDgwMzc1Nzc0Ng==,9599,simonw,2021-03-22T04:50:40Z,2021-03-22T04:51:52Z,OWNER,"Here's a fun debugging trick:
docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash
root@16197781a7b5:/# python3 -m trace --trace $(which datasette) \
-p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite \
--load-extension=spatialite
A huge amount of stuff scrolls past as Datasette starts up, since we are tracing every executed line of Python.
After about a minute it's finished starting and gets to this point:
```
selectors.py(452): if timeout is None:
selectors.py(454): elif timeout <= 0:
selectors.py(459): timeout = math.ceil(timeout * 1e3) * 1e-3
selectors.py(464): max_ev = max(len(self._fd_to_key), 1)
selectors.py(466): ready = []
selectors.py(467): try:
selectors.py(468): fd_event_list = self._selector.poll(timeout, max_ev)
```
Now I can make some HTTP requests against it.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803756495,https://api.github.com/repos/simonw/datasette/issues/1268,803756495,MDEyOklzc3VlQ29tbWVudDgwMzc1NjQ5NQ==,9599,simonw,2021-03-22T04:46:04Z,2021-03-22T04:46:04Z,OWNER,`gdb` may be able to help debug this: https://www.podoliaka.org/2016/04/10/debugging-cpython-gdb/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1249#issuecomment-803755698,https://api.github.com/repos/simonw/datasette/issues/1249,803755698,MDEyOklzc3VlQ29tbWVudDgwMzc1NTY5OA==,9599,simonw,2021-03-22T04:43:02Z,2021-03-22T04:43:02Z,OWNER,I'll spin off a separate ticket to investigate the hang.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1267#issuecomment-803754226,https://api.github.com/repos/simonw/datasette/issues/1267,803754226,MDEyOklzc3VlQ29tbWVudDgwMzc1NDIyNg==,9599,simonw,2021-03-22T04:37:26Z,2021-03-22T04:37:26Z,OWNER,"Thanks for doing this - I've used alternativeto.net a bunch in the past, it's great to see Datasette listed there.
This does raise some interesting philosophical questions: three years into the project I'm still not entirely sure what Datasette competes with! Could be SQLite desktop packages, could be visualization software like Tableau, could even be something like Airtable (given a few more plugins).
It will be interesting to see how the alternativeto listing evolves, maybe it will help me answer that question!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837208901,Update Datasette alternativeto listening with details,
https://github.com/simonw/datasette/issues/1249#issuecomment-803753388,https://api.github.com/repos/simonw/datasette/issues/1249,803753388,MDEyOklzc3VlQ29tbWVudDgwMzc1MzM4OA==,9599,simonw,2021-03-22T04:34:20Z,2021-03-22T04:35:10Z,OWNER,"Well this is frustrating. I finally found a Dockerfile that worked and installed an Ubuntu pre-compiled SpatiaLite module that would load...
```dockerfile
FROM ubuntu:20.10 as install_spatialite
RUN apt update && \
apt install -y libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
FROM ubuntu:20.10
RUN apt update && \
apt install -y python3-pip && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
RUN pip install datasette
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/
ENV LD_LIBRARY_PATH=/usr/local/lib
EXPOSE 8001
CMD [""datasette""]
```
(Which produced a 550MB image)
And when I ran Datasette I got that same error where the database listing page hangs!
```
docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite --load-extension=spatialite
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803751068,https://api.github.com/repos/simonw/datasette/issues/1249,803751068,MDEyOklzc3VlQ29tbWVudDgwMzc1MTA2OA==,9599,simonw,2021-03-22T04:26:45Z,2021-03-22T04:26:45Z,OWNER,"Here's why:
```
datasette % docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash
root@3430352ff378:/# datasette
bash: /usr/local/bin/datasette: /usr/bin/python3: bad interpreter: No such file or directory
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803750617,https://api.github.com/repos/simonw/datasette/issues/1249,803750617,MDEyOklzc3VlQ29tbWVudDgwMzc1MDYxNw==,9599,simonw,2021-03-22T04:25:14Z,2021-03-22T04:25:14Z,OWNER,"Got this error attempting to run Datasette (with or without SpatiaLite):
```
standard_init_linux.go:219: exec user process caused: no such file or directory
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803750399,https://api.github.com/repos/simonw/datasette/issues/1249,803750399,MDEyOklzc3VlQ29tbWVudDgwMzc1MDM5OQ==,9599,simonw,2021-03-22T04:24:25Z,2021-03-22T04:24:25Z,OWNER,"I'll try using `ubuntu:20.10` for everything:
```dockerfile
FROM ubuntu:20.10 as install_spatialite
RUN apt update && \
apt install -y libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
FROM ubuntu:20.10 as build
RUN apt update && \
apt install -y python3-pip && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
RUN pip install datasette
#COPY . /datasette
#RUN pip install /datasette
FROM ubuntu:20.10
# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so
ENV LD_LIBRARY_PATH=/usr/local/lib
EXPOSE 8001
CMD [""datasette""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803749831,https://api.github.com/repos/simonw/datasette/issues/1249,803749831,MDEyOklzc3VlQ29tbWVudDgwMzc0OTgzMQ==,9599,simonw,2021-03-22T04:22:35Z,2021-03-22T04:22:35Z,OWNER,"I tried copying just the `mod_spatialite.so` file:
```dockerfile
FROM ubuntu:20.10 as install_spatialite
RUN apt update && \
apt install -y libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
FROM python:3.9.2-slim as build
RUN pip install datasette
#COPY . /datasette
#RUN pip install /datasette
FROM python:3.9.2-slim
# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so
ENV LD_LIBRARY_PATH=/usr/local/lib
EXPOSE 8001
CMD [""datasette""]
```
But when I ran Datasette with `--load-extension=spatialite` I got this:
```
File ""/usr/local/lib/python3.9/site-packages/datasette/database.py"", line 151, in in_thread
self.ds._prepare_connection(conn, self.name)
File ""/usr/local/lib/python3.9/site-packages/datasette/app.py"", line 502, in _prepare_connection
conn.execute(f""SELECT load_extension('{extension}')"")
sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803748469,https://api.github.com/repos/simonw/datasette/issues/1249,803748469,MDEyOklzc3VlQ29tbWVudDgwMzc0ODQ2OQ==,9599,simonw,2021-03-22T04:17:51Z,2021-03-22T04:17:51Z,OWNER,"... except my clever image using SpatiaLite installed for Ubuntu doesn't actually work:
```
datasette % docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
File ""/usr/local/lib/python3.9/sqlite3/dbapi2.py"", line 27, in
from _sqlite3 import *
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /usr/lib/x86_64-linux-gnu/libsqlite3.so.0)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803748158,https://api.github.com/repos/simonw/datasette/issues/1249,803748158,MDEyOklzc3VlQ29tbWVudDgwMzc0ODE1OA==,9599,simonw,2021-03-22T04:16:57Z,2021-03-22T04:16:57Z,OWNER,"Which is great, because the image on Docker Hub right now is 383MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803747701,https://api.github.com/repos/simonw/datasette/issues/1249,803747701,MDEyOklzc3VlQ29tbWVudDgwMzc0NzcwMQ==,9599,simonw,2021-03-22T04:15:40Z,2021-03-22T04:15:40Z,OWNER,"Here's a trick: install SpatiaLite in `ubuntu:20.10` and then copy it into the final `python:3.9.2-slim` image.
```dockerfile
FROM ubuntu:20.10 as install_spatialite
RUN apt update && \
apt install -y libsqlite3-mod-spatialite && \
apt clean && \
rm -rf /var/lib/{apt,dpkg,cache,log}/
FROM python:3.9.2-slim as build
RUN pip install datasette
#COPY . /datasette
#RUN pip install /datasette
FROM python:3.9.2-slim
# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu
ENV LD_LIBRARY_PATH=/usr/local/lib
EXPOSE 8001
CMD [""datasette""]
```
That produced a 265MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/942#issuecomment-803631102,https://api.github.com/repos/simonw/datasette/issues/942,803631102,MDEyOklzc3VlQ29tbWVudDgwMzYzMTEwMg==,192568,mroswell,2021-03-21T17:48:42Z,2021-03-21T17:48:42Z,CONTRIBUTOR,"I like this idea. Though it might be nice to have some kind of automated system from database to file, so that developers could easily track diffs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/1249#issuecomment-803698983,https://api.github.com/repos/simonw/datasette/issues/1249,803698983,MDEyOklzc3VlQ29tbWVudDgwMzY5ODk4Mw==,9599,simonw,2021-03-22T01:05:36Z,2021-03-22T01:06:23Z,OWNER,"It's pretty big though. I tried this version which avoids copying junk from my laptop in:
```dockerfile
FROM ubuntu:20.10
# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean
RUN pip install datasette
EXPOSE 8001
CMD [""datasette""]
```
And got this:
```
datasette % docker images datasette-spatialite
REPOSITORY TAG IMAGE ID CREATED SIZE
datasette-spatialite latest 0796950653c2 2 seconds ago 528MB
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803698168,https://api.github.com/repos/simonw/datasette/issues/1249,803698168,MDEyOklzc3VlQ29tbWVudDgwMzY5ODE2OA==,9599,simonw,2021-03-22T01:02:02Z,2021-03-22T01:02:30Z,OWNER,"This is the shortest Dockerfile that appeared to give me a working SpatiaLite module:
```dockerfile
FROM ubuntu:20.10
# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean
# Add local code to the image instead of fetching from pypi.
COPY . /datasette
RUN pip install /datasette
RUN rm -rf /datasette
EXPOSE 8001
CMD [""datasette""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803697546,https://api.github.com/repos/simonw/datasette/issues/1249,803697546,MDEyOklzc3VlQ29tbWVudDgwMzY5NzU0Ng==,9599,simonw,2021-03-22T00:59:47Z,2021-03-22T00:59:47Z,OWNER,"To debug I'm running:
docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash
This gets me a shell I can use.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803697211,https://api.github.com/repos/simonw/datasette/issues/1249,803697211,MDEyOklzc3VlQ29tbWVudDgwMzY5NzIxMQ==,9599,simonw,2021-03-22T00:58:01Z,2021-03-22T00:58:01Z,OWNER,"I'm messing around with the `Dockerfile` and after each change I'm running:
docker build . -t datasette-spatialite
And then:
docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803694661,https://api.github.com/repos/simonw/datasette/issues/1249,803694661,MDEyOklzc3VlQ29tbWVudDgwMzY5NDY2MQ==,9599,simonw,2021-03-22T00:46:49Z,2021-03-22T00:46:49Z,OWNER,Actually for the loadable module I think I need https://packages.ubuntu.com/groovy/libsqlite3-mod-spatialite,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803694436,https://api.github.com/repos/simonw/datasette/issues/1249,803694436,MDEyOklzc3VlQ29tbWVudDgwMzY5NDQzNg==,9599,simonw,2021-03-22T00:46:00Z,2021-03-22T00:46:00Z,OWNER,So I'm going to try `20.10` and see where that gets me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803694359,https://api.github.com/repos/simonw/datasette/issues/1249,803694359,MDEyOklzc3VlQ29tbWVudDgwMzY5NDM1OQ==,9599,simonw,2021-03-22T00:45:47Z,2021-03-22T00:45:47Z,OWNER,https://pythonspeed.com/articles/base-image-python-docker-images/ suggests using `python:3.9-slim-buster` or `ubuntu:20.04` - but 20.04 is focal which still has SpatiaLite `4.3.0a-6build1` - It's `20.10` that has 5.0: https://packages.ubuntu.com/groovy/libspatialite-dev,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803693181,https://api.github.com/repos/simonw/datasette/issues/1249,803693181,MDEyOklzc3VlQ29tbWVudDgwMzY5MzE4MQ==,9599,simonw,2021-03-22T00:41:02Z,2021-03-22T00:41:02Z,OWNER,Debian sid has it too: https://packages.debian.org/sid/libspatialite-dev,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803692673,https://api.github.com/repos/simonw/datasette/issues/1249,803692673,MDEyOklzc3VlQ29tbWVudDgwMzY5MjY3Mw==,9599,simonw,2021-03-22T00:38:42Z,2021-03-22T00:38:42Z,OWNER,Ubuntu Groovy has a package for SpatiaLite 5 - I could try using that instead: https://packages.ubuntu.com/groovy/libspatialite-dev,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803691236,https://api.github.com/repos/simonw/datasette/issues/1249,803691236,MDEyOklzc3VlQ29tbWVudDgwMzY5MTIzNg==,9599,simonw,2021-03-22T00:32:03Z,2021-03-22T00:32:03Z,OWNER,"Here's something odd: when I run `datasette tuscany_housenumbers.sqlite --load-extension=spatialite` on macOS against SpatiaLite installed using Homebrew (which reports `""spatialite"": ""5.0.0""` on the `/-/versions` page) I don't get any weird errors at all, everything works fine.
But when I tried compiling SpatiaLite inside the Docker container I had hanging errors on some pages.
This is using https://www.gaia-gis.it/gaia-sins/knn/tuscany_housenumbers.7z from the SpatiaLite KNN tutorial at https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1259#issuecomment-803674728,https://api.github.com/repos/simonw/datasette/issues/1259,803674728,MDEyOklzc3VlQ29tbWVudDgwMzY3NDcyOA==,9599,simonw,2021-03-21T22:55:31Z,2021-03-21T22:55:31Z,OWNER,CTEs were added in 2014-02-03 SQLite 3.8.3 - so I think it's OK to depend on them for Datasette.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830567275,Research using CTEs for faster facet counts,
https://github.com/simonw/datasette/issues/647#issuecomment-803673225,https://api.github.com/repos/simonw/datasette/issues/647,803673225,MDEyOklzc3VlQ29tbWVudDgwMzY3MzIyNQ==,9599,simonw,2021-03-21T22:44:19Z,2021-03-21T22:44:19Z,OWNER,"Now that I'm looking at refactoring how views work in #878 it's clear that the gnarliest, most convoluted code I need to deal with relates to this old feature.
I'm going to remove it entirely. Any performance enhancement or provides can be achieved just as well by using regular URLs and a caching proxy.
I may provide a 404 handling plugin that attempts to rewrite old URLs that used this mechanism, but I won't do any more than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin,
https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803502424,https://api.github.com/repos/simonw/sqlite-utils/issues/249,803502424,MDEyOklzc3VlQ29tbWVudDgwMzUwMjQyNA==,36287,prabhur,2021-03-21T02:43:32Z,2021-03-21T02:43:32Z,NONE,"> Did you run `enable-fts` before you inserted the data?
>
> If so you'll need to run `populate-fts` after the insert to populate the FTS index.
>
> A better solution may be to add `--create-triggers` to the `enable-fts` command to add triggers that will automatically keep the index updated as you insert new records.
Wow. Wasn't expecting a response this quick, especially during a weekend. :-) Sincerely appreciate it.
I tried the `populate-fts` and that did the trick. My bad for not consulting the docs again. I think I forgot to add that step when I automated the workflow.
Thanks for the suggestion. I'll close this issue. Have a great weekend and many many thanks for creating these suite of tools around sqlite.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836963850,Full text search possibly broken?,
https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803501756,https://api.github.com/repos/simonw/sqlite-utils/issues/249,803501756,MDEyOklzc3VlQ29tbWVudDgwMzUwMTc1Ng==,9599,simonw,2021-03-21T02:33:45Z,2021-03-21T02:33:45Z,OWNER,"Did you run `enable-fts` before you inserted the data?
If so you'll need to run `populate-fts` after the insert to populate the FTS index.
A better solution may be to add `--create-triggers` to the `enable-fts` command to add triggers that will automatically keep the index updated as you insert new records.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836963850,Full text search possibly broken?,
https://github.com/simonw/datasette/issues/1261#issuecomment-803499509,https://api.github.com/repos/simonw/datasette/issues/1261,803499509,MDEyOklzc3VlQ29tbWVudDgwMzQ5OTUwOQ==,812795,brimstone,2021-03-21T02:06:43Z,2021-03-21T02:06:43Z,NONE,I can confirm 0.9.2 fixes the problem. Thanks for the fast response!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832092321,Some links aren't properly URL encoded.,
https://github.com/simonw/datasette/issues/878#issuecomment-803473015,https://api.github.com/repos/simonw/datasette/issues/878,803473015,MDEyOklzc3VlQ29tbWVudDgwMzQ3MzAxNQ==,9599,simonw,2021-03-20T22:33:05Z,2021-03-20T22:33:05Z,OWNER,"Things this mechanism needs to be able to support:
- Returning a default JSON representation
- Defining ""extra"" JSON representations blocks, which can be requested using `?_extra=`
- Returning rendered HTML, based on the default JSON + one or more extras + a template
- Using Datasette output renderers to return e.g. CSV data
- Potentially also supporting streaming output renderers for streaming CSV/TSV/JSON-nl etc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-803472595,https://api.github.com/repos/simonw/datasette/issues/878,803472595,MDEyOklzc3VlQ29tbWVudDgwMzQ3MjU5NQ==,9599,simonw,2021-03-20T22:28:12Z,2021-03-20T22:28:12Z,OWNER,"Another idea I had: a view is a class that takes the `datasette` instance in its constructor, and defines a `__call__` method that accepts a request and returns a response. Except `await __call__` looks like it might be a bit messy, discussion in https://github.com/encode/starlette/issues/886","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-803472278,https://api.github.com/repos/simonw/datasette/issues/878,803472278,MDEyOklzc3VlQ29tbWVudDgwMzQ3MjI3OA==,9599,simonw,2021-03-20T22:25:04Z,2021-03-20T22:25:04Z,OWNER,"I came up with a slightly wild idea for this that would involve pytest-style dependency injection.
Prototype here: https://gist.github.com/simonw/496b24fdad44f6f8b7237fe394a0ced7
Copying from my private notes:
> Use the lazy evaluated DI mechanism to break up table view into different pieces eg for faceting
>
> Use that to solve JSON vs HTML views
>
> Oh here's an idea: what if the various components of the table view were each defined as async functions.... and then executed using asyncio.gather in order to run the SQL queries in parallel? Then try benchmarking with different numbers of threads?
>
> The async_call_with_arguments function could do this automatically for any awaitable dependencies
>
> This would give me massively parallel dependency injection
>
> (I could build an entire framework around this and call it c64)
>
> Idea: arguments called eg ""count"" are executed and the result passed to the function. If called count_fn then a reference to the not-yet-called function is passed instead
>
> I'm not going to completely combine the views mechanism and the render hooks. Instead, the core view will define a bunch of functions used to compose the page and the render hook will have conditional access to those functions - which will otherwise be asyncio.gather executed directly by the HTML page version
>
> Using asyncio.gather to execute facets and suggest facets in parallel would be VERY interesting
>
> suggest facets should be VERY cachable - doesn't matter if it's wrong unlike actual facets themselves
>
> What if all Datasette views were defined in terms of dependency injection - and those dependency functions could themselves depend on others just like pytest fixtures. Everything would become composable and async stuff could execute in parallel
>
> FURTHER IDEA: use this for the ?_extra= mechanism as well.
>
> Any view in Datasette can be defined as a collection of named keys. Each of those keys maps to a function or an async function that accepts as input other named keys, using DI to handle them.
>
> The HTML view is a defined function. So are the other outputs.
>
> Default original inputs include “request” and “datasette”.
>
> So… maybe a view function is a class methods that use DI. One of those methods as an .html() method used for the default page.
>
> Output formats are a bit more complicated because they are supposed to be defined separately in plugins. They are unified across query, row and table though.
>
> I’m going to try breaking up the TableView to see what happens.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-803471917,https://api.github.com/repos/simonw/datasette/issues/878,803471917,MDEyOklzc3VlQ29tbWVudDgwMzQ3MTkxNw==,9599,simonw,2021-03-20T22:21:33Z,2021-03-20T22:21:33Z,OWNER,"This has been blocking things for too long.
If this becomes a documented pattern, things like adding a JSON output to https://github.com/dogsheep/dogsheep-beta becomes easier too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/1258#issuecomment-803471702,https://api.github.com/repos/simonw/datasette/issues/1258,803471702,MDEyOklzc3VlQ29tbWVudDgwMzQ3MTcwMg==,9599,simonw,2021-03-20T22:19:39Z,2021-03-20T22:19:39Z,OWNER,"This is a good idea. I avoided this initially because it should be possible to run a canned query with a parameter set to the empty string, but that view could definitely be smart enough to differentiate between `?sql=...¶m=` and `?sql=` with no `param` specified at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421,Allow canned query params to specify default values,
https://github.com/simonw/datasette/issues/782#issuecomment-803469623,https://api.github.com/repos/simonw/datasette/issues/782,803469623,MDEyOklzc3VlQ29tbWVudDgwMzQ2OTYyMw==,9599,simonw,2021-03-20T22:01:23Z,2021-03-20T22:01:23Z,OWNER,"I'm going to keep `?_shape=array` working on the assumption that many existing uses of the Datasette API are already using that option, so it would be nice not to break them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1261#issuecomment-803468314,https://api.github.com/repos/simonw/datasette/issues/1261,803468314,MDEyOklzc3VlQ29tbWVudDgwMzQ2ODMxNA==,9599,simonw,2021-03-20T21:48:48Z,2021-03-20T21:48:48Z,OWNER,That's fixed in this release of `datasette-publish-vercel`: https://github.com/simonw/datasette-publish-vercel/releases/tag/0.9.2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832092321,Some links aren't properly URL encoded.,
https://github.com/simonw/datasette/issues/1261#issuecomment-803466868,https://api.github.com/repos/simonw/datasette/issues/1261,803466868,MDEyOklzc3VlQ29tbWVudDgwMzQ2Njg2OA==,9599,simonw,2021-03-20T21:36:06Z,2021-03-20T21:36:06Z,OWNER,"This isn't a Datasette bug - it's a Vercel bug: https://github.com/simonw/datasette-publish-vercel/issues/28
I'm looking at a fix for that now, so watch that issue for updates.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832092321,Some links aren't properly URL encoded.,
https://github.com/simonw/datasette/issues/1266#issuecomment-803466730,https://api.github.com/repos/simonw/datasette/issues/1266,803466730,MDEyOklzc3VlQ29tbWVudDgwMzQ2NjczMA==,9599,simonw,2021-03-20T21:35:00Z,2021-03-20T21:35:00Z,OWNER,https://docs.datasette.io/en/latest/internals.html#returning-a-response-with-asgi-send-send,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836273891,Documentation for Response.asgi_send(send) method,
https://github.com/simonw/datasette/issues/1265#issuecomment-803130332,https://api.github.com/repos/simonw/datasette/issues/1265,803130332,MDEyOklzc3VlQ29tbWVudDgwMzEzMDMzMg==,9599,simonw,2021-03-19T21:03:09Z,2021-03-19T21:03:09Z,OWNER,This is now available in `datasette-auth-passwords` 0.4! https://github.com/simonw/datasette-auth-passwords/releases/tag/0.4,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",836123030,Support for HTTP Basic Authentication,
https://github.com/simonw/datasette/issues/1265#issuecomment-803160804,https://api.github.com/repos/simonw/datasette/issues/1265,803160804,MDEyOklzc3VlQ29tbWVudDgwMzE2MDgwNA==,468612,yunzheng,2021-03-19T22:05:12Z,2021-03-19T22:05:12Z,NONE,Wow that was fast! Thanks for this very cool project and quick update! 👍 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836123030,Support for HTTP Basic Authentication,
https://github.com/simonw/datasette/issues/1265#issuecomment-802923254,https://api.github.com/repos/simonw/datasette/issues/1265,802923254,MDEyOklzc3VlQ29tbWVudDgwMjkyMzI1NA==,7476523,bobwhitelock,2021-03-19T15:39:15Z,2021-03-19T15:39:15Z,CONTRIBUTOR,"It doesn't use basic auth, but you can put a whole datasette instance, or parts of this, behind a username/password prompt using https://github.com/simonw/datasette-auth-passwords","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836123030,Support for HTTP Basic Authentication,
https://github.com/simonw/datasette/issues/1262#issuecomment-802164134,https://api.github.com/repos/simonw/datasette/issues/1262,802164134,MDEyOklzc3VlQ29tbWVudDgwMjE2NDEzNA==,19328961,henry501,2021-03-18T17:55:00Z,2021-03-18T17:55:00Z,NONE,"Thanks for the comments. I'll take a look at the documentation to familiarize myself, as I haven't tried to write any plugins yet. With some luck I might be ready to write it when the hook is implemented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",834602299,Plugin hook that could support 'order by random()' for table view,
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-802032152,https://api.github.com/repos/simonw/sqlite-utils/issues/159,802032152,MDEyOklzc3VlQ29tbWVudDgwMjAzMjE1Mg==,1025224,limar,2021-03-18T15:42:52Z,2021-03-18T15:42:52Z,NONE,"I confirm the bug. Happens for me in version 3.6. I use the call to delete all the records:
`table.delete_where()`
This does not delete anything.
I see that `delete()` method DOES use context manager `with self.db.conn:` which should help. You may want to align the code of both methods.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/datasette/issues/1262#issuecomment-802099264,https://api.github.com/repos/simonw/datasette/issues/1262,802099264,MDEyOklzc3VlQ29tbWVudDgwMjA5OTI2NA==,9599,simonw,2021-03-18T16:43:09Z,2021-03-18T16:43:09Z,OWNER,"I often find myself wanting this too, when I'm exploring a new dataset.
i agree with Bob that this is a good candidate for a plugin. The plugin system isn't quite setup for this yet though - there isn't an obvious mechanism for adding extra sort orders or other interface elements that manipulate the query used by the table view in some way.
I'm going to promote this issue to status of a plugin hook feature request - I have a hunch that a plugin hook that enables `order by random()` could enable a lot of other useful plugin features too.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",834602299,Plugin hook that could support 'order by random()' for table view,
https://github.com/simonw/datasette/issues/1262#issuecomment-802095132,https://api.github.com/repos/simonw/datasette/issues/1262,802095132,MDEyOklzc3VlQ29tbWVudDgwMjA5NTEzMg==,7476523,bobwhitelock,2021-03-18T16:37:45Z,2021-03-18T16:37:45Z,CONTRIBUTOR,"This sounds like a good use case for a plugin, since this will only be useful for a subset of Datasette users. It shouldn't be too difficult to add a button to do this with the available plugin hooks - have you taken a look at https://docs.datasette.io/en/latest/writing_plugins.html?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",834602299,Plugin hook that could support 'order by random()' for table view,
https://github.com/simonw/sqlite-utils/issues/246#issuecomment-801816980,https://api.github.com/repos/simonw/sqlite-utils/issues/246,801816980,MDEyOklzc3VlQ29tbWVudDgwMTgxNjk4MA==,37962604,polyrand,2021-03-18T10:40:32Z,2021-03-18T10:43:04Z,NONE,"I have found a similar problem, but I only when using that type of query (with `*` for doing a prefix search). I'm also building something on top of FTS5/sqlite-utils, and the way I decided to handle it was creating a specific function for prefixes. According to [the docs](https://www2.sqlite.org/fts5.html#fts5_prefix_queries), the query can be done in this 2 ways:
```sql
... MATCH '""one two thr"" * '
... MATCH 'one + two + thr*'
```
I thought I could build a query like the first one using this function:
```python
def prefix(query: str):
return f'""{query}"" *'
```
And then I use the output of that function as the query parameter for the standard `.search()` method in sqlite-utils.
However, my use case is different because I'm the one ""deciding"" when to use a prefix search, not the end user. I also haven't done many tests, but maybe you found that useful. One thing I could think of is checking if the query has an `*` at the end, remove it and build the prefix query using the function above.
This is just for prefix queries, I think having the escaping function is still useful for other use cases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",831751367,Escaping FTS search strings,
https://github.com/simonw/datasette/issues/1394#issuecomment-880900534,https://api.github.com/repos/simonw/datasette/issues/1394,880900534,MDEyOklzc3VlQ29tbWVudDg4MDkwMDUzNA==,9599,simonw,2021-07-15T17:58:03Z,2021-07-15T17:58:03Z,OWNER,Started a conversation about this on the SQLite forum: https://sqlite.org/forum/forumpost/2d76f2bcf65d256a?t=h,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by,
https://github.com/simonw/datasette/issues/1396#issuecomment-880967052,https://api.github.com/repos/simonw/datasette/issues/1396,880967052,MDEyOklzc3VlQ29tbWVudDg4MDk2NzA1Mg==,9599,simonw,2021-07-15T19:47:25Z,2021-07-15T19:47:25Z,OWNER,Actually I'm going to close this now and re-open it if the problem occurs again in the future.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880326049,https://api.github.com/repos/simonw/datasette/issues/1396,880326049,MDEyOklzc3VlQ29tbWVudDg4MDMyNjA0OQ==,9599,simonw,2021-07-15T01:50:05Z,2021-07-15T01:50:05Z,OWNER,"I think I made a mistake in this commit: https://github.com/simonw/datasette/commit/0486303b60ce2784fd2e2ecdbecf304b7d6e6659
![]()
It looks like I copied `$VERSION_TAG` from here - but it's not available in the `publish.yml` flow: https://github.com/simonw/datasette/blob/0486303b60ce2784fd2e2ecdbecf304b7d6e6659/.github/workflows/push_docker_tag.yml#L18-L25","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880325362,https://api.github.com/repos/simonw/datasette/issues/1396,880325362,MDEyOklzc3VlQ29tbWVudDg4MDMyNTM2Mg==,9599,simonw,2021-07-15T01:48:11Z,2021-07-15T01:48:11Z,OWNER,In particular these three lines: https://github.com/simonw/datasette/blob/084cfe1e00e1a4c0515390a513aca286eeea20c2/.github/workflows/publish.yml#L117-L119,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880325004,https://api.github.com/repos/simonw/datasette/issues/1396,880325004,MDEyOklzc3VlQ29tbWVudDg4MDMyNTAwNA==,9599,simonw,2021-07-15T01:47:17Z,2021-07-15T01:47:17Z,OWNER,"This is the part of the publish workflow that failed and threw the ""invalid reference format"" error: https://github.com/simonw/datasette/blob/084cfe1e00e1a4c0515390a513aca286eeea20c2/.github/workflows/publish.yml#L100-L119","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880324637,https://api.github.com/repos/simonw/datasette/issues/1396,880324637,MDEyOklzc3VlQ29tbWVudDg4MDMyNDYzNw==,9599,simonw,2021-07-15T01:46:26Z,2021-07-15T01:46:26Z,OWNER,"I manually published the Docker image using https://github.com/simonw/datasette/actions/workflows/push_docker_tag.yml https://github.com/simonw/datasette/runs/3072505126
The 0.58 release shows up on https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated now - BUT the `latest` tag still points to a version from a month ago.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880374156,https://api.github.com/repos/simonw/datasette/issues/1396,880374156,MDEyOklzc3VlQ29tbWVudDg4MDM3NDE1Ng==,9599,simonw,2021-07-15T04:03:18Z,2021-07-15T04:03:18Z,OWNER,"I fixed `datasette:latest` by running the following on my laptop:
```
docker pull datasetteproject/datasette:0.58
docker tag datasetteproject/datasette:0.58 datasetteproject/datasette:latest
docker login -u datasetteproject -p ...
docker push datasetteproject/datasette:latest
```
Confirmed on https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated that `datasette:latest` and `datasette:0.58` both now have the same digest of `3b5ba478040e`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880372149,https://api.github.com/repos/simonw/datasette/issues/1396,880372149,MDEyOklzc3VlQ29tbWVudDg4MDM3MjE0OQ==,9599,simonw,2021-07-15T03:56:49Z,2021-07-15T03:56:49Z,OWNER,I'm going to leave this open until I next successfully publish a new version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1394#issuecomment-880287483,https://api.github.com/repos/simonw/datasette/issues/1394,880287483,MDEyOklzc3VlQ29tbWVudDg4MDI4NzQ4Mw==,9599,simonw,2021-07-15T00:01:47Z,2021-07-15T00:01:47Z,OWNER,"I wrote this code:
```python
_order_by_re = re.compile(r""(^.*) order by [a-zA-Z_][a-zA-Z0-9_]+( desc)?$"", re.DOTALL)
_order_by_braces_re = re.compile(r""(^.*) order by \[[^\]]+\]( desc)?$"", re.DOTALL)
def strip_order_by(sql):
for regex in (_order_by_re, _order_by_braces_re):
match = regex.match(sql)
if match is not None:
return match.group(1)
return sql
@pytest.mark.parametrize(
""sql,expected"",
[
(""blah"", ""blah""),
(""select * from foo"", ""select * from foo""),
(""select * from foo order by bah"", ""select * from foo""),
(""select * from foo order by bah desc"", ""select * from foo""),
(""select * from foo order by [select]"", ""select * from foo""),
(""select * from foo order by [select] desc"", ""select * from foo""),
],
)
def test_strip_order_by(sql, expected):
assert strip_order_by(sql) == expected
```
But it turns out I don't need it! The SQL that is passed to the facet class is created by this code: https://github.com/simonw/datasette/blob/ba11ef27edd6981eeb26d7ecf5aa236707f5f8ce/datasette/views/table.py#L677-L684
And the only place that uses that `sql_no_limit` variable is here: https://github.com/simonw/datasette/blob/ba11ef27edd6981eeb26d7ecf5aa236707f5f8ce/datasette/views/table.py#L733-L745
So I can change that to `sql_no_limit_no_order` and fix the bug that way instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by,
https://github.com/simonw/datasette/issues/1394#issuecomment-880278256,https://api.github.com/repos/simonw/datasette/issues/1394,880278256,MDEyOklzc3VlQ29tbWVudDg4MDI3ODI1Ng==,9599,simonw,2021-07-14T23:35:18Z,2021-07-14T23:35:18Z,OWNER,"The challenge here is that faceting doesn't currently modify the inner SQL at all - it wraps it so that it can work against any SQL statement (though Datasette itself does not yet take advantage of that ability, only offering faceting on table pages).
So just removing the order by wouldn't be appropriate if the inner query looked something like this:
```sql
select * from items order by created desc limit 100
```
Since the intent there would be to return facet counts against only the most recent 100 items.
In SQLite the `limit` has to come after the `order by` though, so the fix here could be as easy as using a regular expression to identify queries that end with `order by COLUMN (desc)?` and stripping off that clause.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880259255,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880259255,MDEyOklzc3VlQ29tbWVudDg4MDI1OTI1NQ==,9599,simonw,2021-07-14T22:48:41Z,2021-07-14T22:48:41Z,OWNER,Should also take advantage of `.mode tabs` to support `sqlite-utils insert blah.db blah blah.csv --tsv --fast`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880257587,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880257587,MDEyOklzc3VlQ29tbWVudDg4MDI1NzU4Nw==,9599,simonw,2021-07-14T22:44:05Z,2021-07-14T22:44:05Z,OWNER,"https://unix.stackexchange.com/a/642364 suggests you can also use this to import from stdin, like so:
sqlite3 -csv $database_file_name "".import '|cat -' $table_name""
Here the `sqlite3 -csv` is an alternative to using `.mode csv`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880256865,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880256865,MDEyOklzc3VlQ29tbWVudDg4MDI1Njg2NQ==,9599,simonw,2021-07-14T22:42:11Z,2021-07-14T22:42:11Z,OWNER,"Potential workaround for missing `--skip` implementation is that the filename can be a command instead, so maybe it could shell out to `tail -n +1 filename`:
> The source argument is the name of a file to be read or, if it begins with a ""|"" character, specifies a command which will be run to produce the input CSV data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880256058,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880256058,MDEyOklzc3VlQ29tbWVudDg4MDI1NjA1OA==,9599,simonw,2021-07-14T22:40:01Z,2021-07-14T22:40:47Z,OWNER,"Full docs here: https://www.sqlite.org/draft/cli.html#csv
One catch: how this works has changed in recent SQLite versions: https://www.sqlite.org/changes.html
- 2020-12-01 (3.34.0) - ""Table name quoting works correctly for the .import dot-command""
- 2020-05-22 (3.32.0) - ""Add options to the .import command: --csv, --ascii, --skip""
- 2017-08-01 (3.20.0) - "" The "".import"" command ignores an initial UTF-8 BOM.""
The ""skip"" feature is particularly important to understand. https://www.sqlite.org/draft/cli.html#csv says:
> There are two cases to consider: (1) Table ""tab1"" does not previously exist and (2) table ""tab1"" does already exist.
>
> In the first case, when the table does not previously exist, the table is automatically created and the content of the first row of the input CSV file is used to determine the name of all the columns in the table. In other words, if the table does not previously exist, the first row of the CSV file is interpreted to be column names and the actual data starts on the second row of the CSV file.
>
> For the second case, when the table already exists, every row of the CSV file, including the first row, is assumed to be actual content. If the CSV file contains an initial row of column labels, you can cause the .import command to skip that initial row using the ""--skip 1"" option.
But the `--skip 1` option is only available in 3.32.0 and higher.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/datasette/issues/268#issuecomment-880153069,https://api.github.com/repos/simonw/datasette/issues/268,880153069,MDEyOklzc3VlQ29tbWVudDg4MDE1MzA2OQ==,9599,simonw,2021-07-14T19:31:00Z,2021-07-14T19:31:00Z,OWNER,"... though interestingly I can't replicate that error on `latest.datasette.io` - https://latest.datasette.io/fixtures/searchable?_search=park.&_searchmode=raw
That's running https://latest.datasette.io/-/versions SQLite 3.35.4 whereas https://www.niche-museums.com/-/versions is running 3.27.2 (the most recent version available with Vercel) - but there's nothing in the SQLite changelog between those two versions that suggests changes to how the FTS5 parser works. https://www.sqlite.org/changes.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/268#issuecomment-880150755,https://api.github.com/repos/simonw/datasette/issues/268,880150755,MDEyOklzc3VlQ29tbWVudDg4MDE1MDc1NQ==,9599,simonw,2021-07-14T19:26:47Z,2021-07-14T19:29:08Z,OWNER,"> What are the side-effects of turning that on in the query string, or even by default as you suggested? I see that you stated in the docs... ""to ensure they do not cause any confusion for users who are not aware of them"", but I'm not sure what those could be.
Mainly that it's possible to generate SQL queries that crash with an error. This was the example that convinced me to default to escaping:
- https://www.niche-museums.com/browse/museums?_search=park.&_searchmode=raw (returns `fts5: syntax error near "".""`)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-879477586,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12,879477586,MDEyOklzc3VlQ29tbWVudDg3OTQ3NzU4Ng==,9599,simonw,2021-07-13T23:50:06Z,2021-07-13T23:50:06Z,MEMBER,"Unfortunately I don't think updating the database is practical, because the export doesn't include unique identifiers which can be used to update existing records and create new ones. Recreating from scratch works around that limitation.
I've not explored workouts with SpatiaLite but that's a really good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727848625,"Some workout columns should be float, not text",
https://github.com/simonw/datasette/pull/1393#issuecomment-879309636,https://api.github.com/repos/simonw/datasette/issues/1393,879309636,MDEyOklzc3VlQ29tbWVudDg3OTMwOTYzNg==,9599,simonw,2021-07-13T18:32:25Z,2021-07-13T18:32:25Z,OWNER,Thanks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941412189,Update deploying.rst,
https://github.com/simonw/datasette/pull/1392#issuecomment-879277953,https://api.github.com/repos/simonw/datasette/issues/1392,879277953,MDEyOklzc3VlQ29tbWVudDg3OTI3Nzk1Mw==,9599,simonw,2021-07-13T17:42:31Z,2021-07-13T17:42:31Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941403676,Update deploying.rst,
https://github.com/simonw/datasette/issues/511#issuecomment-877835171,https://api.github.com/repos/simonw/datasette/issues/511,877835171,MDEyOklzc3VlQ29tbWVudDg3NzgzNTE3MQ==,9599,simonw,2021-07-11T17:23:05Z,2021-07-11T17:23:05Z,OWNER," == 87 failed, 819 passed, 7 skipped, 29 errors in 2584.85s (0:43:04) ==
https://github.com/simonw/datasette/runs/3038188870?check_suite_focus=true
Full copy of log here: https://gist.github.com/simonw/4b1fdd24496b989fca56bc757be345ad","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-877805513,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12,877805513,MDEyOklzc3VlQ29tbWVudDg3NzgwNTUxMw==,956433,Mjboothaus,2021-07-11T14:03:01Z,2021-07-11T14:03:01Z,NONE,"Hi Simon -- just experimenting with your excellent software!
Up to this point in time I have been using the (paid) [HealthFit App](https://apps.apple.com/au/app/healthfit/id1202650514) to export my workouts from my Apple Watch, one walk at the time into either .GPX or .FIT format and then using another library to suck it into Python and eventually here to my ""Emmaus Walking"" app:
https://share.streamlit.io/mjboothaus/emmaus_walking/emmaus_walking/app.py
I just used `healthkit-to-sqlite` to convert my export.zip file and it all ""just worked"".
I did notice the issue with various numeric fields being stored in the SQLite db as TEXT for now and just thought I'd flag it - but you're already self-reported this issue.
Keep up the great work!
I was curious if you have any thoughts about periodically exporting ""export.zip"" and how to just update the SQLite file instead of re-creating it each time. Hopefully Apple will give some thought to managing this data in a more sensible fashion as it grows over time. Ideally one could pull it from iCloud (where it is allegedly being backed up).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727848625,"Some workout columns should be float, not text",
https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-877874117,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12,877874117,MDEyOklzc3VlQ29tbWVudDg3Nzg3NDExNw==,956433,Mjboothaus,2021-07-11T23:03:37Z,2021-07-11T23:03:37Z,NONE,P.s. wondering if you have explored using the spatialite functionality with the location data in workouts?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727848625,"Some workout columns should be float, not text",
https://github.com/simonw/datasette/issues/511#issuecomment-877726495,https://api.github.com/repos/simonw/datasette/issues/511,877726495,MDEyOklzc3VlQ29tbWVudDg3NzcyNjQ5NQ==,9599,simonw,2021-07-11T01:32:27Z,2021-07-11T01:32:27Z,OWNER,"I'm using `pytest-xdist` and this:
pytest -n auto -m ""not serial""
I'll try not using the `-n auto` bit on Windows and see if that helps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877726288,https://api.github.com/repos/simonw/datasette/issues/511,877726288,MDEyOklzc3VlQ29tbWVudDg3NzcyNjI4OA==,9599,simonw,2021-07-11T01:29:41Z,2021-07-11T01:29:41Z,OWNER,"Lots of errors that look like this:
```
2021-07-11T00:40:32.1189321Z E NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpdr41pgwg\\data.db'
2021-07-11T00:40:32.1190083Z
2021-07-11T00:40:32.1191128Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:596: NotADirectoryError
2021-07-11T00:40:32.1191999Z ___________________ ERROR at teardown of test_insert_error ____________________
2021-07-11T00:40:32.1192842Z [gw1] win32 -- Python 3.8.10 c:\hostedtoolcache\windows\python\3.8.10\x64\python.exe
2021-07-11T00:40:32.1193387Z
2021-07-11T00:40:32.1193930Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_'
2021-07-11T00:40:32.1194876Z onerror = .onerror at 0x00000291FCEA93A0>
2021-07-11T00:40:32.1195480Z
2021-07-11T00:40:32.1195927Z def _rmtree_unsafe(path, onerror):
2021-07-11T00:40:32.1196435Z try:
2021-07-11T00:40:32.1196910Z with os.scandir(path) as scandir_it:
2021-07-11T00:40:32.1197504Z entries = list(scandir_it)
2021-07-11T00:40:32.1198002Z except OSError:
2021-07-11T00:40:32.1198607Z onerror(os.scandir, path, sys.exc_info())
2021-07-11T00:40:32.1199137Z entries = []
2021-07-11T00:40:32.1199637Z for entry in entries:
2021-07-11T00:40:32.1200184Z fullname = entry.path
2021-07-11T00:40:32.1200692Z if _rmtree_isdir(entry):
2021-07-11T00:40:32.1201198Z try:
2021-07-11T00:40:32.1201643Z if entry.is_symlink():
2021-07-11T00:40:32.1202280Z # This can only happen if someone replaces
2021-07-11T00:40:32.1202944Z # a directory with a symlink after the call to
2021-07-11T00:40:32.1203623Z # os.scandir or entry.is_dir above.
2021-07-11T00:40:32.1204303Z raise OSError(""Cannot call rmtree on a symbolic link"")
2021-07-11T00:40:32.1204942Z except OSError:
2021-07-11T00:40:32.1206416Z onerror(os.path.islink, fullname, sys.exc_info())
2021-07-11T00:40:32.1207022Z continue
2021-07-11T00:40:32.1207584Z _rmtree_unsafe(fullname, onerror)
2021-07-11T00:40:32.1208074Z else:
2021-07-11T00:40:32.1208496Z try:
2021-07-11T00:40:32.1208926Z > os.unlink(fullname)
2021-07-11T00:40:32.1210053Z E PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1210974Z
2021-07-11T00:40:32.1211638Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:616: PermissionError
2021-07-11T00:40:32.1212211Z
2021-07-11T00:40:32.1212846Z During handling of the above exception, another exception occurred:
2021-07-11T00:40:32.1213320Z
2021-07-11T00:40:32.1213797Z func =
2021-07-11T00:40:32.1214529Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1215763Z exc_info = (, PermissionError(13, 'The process cannot access the file because it is being used by another process'), )
2021-07-11T00:40:32.1217263Z
2021-07-11T00:40:32.1217777Z def onerror(func, path, exc_info):
2021-07-11T00:40:32.1218421Z if issubclass(exc_info[0], PermissionError):
2021-07-11T00:40:32.1219079Z def resetperms(path):
2021-07-11T00:40:32.1219518Z try:
2021-07-11T00:40:32.1219992Z _os.chflags(path, 0)
2021-07-11T00:40:32.1220535Z except AttributeError:
2021-07-11T00:40:32.1221110Z pass
2021-07-11T00:40:32.1221545Z _os.chmod(path, 0o700)
2021-07-11T00:40:32.1221984Z
2021-07-11T00:40:32.1222330Z try:
2021-07-11T00:40:32.1222768Z if path != name:
2021-07-11T00:40:32.1223332Z resetperms(_os.path.dirname(path))
2021-07-11T00:40:32.1223963Z resetperms(path)
2021-07-11T00:40:32.1224408Z
2021-07-11T00:40:32.1224749Z try:
2021-07-11T00:40:32.1225954Z > _os.unlink(path)
2021-07-11T00:40:32.1227032Z E PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1227927Z
2021-07-11T00:40:32.1228646Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:802: PermissionError
2021-07-11T00:40:32.1229200Z
2021-07-11T00:40:32.1229842Z During handling of the above exception, another exception occurred:
2021-07-11T00:40:32.1230355Z
2021-07-11T00:40:32.1230783Z @pytest.fixture
2021-07-11T00:40:32.1231322Z def canned_write_client():
2021-07-11T00:40:32.1231805Z with make_app_client(
2021-07-11T00:40:32.1232467Z extra_databases={""data.db"": ""create table names (name text)""},
2021-07-11T00:40:32.1233104Z metadata={
2021-07-11T00:40:32.1233535Z ""databases"": {
2021-07-11T00:40:32.1233989Z ""data"": {
2021-07-11T00:40:32.1234416Z ""queries"": {
2021-07-11T00:40:32.1235001Z ""canned_read"": {""sql"": ""select * from names""},
2021-07-11T00:40:32.1235527Z ""add_name"": {
2021-07-11T00:40:32.1236117Z ""sql"": ""insert into names (name) values (:name)"",
2021-07-11T00:40:32.1236686Z ""write"": True,
2021-07-11T00:40:32.1237317Z ""on_success_redirect"": ""/data/add_name?success"",
2021-07-11T00:40:32.1237882Z },
2021-07-11T00:40:32.1238331Z ""add_name_specify_id"": {
2021-07-11T00:40:32.1239009Z ""sql"": ""insert into names (rowid, name) values (:rowid, :name)"",
2021-07-11T00:40:32.1239610Z ""write"": True,
2021-07-11T00:40:32.1240259Z ""on_error_redirect"": ""/data/add_name_specify_id?error"",
2021-07-11T00:40:32.1240839Z },
2021-07-11T00:40:32.1241320Z ""delete_name"": {
2021-07-11T00:40:32.1242504Z ""sql"": ""delete from names where rowid = :rowid"",
2021-07-11T00:40:32.1243127Z ""write"": True,
2021-07-11T00:40:32.1243721Z ""on_success_message"": ""Name deleted"",
2021-07-11T00:40:32.1244282Z ""allow"": {""id"": ""root""},
2021-07-11T00:40:32.1244749Z },
2021-07-11T00:40:32.1245959Z ""update_name"": {
2021-07-11T00:40:32.1246614Z ""sql"": ""update names set name = :name where rowid = :rowid"",
2021-07-11T00:40:32.1247267Z ""params"": [""rowid"", ""name"", ""extra""],
2021-07-11T00:40:32.1247828Z ""write"": True,
2021-07-11T00:40:32.1248247Z },
2021-07-11T00:40:32.1248653Z }
2021-07-11T00:40:32.1249166Z }
2021-07-11T00:40:32.1249577Z }
2021-07-11T00:40:32.1249962Z },
2021-07-11T00:40:32.1250333Z ) as client:
2021-07-11T00:40:32.1250822Z > yield client
2021-07-11T00:40:32.1251078Z
2021-07-11T00:40:32.1251678Z D:\a\datasette\datasette\tests\test_canned_queries.py:43:
2021-07-11T00:40:32.1252347Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-07-11T00:40:32.1253040Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\contextlib.py:120: in __exit__
2021-07-11T00:40:32.1253759Z next(self.gen)
2021-07-11T00:40:32.1254398Z D:\a\datasette\datasette\tests\fixtures.py:156: in make_app_client
2021-07-11T00:40:32.1255098Z yield TestClient(ds)
2021-07-11T00:40:32.1255796Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:827: in __exit__
2021-07-11T00:40:32.1256510Z self.cleanup()
2021-07-11T00:40:32.1257200Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:831: in cleanup
2021-07-11T00:40:32.1257961Z self._rmtree(self.name)
2021-07-11T00:40:32.1258712Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:813: in _rmtree
2021-07-11T00:40:32.1259487Z _shutil.rmtree(name, onerror=onerror)
2021-07-11T00:40:32.1260280Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:740: in rmtree
2021-07-11T00:40:32.1261039Z return _rmtree_unsafe(path, onerror)
2021-07-11T00:40:32.1261843Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:618: in _rmtree_unsafe
2021-07-11T00:40:32.1262633Z onerror(os.unlink, fullname, sys.exc_info())
2021-07-11T00:40:32.1263456Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:805: in onerror
2021-07-11T00:40:32.1264175Z cls._rmtree(path)
2021-07-11T00:40:32.1264848Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:813: in _rmtree
2021-07-11T00:40:32.1266329Z _shutil.rmtree(name, onerror=onerror)
2021-07-11T00:40:32.1267082Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:740: in rmtree
2021-07-11T00:40:32.1267858Z return _rmtree_unsafe(path, onerror)
2021-07-11T00:40:32.1268615Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:599: in _rmtree_unsafe
2021-07-11T00:40:32.1269440Z onerror(os.scandir, path, sys.exc_info())
2021-07-11T00:40:32.1269979Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-07-11T00:40:32.1270287Z
2021-07-11T00:40:32.1270947Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1273356Z onerror = .onerror at 0x00000291FCF40E50>
2021-07-11T00:40:32.1273999Z
2021-07-11T00:40:32.1274493Z def _rmtree_unsafe(path, onerror):
2021-07-11T00:40:32.1274953Z try:
2021-07-11T00:40:32.1275461Z > with os.scandir(path) as scandir_it:
2021-07-11T00:40:32.1276459Z E NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1277220Z
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877725742,https://api.github.com/repos/simonw/datasette/issues/511,877725742,MDEyOklzc3VlQ29tbWVudDg3NzcyNTc0Mg==,9599,simonw,2021-07-11T01:25:01Z,2021-07-11T01:26:38Z,OWNER,"That's weird. https://github.com/simonw/datasette/runs/3037862798 finished running and came up green - but actually a TON of the tests failed on Windows. Not sure why that didn't fail the whole test suite:
![]()
Also the test suite took 50 minutes on Windows!
Here's a copy of the full log file for the tests on Python 3.8 on Windows: https://gist.github.com/simonw/2900ef33693c1bbda09188eb31c8212d","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/1388#issuecomment-877725193,https://api.github.com/repos/simonw/datasette/issues/1388,877725193,MDEyOklzc3VlQ29tbWVudDg3NzcyNTE5Mw==,9599,simonw,2021-07-11T01:18:38Z,2021-07-11T01:18:38Z,OWNER,Wrote up a TIL: https://til.simonwillison.net/nginx/proxy-domain-sockets,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877721003,https://api.github.com/repos/simonw/datasette/issues/1388,877721003,MDEyOklzc3VlQ29tbWVudDg3NzcyMTAwMw==,9599,simonw,2021-07-11T00:21:19Z,2021-07-11T00:21:19Z,OWNER,Documentation: https://docs.datasette.io/en/latest/deploying.html#nginx-proxy-configuration,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/511#issuecomment-877718364,https://api.github.com/repos/simonw/datasette/issues/511,877718364,MDEyOklzc3VlQ29tbWVudDg3NzcxODM2NA==,9599,simonw,2021-07-10T23:54:37Z,2021-07-10T23:54:37Z,OWNER,"Looks like it's not even 10% of the way through, and already a bunch of errors:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877718286,https://api.github.com/repos/simonw/datasette/issues/511,877718286,MDEyOklzc3VlQ29tbWVudDg3NzcxODI4Ng==,9599,simonw,2021-07-10T23:53:29Z,2021-07-10T23:53:29Z,OWNER,"Test suite on Windows seems to run a lot slower:
![]()
From https://github.com/simonw/datasette/actions/runs/1018938850 which is still going.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877717791,https://api.github.com/repos/simonw/datasette/issues/511,877717791,MDEyOklzc3VlQ29tbWVudDg3NzcxNzc5MQ==,9599,simonw,2021-07-10T23:45:35Z,2021-07-10T23:45:35Z,OWNER,"> Trying to run on Windows today, I get an error from the utils/asgi.py module.
>
> It's trying `from os import EX_CANTCREAT` which is Unix-only. I commented this line out, and (so far) it's working.
Good news: that line was removed in #1094.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/pull/557#issuecomment-877717392,https://api.github.com/repos/simonw/datasette/issues/557,877717392,MDEyOklzc3VlQ29tbWVudDg3NzcxNzM5Mg==,9599,simonw,2021-07-10T23:39:48Z,2021-07-10T23:39:48Z,OWNER,Abandoning this - need to switch to using GitHub Actions for this instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",466996584,Get tests running on Windows using Travis CI,
https://github.com/simonw/datasette/issues/1388#issuecomment-877717262,https://api.github.com/repos/simonw/datasette/issues/1388,877717262,MDEyOklzc3VlQ29tbWVudDg3NzcxNzI2Mg==,9599,simonw,2021-07-10T23:37:54Z,2021-07-10T23:37:54Z,OWNER,"> I wonder if `--fd` is worth supporting too?
I'm going to hold off on implementing this until someone asks for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877716993,https://api.github.com/repos/simonw/datasette/issues/1388,877716993,MDEyOklzc3VlQ29tbWVudDg3NzcxNjk5Mw==,9599,simonw,2021-07-10T23:34:02Z,2021-07-10T23:34:02Z,OWNER,"Figured out an example nginx configuration. This in `nginx.conf`:
daemon off;
events {
worker_connections 1024;
}
http {
server {
listen 8092;
location / {
proxy_pass http://datasette;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
upstream datasette {
server unix:/tmp/datasette.sock;
}
}
Then run `datasette --uds /tmp/datasette.sock`
Then run nginx like this:
nginx -c ./nginx.conf
Then hits to `http://localhost:8092/` will be proxied to Datasette.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877716359,https://api.github.com/repos/simonw/datasette/issues/1388,877716359,MDEyOklzc3VlQ29tbWVudDg3NzcxNjM1OQ==,9599,simonw,2021-07-10T23:24:58Z,2021-07-10T23:24:58Z,OWNER,"Apparently Windows 10 has Unix domain socket support: https://bugs.python.org/issue33408
> Unix socket (AF_UNIX) is now avalible in Windows 10 (April 2018 Update). Please add Python support for it.
> More details about it on https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/
But it's not clear if this is going to work. That same issue thread (the issue is still open) suggests using `hasattr(socket, 'AF_UNIX'))` to detect support in tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877716156,https://api.github.com/repos/simonw/datasette/issues/1388,877716156,MDEyOklzc3VlQ29tbWVudDg3NzcxNjE1Ng==,9599,simonw,2021-07-10T23:22:21Z,2021-07-10T23:22:21Z,OWNER,"I don't have the Datasette test suite running on Windows yet, but I'd like it to run there some day - so ideally this test would be skipped if Unix domain sockets are not supported by the underlying operating system.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877715654,https://api.github.com/repos/simonw/datasette/issues/1388,877715654,MDEyOklzc3VlQ29tbWVudDg3NzcxNTY1NA==,9599,simonw,2021-07-10T23:15:06Z,2021-07-10T23:15:06Z,OWNER,"I can run tests against it using `httpx`: https://www.python-httpx.org/advanced/#usage_1
> ```pycon
> >>> import httpx
> >>> # Connect to the Docker API via a Unix Socket.
> >>> transport = httpx.HTTPTransport(uds=""/var/run/docker.sock"")
> >>> client = httpx.Client(transport=transport)
> >>> response = client.get(""http://docker/info"")
> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877714698,https://api.github.com/repos/simonw/datasette/issues/1388,877714698,MDEyOklzc3VlQ29tbWVudDg3NzcxNDY5OA==,9599,simonw,2021-07-10T23:01:37Z,2021-07-10T23:01:37Z,OWNER,"Can test this with:
```
curl --unix-socket ${socket} -i ""http://localhost/""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1391#issuecomment-877691558,https://api.github.com/repos/simonw/datasette/issues/1391,877691558,MDEyOklzc3VlQ29tbWVudDg3NzY5MTU1OA==,9599,simonw,2021-07-10T19:26:57Z,2021-07-10T19:26:57Z,OWNER,"The `https://latest.datasette.io/fixtures.db` file no longer includes generated columns, which will help avoid confusion such as seen in #1376.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1391#issuecomment-877691427,https://api.github.com/repos/simonw/datasette/issues/1391,877691427,MDEyOklzc3VlQ29tbWVudDg3NzY5MTQyNw==,9599,simonw,2021-07-10T19:26:00Z,2021-07-10T19:26:00Z,OWNER,I had to run the tests locally on my macOS laptop using `pysqlite3` to get a version that supported generated columns - wrote up a TIL about that here: https://til.simonwillison.net/sqlite/pysqlite3-on-macos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1391#issuecomment-877687196,https://api.github.com/repos/simonw/datasette/issues/1391,877687196,MDEyOklzc3VlQ29tbWVudDg3NzY4NzE5Ng==,9599,simonw,2021-07-10T18:58:40Z,2021-07-10T18:58:40Z,OWNER,I can use the `extra_databases` mechanism as demonstrated here: https://github.com/simonw/datasette/blob/9552414e1f968c6fc704031cec349c05e6bc2371/tests/test_canned_queries.py#L8-L12,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1391#issuecomment-877686784,https://api.github.com/repos/simonw/datasette/issues/1391,877686784,MDEyOklzc3VlQ29tbWVudDg3NzY4Njc4NA==,9599,simonw,2021-07-10T18:56:03Z,2021-07-10T18:56:03Z,OWNER,Here's the SQL used to generate the table for the test: https://github.com/simonw/datasette/blob/02b19c7a9afd328f22040ab33b5c1911cd904c7c/tests/fixtures.py#L723-L733,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1391#issuecomment-877682533,https://api.github.com/repos/simonw/datasette/issues/1391,877682533,MDEyOklzc3VlQ29tbWVudDg3NzY4MjUzMw==,9599,simonw,2021-07-10T18:28:05Z,2021-07-10T18:28:05Z,OWNER,"Here's the test in question: https://github.com/simonw/datasette/blob/a6c55afe8c82ead8deb32f90c9324022fd422324/tests/test_api.py#L2033-L2046
Various other places in the test code also need changing - anything that calls `supports_generated_columns()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1389#issuecomment-877681031,https://api.github.com/repos/simonw/datasette/issues/1389,877681031,MDEyOklzc3VlQ29tbWVudDg3NzY4MTAzMQ==,9599,simonw,2021-07-10T18:17:29Z,2021-07-10T18:17:29Z,OWNER,"I don't like `?_searchmode=default` because it suggests ""use the default"" - but it actually over-rides the default that was specified by `""searchmode"": ""raw""` in `metadata.json`.
I'm going with `?_searchmode=escaped` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1390#issuecomment-877310125,https://api.github.com/repos/simonw/datasette/issues/1390,877310125,MDEyOklzc3VlQ29tbWVudDg3NzMxMDEyNQ==,9599,simonw,2021-07-09T16:32:57Z,2021-07-09T16:32:57Z,OWNER,https://docs.datasette.io/en/latest/deploying.html#running-datasette-using-systemd,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940891698,Mention restarting systemd in documentation,
https://github.com/simonw/datasette/issues/1390#issuecomment-877308310,https://api.github.com/repos/simonw/datasette/issues/1390,877308310,MDEyOklzc3VlQ29tbWVudDg3NzMwODMxMA==,9599,simonw,2021-07-09T16:29:48Z,2021-07-09T16:29:48Z,OWNER, sudo systemctl restart datasette.service,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940891698,Mention restarting systemd in documentation,
https://github.com/simonw/datasette/issues/268#issuecomment-876721585,https://api.github.com/repos/simonw/datasette/issues/268,876721585,MDEyOklzc3VlQ29tbWVudDg3NjcyMTU4NQ==,9308268,rayvoelker,2021-07-08T20:22:17Z,2021-07-08T20:22:17Z,NONE,"I do like the idea of there being a option for turning that on by default so that you could use those terms in the default ""Search"" bar presented when you browse to a table where FTS has been enabled. Maybe even a small inline pop up with a short bit explaining the FTS feature and the keywords (e.g. case matters). What are the side-effects of turning that on in the query string, or even by default as you suggested? I see that you stated in the docs... ""to ensure they do not cause any confusion for users who are not aware of them"", but I'm not sure what those could be.
Isn't it the case that those keywords are only picked up by sqlite in where you're using the MATCH clause?
Seems like a really powerful feature (even though there are a lot of hurdles around setting it up in the sqlite db ... sqlite-utils makes that so simple by the way!)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/1389#issuecomment-876620095,https://api.github.com/repos/simonw/datasette/issues/1389,876620095,MDEyOklzc3VlQ29tbWVudDg3NjYyMDA5NQ==,9599,simonw,2021-07-08T17:35:09Z,2021-07-08T17:35:09Z,OWNER,Also came up here: https://github.com/simonw/datasette/issues/268#issuecomment-876616414,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1389#issuecomment-876619531,https://api.github.com/repos/simonw/datasette/issues/1389,876619531,MDEyOklzc3VlQ29tbWVudDg3NjYxOTUzMQ==,9599,simonw,2021-07-08T17:34:16Z,2021-07-08T17:34:16Z,OWNER,If I implement this I'll also set it up so `?_searchmode=default` can be used to over-ride that and go back to the default behaviour.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1389#issuecomment-876619271,https://api.github.com/repos/simonw/datasette/issues/1389,876619271,MDEyOklzc3VlQ29tbWVudDg3NjYxOTI3MQ==,9599,simonw,2021-07-08T17:33:49Z,2021-07-08T17:33:49Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/table.py#L491-L498,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1389#issuecomment-876618847,https://api.github.com/repos/simonw/datasette/issues/1389,876618847,MDEyOklzc3VlQ29tbWVudDg3NjYxODg0Nw==,9599,simonw,2021-07-08T17:33:08Z,2021-07-08T17:33:08Z,OWNER,Relevant documentation: https://docs.datasette.io/en/0.57.1/full_text_search.html#advanced-sqlite-search-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1389#issuecomment-876618582,https://api.github.com/repos/simonw/datasette/issues/1389,876618582,MDEyOklzc3VlQ29tbWVudDg3NjYxODU4Mg==,9599,simonw,2021-07-08T17:32:40Z,2021-07-08T17:32:40Z,OWNER,This makes sense to me since other useful querystring arguments like this can be set as defaults in the `metadata.json` configuration.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/268#issuecomment-876616414,https://api.github.com/repos/simonw/datasette/issues/268,876616414,MDEyOklzc3VlQ29tbWVudDg3NjYxNjQxNA==,9599,simonw,2021-07-08T17:29:04Z,2021-07-08T17:29:04Z,OWNER,"> I had setup a full text search on my instance of Datasette for title data for our public library, and was noticing that some of the features of the SQLite FTS weren't working as expected ... and maybe the issue is in the `escape_fts()` function
That's a deliberate feature (albeit controversial, see #759) - part of the main problem here is that it's easy to construct a SQLite full-text search string which results in a database error. This is a bad user-experience!
You can opt-in to raw SQL queries by appending `?_searchmode=raw` to the page, see https://docs.datasette.io/en/stable/full_text_search.html#advanced-sqlite-search-queries
But maybe there should be an option for turning that on by default without needing the query string?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/268#issuecomment-876428348,https://api.github.com/repos/simonw/datasette/issues/268,876428348,MDEyOklzc3VlQ29tbWVudDg3NjQyODM0OA==,9308268,rayvoelker,2021-07-08T13:13:12Z,2021-07-08T13:13:12Z,NONE,"I had setup a full text search on my instance of Datasette for title data for our public library, and was noticing that some of the features of the SQLite FTS weren't working as expected ... and maybe the issue is in the `escape_fts()` function
vs removing the function...
Also, on the issue of sorting by rank by default .. perhaps something like this could work for the baked-in default SQL query for Datasette?
[link to the above search in my instance of Datasette](https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-87a9011?sql=with+fts_search+as+%28%0D%0A++select%0D%0A++rowid%2C%0D%0A++rank%0D%0A++++from%0D%0A++++++bib_fts%0D%0A++++where%0D%0A++++++bib_fts+match+%3Asearch%0D%0A%29%0D%0A%0D%0Aselect%0D%0A++%0D%0A++bib_record_num%2C%0D%0A++creation_date%2C%0D%0A++record_last_updated%2C%0D%0A++isbn%2C%0D%0A++best_author%2C%0D%0A++best_title%2C%0D%0A++publisher%2C%0D%0A++publish_year%2C%0D%0A++bib_level_callnumber%2C%0D%0A++indexed_subjects%0D%0Afrom%0D%0A++fts_search%0D%0A++join+bib+on+bib.rowid+%3D+fts_search.rowid%0D%0A++%0D%0Aorder+by%0D%0Arank%0D%0A&search=black+death+NOT+fiction)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/1388#issuecomment-876213177,https://api.github.com/repos/simonw/datasette/issues/1388,876213177,MDEyOklzc3VlQ29tbWVudDg3NjIxMzE3Nw==,80737,aslakr,2021-07-08T07:47:17Z,2021-07-08T07:47:17Z,CONTRIBUTOR,"> This sounds like a valuable feature for people running Datasette behind a proxy.
Yes, in some cases it is easer to use e.g. Apache's [ProxyPass Directive](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass) with Unix Domain Socket like `unix:/home/www.socket|http://localhost/whatever/`.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-875738149,https://api.github.com/repos/simonw/datasette/issues/1388,875738149,MDEyOklzc3VlQ29tbWVudDg3NTczODE0OQ==,9599,simonw,2021-07-07T16:14:29Z,2021-07-07T16:14:29Z,OWNER,This sounds like a valuable feature for people running Datasette behind a proxy.,"{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-875742910,https://api.github.com/repos/simonw/datasette/issues/1388,875742910,MDEyOklzc3VlQ29tbWVudDg3NTc0MjkxMA==,9599,simonw,2021-07-07T16:20:50Z,2021-07-07T16:23:02Z,OWNER,"I wonder if `--fd` is worth supporting too? Uvicorn documentation says that's useful for running under process managers, I'd want to understand exactly how to use that (and test it) before adding the feature though.
https://www.uvicorn.org/settings/
Docs on how to use a process manager: https://www.uvicorn.org/deployment/#supervisor","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-875741410,https://api.github.com/repos/simonw/datasette/issues/1388,875741410,MDEyOklzc3VlQ29tbWVudDg3NTc0MTQxMA==,9599,simonw,2021-07-07T16:18:50Z,2021-07-07T16:18:50Z,OWNER,"You could actually run Datasette like this today without modifications by running a thin Python script that imports from `datasette.app`, instantiates the ASGI app and passes that to `uvicorn.run` - but I like this as a supported feature too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-875740085,https://api.github.com/repos/simonw/datasette/issues/1388,875740085,MDEyOklzc3VlQ29tbWVudDg3NTc0MDA4NQ==,9599,simonw,2021-07-07T16:17:08Z,2021-07-07T16:17:08Z,OWNER,"Looks pretty easy to implement - here's a hint from Uvicorn source code: https://github.com/encode/uvicorn/blob/b5af1049e63c059dc750a450c807b9768f91e906/uvicorn/main.py#L368
Need to work out a simple pattern for testing this too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1387#issuecomment-873139138,https://api.github.com/repos/simonw/datasette/issues/1387,873139138,MDEyOklzc3VlQ29tbWVudDg3MzEzOTEzOA==,9599,simonw,2021-07-02T17:05:47Z,2021-07-02T17:05:47Z,OWNER,"In this case the proxy is Apache. So there are a couple of potential fixes:
- Configure Apache to pass the original HTTP request `Host:` header through to the proxied application. This should then be documented.
- Add a new optional feature to Datasette called something like `base_host` which, if set, is always used in place of the host in `request.url` when constructing new URLs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873137935,https://api.github.com/repos/simonw/datasette/issues/1387,873137935,MDEyOklzc3VlQ29tbWVudDg3MzEzNzkzNQ==,9599,simonw,2021-07-02T17:03:36Z,2021-07-02T17:03:36Z,OWNER,"And the links to apply a facet value are broken too! https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-3d4a4b7/bib?_facet=bib_level_callnumber
```json
{
""value"": ""g l fiction"",
""label"": ""g l fiction"",
""count"": 212,
""toggle_url"": ""https://127.0.0.1:8010/collection-analysis/current_collection-3d4a4b7/bib.json?_facet=bib_level_callnumber&bib_level_callnumber=g+l+fiction"",
""selected"": false
}
```
Same problem: https://github.com/simonw/datasette/blob/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f/datasette/facets.py#L251-L261","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873136440,https://api.github.com/repos/simonw/datasette/issues/1387,873136440,MDEyOklzc3VlQ29tbWVudDg3MzEzNjQ0MA==,9599,simonw,2021-07-02T17:01:48Z,2021-07-02T17:01:48Z,OWNER,"Here's what's happening: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/table.py#L827-L829
This is being run through `absolute_url()` - defined here: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/app.py#L633-L637
That's because the `next_url` in the JSON needs to be a full URL that a client can retrieve - as opposed to the other links on that page which are all relative links that start with `/`: https://github.com/simonw/datasette/blob/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f/datasette/templates/_table.html#L11-L15
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873134866,https://api.github.com/repos/simonw/datasette/issues/1387,873134866,MDEyOklzc3VlQ29tbWVudDg3MzEzNDg2Ng==,9599,simonw,2021-07-02T16:58:52Z,2021-07-02T16:58:52Z,OWNER,What's weird here is that the URL itself is correct - it starts with `/collection-analysis/` as expected - but the hostname is wrong.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873166836,https://api.github.com/repos/simonw/datasette/issues/1387,873166836,MDEyOklzc3VlQ29tbWVudDg3MzE2NjgzNg==,9308268,rayvoelker,2021-07-02T17:58:23Z,2021-07-02T17:58:23Z,NONE,"Thanks Simon for nailing that one down! It does seem a little confusing that the ProxyPreservehost option is set to Off By default, but this config totally did the trick and fixed the issue
```
ProxyPass http://127.0.0.1:8010/collection-analysis/
ProxyPreservehost On
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873156408,https://api.github.com/repos/simonw/datasette/issues/1387,873156408,MDEyOklzc3VlQ29tbWVudDg3MzE1NjQwOA==,9599,simonw,2021-07-02T17:37:30Z,2021-07-02T17:37:30Z,OWNER,Updated documentation is here: https://docs.datasette.io/en/latest/deploying.html#apache-proxy-configuration,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873141222,https://api.github.com/repos/simonw/datasette/issues/1387,873141222,MDEyOklzc3VlQ29tbWVudDg3MzE0MTIyMg==,9599,simonw,2021-07-02T17:09:32Z,2021-07-02T17:09:32Z,OWNER,I'm going to add this to the suggested Apache configuration at https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873140742,https://api.github.com/repos/simonw/datasette/issues/1387,873140742,MDEyOklzc3VlQ29tbWVudDg3MzE0MDc0Mg==,9599,simonw,2021-07-02T17:08:40Z,2021-07-02T17:08:40Z,OWNER,`ProxyPreserveHost On` is the Apache setting - it defaults to `Off`: https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypreservehost ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1101#issuecomment-869812567,https://api.github.com/repos/simonw/datasette/issues/1101,869812567,MDEyOklzc3VlQ29tbWVudDg2OTgxMjU2Nw==,9599,simonw,2021-06-28T16:06:57Z,2021-06-28T16:07:24Z,OWNER,"Relevant blog post: https://simonwillison.net/2021/Jun/25/streaming-large-api-responses/ - including notes on efficiently streaming formats with some kind of separator in between the records (regular JSON).
> Some export formats are friendlier for streaming than others. CSV and TSV are pretty easy to stream, as is newline-delimited JSON.
>
> Regular JSON requires a bit more thought: you can output a `[` character, then output each row in a stream with a comma suffix, then skip the comma for the last row and output a `]`. Doing that requires peeking ahead (looping two at a time) to verify that you haven't yet reached the end.
>
> Or... Martin De Wulf [pointed out](https://twitter.com/madewulf/status/1405559088994467844) that you can output the first row, then output every other row with a preceeding comma---which avoids the whole ""iterate two at a time"" problem entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/pull/1386#issuecomment-869677851,https://api.github.com/repos/simonw/datasette/issues/1386,869677851,MDEyOklzc3VlQ29tbWVudDg2OTY3Nzg1MQ==,22429695,codecov[bot],2021-06-28T13:18:50Z,2021-06-28T13:18:50Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1386](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e974ed1) into [main](https://codecov.io/gh/simonw/datasette/commit/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ea627ba) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #1386 +/- ##
=======================================
Coverage 91.70% 91.70%
=======================================
Files 34 34
Lines 4364 4364
=======================================
Hits 4002 4002
Misses 362 362
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [ea627ba...e974ed1](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",931557895,"Update asgiref requirement from <3.4.0,>=3.2.10 to >=3.2.10,<3.5.0",
https://github.com/simonw/datasette/pull/1385#issuecomment-869105782,https://api.github.com/repos/simonw/datasette/issues/1385,869105782,MDEyOklzc3VlQ29tbWVudDg2OTEwNTc4Mg==,22429695,codecov[bot],2021-06-27T05:48:55Z,2021-09-13T17:29:30Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1385](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (db78094) into [main](https://codecov.io/gh/simonw/datasette/commit/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ea627ba) will **not change** coverage.
> The diff coverage is `n/a`.
> :exclamation: Current head db78094 differs from pull request most recent head 8d78c8c. Consider uploading reports for the commit 8d78c8c to get more accurate results
[](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #1385 +/- ##
=======================================
Coverage 91.70% 91.70%
=======================================
Files 34 34
Lines 4364 4364
=======================================
Hits 4002 4002
Misses 362 362
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [ea627ba...8d78c8c](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930855052,Fix + improve get_metadata plugin hook docs,
https://github.com/simonw/datasette/issues/1101#issuecomment-869191854,https://api.github.com/repos/simonw/datasette/issues/1101,869191854,MDEyOklzc3VlQ29tbWVudDg2OTE5MTg1NA==,25778,eyeseast,2021-06-27T16:42:14Z,2021-06-27T16:42:14Z,CONTRIBUTOR,This would really help with this issue: https://github.com/eyeseast/datasette-geojson/issues/7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1168#issuecomment-869076254,https://api.github.com/repos/simonw/datasette/issues/1168,869076254,MDEyOklzc3VlQ29tbWVudDg2OTA3NjI1NA==,2670795,brandonrobertz,2021-06-27T00:03:16Z,2021-06-27T00:05:51Z,CONTRIBUTOR,"> Related: Here's an implementation of a `get_metadata()` plugin hook by @brandonrobertz [next-LI@3fd8ce9](https://github.com/next-LI/datasette/commit/3fd8ce91f3108c82227bf65ff033929426c60437)
Here's a plugin that implements metadata-within-DBs: [next-LI/datasette-live-config](https://github.com/next-LI/datasette-live-config)
How it works: If a database has a `__metadata` table, then it gets parsed and included in the global metadata. It also implements a database-action hook with a UI for managing config.
More context: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L109-L140","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1384#issuecomment-869075395,https://api.github.com/repos/simonw/datasette/issues/1384,869075395,MDEyOklzc3VlQ29tbWVudDg2OTA3NTM5NQ==,9599,simonw,2021-06-26T23:54:21Z,2021-06-26T23:59:21Z,OWNER,(It may well be that implementing #1168 involves a switch to async metadata),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869075368,https://api.github.com/repos/simonw/datasette/issues/1384,869075368,MDEyOklzc3VlQ29tbWVudDg2OTA3NTM2OA==,9599,simonw,2021-06-26T23:53:55Z,2021-06-26T23:53:55Z,OWNER,"Great, let's drop fallback then.
My instinct at the moment is to ship this plugin hook as-is but with a warning that it may change before Datasette 1.0 - then before 1.0 either figure out an async variant or finish the database-backed metadata concept from #1168 and recommend that as an alternative.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869074701,https://api.github.com/repos/simonw/datasette/issues/1384,869074701,MDEyOklzc3VlQ29tbWVudDg2OTA3NDcwMQ==,2670795,brandonrobertz,2021-06-26T23:45:18Z,2021-06-26T23:45:37Z,CONTRIBUTOR,"> Here's where the plugin hook is called, demonstrating the `fallback=` argument:
>
> https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472
>
> I'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?
I think you're right. I can't think of a reason why the plugin would care about the `fallback` parameter since plugins are currently mandated to return a full, global metadata dict.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869074182,https://api.github.com/repos/simonw/datasette/issues/1384,869074182,MDEyOklzc3VlQ29tbWVudDg2OTA3NDE4Mg==,2670795,brandonrobertz,2021-06-26T23:37:42Z,2021-06-26T23:37:42Z,CONTRIBUTOR,"> > Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.
> > @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?
>
> Answering my own question: here's how Brandon implements it in his `datasette-live-config` plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160
>
> That's using a completely separate SQLite connection (actually wrapped in `sqlite-utils`) and making blocking synchronous calls to it.
>
> This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.
>
> But... it's weird. Everywhere else in Datasette land uses `await db.execute(...)` - but here's an example where users are encouraged to use blocking calls instead.
_Ideally_ this hook would be asynchronous, but when I started down that path I quickly realized how large of a change this would be, since metadata gets used synchronously across the entire Datasette codebase. (And calling async code from sync is non-trivial.)
In my live-configuration implementation I use synchronous reads using a persistent sqlite connection. This works pretty well in practice, but I agree it's limiting. My thinking around this was to go with the path of least change as `Datasette.metadata()` is a critical core function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869071790,https://api.github.com/repos/simonw/datasette/issues/1384,869071790,MDEyOklzc3VlQ29tbWVudDg2OTA3MTc5MA==,9599,simonw,2021-06-26T23:04:12Z,2021-06-26T23:04:12Z,OWNER,"> Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.
>
> @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?
Answering my own question: here's how Brandon implements it in his `datasette-live-config` plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160
That's using a completely separate SQLite connection (actually wrapped in `sqlite-utils`) and making blocking synchronous calls to it.
This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.
But... it's weird. Everywhere else in Datasette land uses `await db.execute(...)` - but here's an example where users are encouraged to use blocking calls instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869071435,https://api.github.com/repos/simonw/datasette/issues/1384,869071435,MDEyOklzc3VlQ29tbWVudDg2OTA3MTQzNQ==,9599,simonw,2021-06-26T22:59:26Z,2021-06-26T22:59:26Z,OWNER,"The other alternative is to finish the work to build a `_metadata` internal table, see #1168. The idea there was that if we want to support efficient pagination and search across the metadata for thousands of attached tables powering it with a plugin hook doesn't work well - we don't want to call the hook once for every one of 1,000+ tables just to implement the homepage.
So instead, all metadata for all attached databases would be loaded into an in-memory database called `_metadata`. Plugins that want to modify stored metadata could then do so by directly writing to that table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-869071236,https://api.github.com/repos/simonw/datasette/issues/860,869071236,MDEyOklzc3VlQ29tbWVudDg2OTA3MTIzNg==,9599,simonw,2021-06-26T22:56:28Z,2021-06-26T22:56:28Z,OWNER,This work is continuing in #1384.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869071167,https://api.github.com/repos/simonw/datasette/issues/1384,869071167,MDEyOklzc3VlQ29tbWVudDg2OTA3MTE2Nw==,9599,simonw,2021-06-26T22:55:36Z,2021-06-26T22:55:36Z,OWNER,"Just realized I already have an issue open for this, at #860. I'm going to close that and continue work on this in this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869070941,https://api.github.com/repos/simonw/datasette/issues/1384,869070941,MDEyOklzc3VlQ29tbWVudDg2OTA3MDk0MQ==,9599,simonw,2021-06-26T22:53:34Z,2021-06-26T22:53:34Z,OWNER,"The `await` thing is worrying me a lot - it feels like this plugin hook is massively less useful if it can't make it's own DB queries and generally do asynchronous stuff - but I'd also like not to break every existing plugin that calls `datasette.metadata(...)`.
One solution that could work: introduce a new method, maybe `await datasette.get_metadata(...)`, which uses this plugin hook - and keep the existing `datasette.metadata()` method (which doesn't call the hook) around. This would ensure existing plugins keep on working.
Then, upgrade those plugins separately - with the goal of deprecating and removing `.metadata()` entirely in Datasette 1.0 - having upgraded the plugins in the meantime.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869070348,https://api.github.com/repos/simonw/datasette/issues/1384,869070348,MDEyOklzc3VlQ29tbWVudDg2OTA3MDM0OA==,9599,simonw,2021-06-26T22:46:18Z,2021-06-26T22:46:18Z,OWNER,"Here's where the plugin hook is called, demonstrating the `fallback=` argument: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472
I'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869070076,https://api.github.com/repos/simonw/datasette/issues/1384,869070076,MDEyOklzc3VlQ29tbWVudDg2OTA3MDA3Ng==,9599,simonw,2021-06-26T22:42:21Z,2021-06-26T22:42:21Z,OWNER,"Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.
@brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869069926,https://api.github.com/repos/simonw/datasette/issues/1384,869069926,MDEyOklzc3VlQ29tbWVudDg2OTA2OTkyNg==,9599,simonw,2021-06-26T22:40:15Z,2021-06-26T22:40:53Z,OWNER,"The documentation says:
> **datasette**: You can use this to access plugin configuration options via `datasette.plugin_config(your_plugin_name)`, or to execute SQL queries.
That's not accurate: since the plugin hook is a regular function, not an awaitable, you can't use it to run `await db.execute(...)` so you can't execute SQL queries.
I can fix this with the await-me-maybe pattern, used for other plugin hooks: https://simonwillison.net/2020/Sep/2/await-me-maybe/
BUT... that requires changing the `ds.metadata()` function to be awaitable, which will affect every existing plugn that uses that documented internal method!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869069768,https://api.github.com/repos/simonw/datasette/issues/1384,869069768,MDEyOklzc3VlQ29tbWVudDg2OTA2OTc2OA==,9599,simonw,2021-06-26T22:37:53Z,2021-06-26T22:37:53Z,OWNER,The documentation doesn't describe the ``fallback`` argument at the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869069655,https://api.github.com/repos/simonw/datasette/issues/1384,869069655,MDEyOklzc3VlQ29tbWVudDg2OTA2OTY1NQ==,9599,simonw,2021-06-26T22:36:14Z,2021-06-26T22:37:37Z,OWNER,"Documentation for the new hook is now live at https://docs.datasette.io/en/latest/plugin_hooks.html#get-metadata-datasette-key-database-table-fallback
Link to the current snapshot of that documentation: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/docs/plugin_hooks.rst#get-metadata-datasette-key-database-table-fallback","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/pull/1368#issuecomment-869068554,https://api.github.com/repos/simonw/datasette/issues/1368,869068554,MDEyOklzc3VlQ29tbWVudDg2OTA2ODU1NA==,9599,simonw,2021-06-26T22:23:57Z,2021-06-26T22:23:57Z,OWNER,The only test failure is Black. I'm going to merge this and then reformat.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881190,https://api.github.com/repos/simonw/sqlite-utils/issues/37,868881190,MDEyOklzc3VlQ29tbWVudDg2ODg4MTE5MA==,9599,simonw,2021-06-25T23:24:28Z,2021-06-25T23:24:28Z,OWNER,Maybe I could release a separate Python package `types-sqlite-utils-numpy` which adds an over-ridden type definition that includes the `numpy` types?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints,
https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881033,https://api.github.com/repos/simonw/sqlite-utils/issues/37,868881033,MDEyOklzc3VlQ29tbWVudDg2ODg4MTAzMw==,9599,simonw,2021-06-25T23:23:49Z,2021-06-25T23:23:49Z,OWNER,"Twitter conversation about how to add types to the `.create_table(columns=)` parameter: https://twitter.com/simonw/status/1408532867592818693
> Anyone know how to write a mypy type definition for this?
>
> {""id"": int, ""name"": str, ""image"": bytes, ""weight"": float}
>
> It's a dict where keys are strings and values are one of int/str/bytes/float (weird API design I know, but I designed this long before I was thinking about mypy)
Looks like this could work:
```python
def create_table(
self,
name,
columns: Dict[str, Union[Type[int], Type[bytes], Type[str], Type[float]]],
pk=None,
foreign_keys=None,
column_order=None,
not_null=None,
defaults=None,
hash_id=None,
extracts=None,
):
```
Except... that method can optionally also accept `numpy` types if `numpy` is installed. I don't know if it's possible to dynamically change a signature based on an import, since `mypy` is a static type analyzer and doesn't ever execute the code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints,
https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868728092,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868728092,MDEyOklzc3VlQ29tbWVudDg2ODcyODA5Mg==,9599,simonw,2021-06-25T17:39:35Z,2021-06-25T17:39:35Z,OWNER,Here's more about this problem: https://github.com/numpy/numpy/issues/15947,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",929748885,Test against Python 3.10-dev,
https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868134040,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868134040,MDEyOklzc3VlQ29tbWVudDg2ODEzNDA0MA==,9599,simonw,2021-06-25T01:49:44Z,2021-06-25T01:50:33Z,OWNER,"Test failed on 3.10 with `numpy` on macOS:
```
sqlite_utils/__init__.py:1: in
11
from .db import Database
12
sqlite_utils/db.py:48: in
13
import numpy as np # type: ignore
14
../../../hostedtoolcache/Python/3.10.0-beta.3/x64/lib/python3.10/site-packages/numpy/__init__.py:391: in
15
raise RuntimeError(msg)
16
E RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, more information is available at https://numpy.org/doc/stable/user/building.html#accelerated-blas-lapack-libraries Otherwise report this to the vendor that provided NumPy.
17
E RankWarning: Polyfit may be poorly conditioned
18
Error: Process completed with exit code 4.
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",929748885,Test against Python 3.10-dev,
https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868125750,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868125750,MDEyOklzc3VlQ29tbWVudDg2ODEyNTc1MA==,22429695,codecov[bot],2021-06-25T01:42:43Z,2021-06-25T01:42:43Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#293](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ae0f46a) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/747be6057d09a4e5d9d726e29d5cf99b10c59dea?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (747be60) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #293 +/- ##
=======================================
Coverage 96.03% 96.03%
=======================================
Files 4 4
Lines 1994 1994
=======================================
Hits 1915 1915
Misses 79 79
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [747be60...ae0f46a](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",929748885,Test against Python 3.10-dev,
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-868021624,https://api.github.com/repos/simonw/sqlite-utils/issues/290,868021624,MDEyOklzc3VlQ29tbWVudDg2ODAyMTYyNA==,9599,simonw,2021-06-24T23:17:38Z,2021-06-24T23:17:38Z,OWNER,The new documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#executing-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/datasette/issues/1377#issuecomment-867209791,https://api.github.com/repos/simonw/datasette/issues/1377,867209791,MDEyOklzc3VlQ29tbWVudDg2NzIwOTc5MQ==,9599,simonw,2021-06-23T22:51:32Z,2021-06-23T22:51:32Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#skip-csrf-datasette-scope,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920884085,Mechanism for plugins to exclude certain paths from CSRF checks,
https://github.com/simonw/datasette/pull/1368#issuecomment-867102944,https://api.github.com/repos/simonw/datasette/issues/1368,867102944,MDEyOklzc3VlQ29tbWVudDg2NzEwMjk0NA==,9599,simonw,2021-06-23T19:32:01Z,2021-06-23T19:32:01Z,OWNER,"Yes, let's move ahead with getting this into an alpha.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/sqlite-utils/issues/291#issuecomment-866466388,https://api.github.com/repos/simonw/sqlite-utils/issues/291,866466388,MDEyOklzc3VlQ29tbWVudDg2NjQ2NjM4OA==,9599,simonw,2021-06-23T02:10:24Z,2021-06-23T02:10:24Z,OWNER,This already helped me spot a bug in a test: https://github.com/simonw/sqlite-utils/commit/90e211e3e2f36d2ff911ecf1afe4470ff45c7c0d#diff-4e8715c7a425ee52e74b7df4d34efd32e8c92f3e60bd51bc2e1ad5943b82032eL1160-L1161,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",927766296,Adopt flake8,
https://github.com/simonw/sqlite-utils/issues/291#issuecomment-866461926,https://api.github.com/repos/simonw/sqlite-utils/issues/291,866461926,MDEyOklzc3VlQ29tbWVudDg2NjQ2MTkyNg==,9599,simonw,2021-06-23T01:59:57Z,2021-06-23T01:59:57Z,OWNER,"That shouldn't be failing: it's complaining about these: https://github.com/simonw/sqlite-utils/blob/02898bf7af4a4e484ecc8ec852d5fee98463277b/tests/test_register_function.py#L56-L67
But I added `# noqa: F811` to them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",927766296,Adopt flake8,
https://github.com/simonw/sqlite-utils/issues/289#issuecomment-866219755,https://api.github.com/repos/simonw/sqlite-utils/issues/289,866219755,MDEyOklzc3VlQ29tbWVudDg2NjIxOTc1NQ==,9599,simonw,2021-06-22T18:13:26Z,2021-06-22T18:13:26Z,OWNER,Thanks @adamchainz - `mypy` now has a foothold on this project (and runs in CI).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925677191,Mypy fixes for rows_from_file(),
https://github.com/simonw/sqlite-utils/issues/289#issuecomment-866241836,https://api.github.com/repos/simonw/sqlite-utils/issues/289,866241836,MDEyOklzc3VlQ29tbWVudDg2NjI0MTgzNg==,857609,adamchainz,2021-06-22T18:44:36Z,2021-06-22T18:44:36Z,NONE,Great!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925677191,Mypy fixes for rows_from_file(),
https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866184260,https://api.github.com/repos/simonw/sqlite-utils/issues/267,866184260,MDEyOklzc3VlQ29tbWVudDg2NjE4NDI2MA==,9599,simonw,2021-06-22T17:26:18Z,2021-06-22T17:27:27Z,OWNER,"If an`.update()` method doesn't work because it collides with an existing dictionary method a `.pk` property could still be nice:
```python
for row in db[""sometable""].rows:
db[""sometable""].update(row.pk, {""modified"": 1})
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk,
https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866182655,https://api.github.com/repos/simonw/sqlite-utils/issues/267,866182655,MDEyOklzc3VlQ29tbWVudDg2NjE4MjY1NQ==,9599,simonw,2021-06-22T17:24:03Z,2021-06-22T17:24:03Z,OWNER,"I'm re-opening this as a research task because it may be possible to cleanly implement this using a `dict` subclass - some notes on that here: https://treyhunner.com/2019/04/why-you-shouldnt-inherit-from-list-and-dict-in-python/
Since this would just be for adding methods (and maybe a property for returning the primary keys for a row) the usual disadvantages of subclassing `dict` described in that article shouldn't apply.
One catch: dictionaries already have a `.update()` method! So would have to pick another name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk,
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865511810,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865511810,MDEyOklzc3VlQ29tbWVudDg2NTUxMTgxMA==,9599,simonw,2021-06-22T04:07:34Z,2021-06-22T18:26:21Z,OWNER,"That documentation section is pretty weak at the moment - here's the whole thing:
> ### Executing queries
>
> The `db.execute()` and `db.executescript()` methods provide wrappers around `.execute()` and `.executescript()` on the underlying SQLite connection. These wrappers log to the tracer function if one has been registered.
> ```python
> db = Database(memory=True)
> db[""dogs""].insert({""name"": ""Cleo""})
> db.execute(""update dogs set name = 'Cleopaws'"")
> ```
> You can pass parameters as an optional second argument, using either a list or a dictionary. These will be correctly quoted and escaped.
> ```python
> # Using ? and a list:
> db.execute(""update dogs set name = ?"", [""Cleopaws""])
> # Or using :name and a dictionary:
> db.execute(""update dogs set name = :name"", {""name"": ""Cleopaws""})
> ```
- Talks about `.execute()` - I want to talk about `.query()` instead
- Doesn't clarify that `.execute()` returns a `Cursor` - and assumes you know what to do with one
- Doesn't show an example of a `select` query at all
- The ""tracer function"" bit is confusing (should at least link to docs further down)
- For `UPDATE` should show how to access the number of rows modified (probably using `.execute()` there)
It does at least cover the two types of parameters, though that could be bulked out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865510796,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865510796,MDEyOklzc3VlQ29tbWVudDg2NTUxMDc5Ng==,9599,simonw,2021-06-22T04:04:40Z,2021-06-22T04:04:48Z,OWNER,"Still needs documentation, which will involve rewriting the whole [Executing queries](https://sqlite-utils.datasette.io/en/3.11/python-api.html#executing-queries) section.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865497846,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865497846,MDEyOklzc3VlQ29tbWVudDg2NTQ5Nzg0Ng==,9599,simonw,2021-06-22T03:21:38Z,2021-06-22T03:21:38Z,OWNER,"The Python docs say: https://docs.python.org/3/library/sqlite3.html
> To retrieve data after executing a SELECT statement, you can either treat the cursor as an iterator, call the cursor’s `fetchone()` method to retrieve a single matching row, or call `fetchall()` to get a list of the matching rows.
Looking at the C source code, both `fetchmany()` and `fetchall()` work under the hood by assembling a Python list: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L907-L972 - see calls to `PyList_Append()`
So it looks like the most efficient way to iterate over a cursor may well be `for row in cursor:` - which I think calls this C function: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L813-L876","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865495370,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865495370,MDEyOklzc3VlQ29tbWVudDg2NTQ5NTM3MA==,9599,simonw,2021-06-22T03:14:30Z,2021-06-22T03:14:30Z,OWNER,"One small problem with the existing method:
https://github.com/simonw/sqlite-utils/blob/8cedc6a8b29180e68326f6b76f249d5e39e4b591/sqlite_utils/db.py#L362-L365
It returns a full list, but what if the user would rather have a generator they can iterate over without loading the results into memory in one go?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865491922,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865491922,MDEyOklzc3VlQ29tbWVudDg2NTQ5MTkyMg==,9599,simonw,2021-06-22T03:05:35Z,2021-06-22T03:05:35Z,OWNER,"Potential names:
- `db.query(sql)` - it's weird to have both this and `db.execute()` but it is at least short and memorable
- `db.sql(sql)`
- `db.execute_d(sql)` - ugly
- `db.execute_dicts(sql)` - confusing
- `db.execute_sql(sql)` - easily confused with `db.execute(sql)`
I think `db.query(sql)` may be the best option here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/datasette/pull/1368#issuecomment-865204472,https://api.github.com/repos/simonw/datasette/issues/1368,865204472,MDEyOklzc3VlQ29tbWVudDg2NTIwNDQ3Mg==,2670795,brandonrobertz,2021-06-21T17:11:37Z,2021-06-21T17:11:37Z,CONTRIBUTOR,If this is a concept ACK then I will move onto fixing the tests (adding new ones) and updating the documentation for the new plugin hook.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/datasette/pull/1368#issuecomment-865160132,https://api.github.com/repos/simonw/datasette/issues/1368,865160132,MDEyOklzc3VlQ29tbWVudDg2NTE2MDEzMg==,9599,simonw,2021-06-21T16:07:06Z,2021-06-21T16:08:48Z,OWNER,"A few tests failed - Black, the test that checks the docs mention the new hook - the most interesting failing test looks like this one:
```
updated_metadata[""databases""][""fixtures""][""queries""][""magic_parameters""][
""allow""
] = (allow if ""query"" in permissions else deny)
> cascade_app_client.ds._metadata = updated_metadata
E AttributeError: can't set attribute
```
From https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_permissions.py#L439-L467
This test is directly manipulating `_metadata` purely for the purposes of simulating different permissions - I think updating it to manipulate `_local_metadata` instead would fix that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864621099,https://api.github.com/repos/simonw/sqlite-utils/issues/278,864621099,MDEyOklzc3VlQ29tbWVudDg2NDYyMTA5OQ==,601708,mcint,2021-06-20T22:39:57Z,2021-06-20T22:39:57Z,CONTRIBUTOR,"Fair. I looked into it, it looks like it could be done, but it would be _a bit ugly_. I can upload and link a gist of my exploration. **Click** can parse a first argument while still recognizing it as a sub-command keyword. From there, the program could:
1. ignore it preemptively if it matches a sub-command
2. and/or check if a (db) file exists at the path.
It would then also need to set a shared db argument variable.
Click also makes it easy to parse arguments from environment variables. If you're amenable, I may submit a patch for only that, which would update each sub-command to check for a DB/SQLITE_UTILS_DB environment variable. The goal would be usage that looks like: `DB=./convenient.db sqlite-utils [operation] [args]`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923697888,"Support db as first parameter before subcommand, or as environment variable",
https://github.com/simonw/sqlite-utils/issues/289#issuecomment-864609271,https://api.github.com/repos/simonw/sqlite-utils/issues/289,864609271,MDEyOklzc3VlQ29tbWVudDg2NDYwOTI3MQ==,9599,simonw,2021-06-20T20:42:07Z,2021-06-20T20:42:07Z,OWNER,"Wow, thank you! I didn't know about `typing.cast()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925677191,Mypy fixes for rows_from_file(),
https://github.com/simonw/sqlite-utils/issues/286#issuecomment-864594956,https://api.github.com/repos/simonw/sqlite-utils/issues/286,864594956,MDEyOklzc3VlQ29tbWVudDg2NDU5NDk1Ng==,9599,simonw,2021-06-20T18:38:05Z,2021-06-20T18:38:05Z,OWNER,3.10 is out in Homebrew now (they turn that around so fast): https://formulae.brew.sh/formula/sqlite-utils,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925487946,Add installation instructions,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864419283,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864419283,MDEyOklzc3VlQ29tbWVudDg2NDQxOTI4Mw==,9599,simonw,2021-06-19T15:15:34Z,2021-06-19T15:15:34Z,OWNER,"I think this code is at fault: https://github.com/simonw/sqlite-utils/blob/5b257949d996fe43dc5d218d4308b88796a90740/sqlite_utils/db.py#L1017-L1023
It's using `.pks` which adds `rowid` if it's missing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418795,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864418795,MDEyOklzc3VlQ29tbWVudDg2NDQxODc5NQ==,9599,simonw,2021-06-19T15:11:05Z,2021-06-19T15:11:14Z,OWNER,"Actually I'm going to go with `use_rowid` instead - because the table doesn't inherently use a rowid itself, but you should use one if you want to query it in a way that gives you back a primary key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418188,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864418188,MDEyOklzc3VlQ29tbWVudDg2NDQxODE4OA==,9599,simonw,2021-06-19T15:05:53Z,2021-06-19T15:05:53Z,OWNER,"```python
@property
def uses_rowid(self):
return not any(column for column in self.columns if column.is_pk)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417808,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417808,MDEyOklzc3VlQ29tbWVudDg2NDQxNzgwOA==,9599,simonw,2021-06-19T15:03:00Z,2021-06-19T15:03:00Z,OWNER,I think I like `table.uses_rowid` best - it reads well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417765,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417765,MDEyOklzc3VlQ29tbWVudDg2NDQxNzc2NQ==,9599,simonw,2021-06-19T15:02:42Z,2021-06-19T15:02:42Z,OWNER,"Some options:
- `table.rowid_only`
- `table.rowid_as_pk`
- `table.no_pks`
- `table.no_pk`
- `table.uses_rowid`
- `table.use_rowid`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417493,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417493,MDEyOklzc3VlQ29tbWVudDg2NDQxNzQ5Mw==,9599,simonw,2021-06-19T15:00:43Z,2021-06-19T15:00:43Z,OWNER,"I have to be careful about the language I use here. Here's the official definition: https://www.sqlite.org/rowidtable.html
> A ""rowid table"" is any table in an SQLite schema that
>
> - is *not* a [virtual table](https://www.sqlite.org/vtab.html), and
> - is *not* a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table.
>
> Most tables in a typical SQLite database schema are rowid tables.
>
> Rowid tables are distinguished by the fact that they all have a unique, non-NULL, signed 64-bit integer [rowid](https://www.sqlite.org/lang_createtable.html#rowid) that is used as the access key for the data in the underlying [B-tree](https://www.sqlite.org/fileformat2.html#btree) storage engine.
So it's not correct to call a table a ""rowid table"" only if it is missing its own primary keys.
Maybe `table.has_rowid` is the right language to use here? No, that's no good - because tables with their own primary keys usually also have a rowid.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417133,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417133,MDEyOklzc3VlQ29tbWVudDg2NDQxNzEzMw==,9599,simonw,2021-06-19T14:57:36Z,2021-06-19T14:57:36Z,OWNER,"So the logic is:
```python
[column.name for column in self.columns if column.is_pk]
```
I need to decide on a property name. Existing names are documented here: https://sqlite-utils.datasette.io/en/stable/python-api.html#introspection","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417031,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417031,MDEyOklzc3VlQ29tbWVudDg2NDQxNzAzMQ==,9599,simonw,2021-06-19T14:56:45Z,2021-06-19T14:56:45Z,OWNER,"```pycon
>>> db = sqlite_utils.Database(memory=True)
>>> db[""rowid_table""].insert({""name"": ""Cleo""})
>>> db[""regular_table""].insert({""id"": 1, ""name"": ""Cleo""}, pk=""id"")
>>> db[""rowid_table""].pks
['rowid']
>>> db[""regular_table""].pks
['id']
```
But that's because the `.pks` property hides the difference: https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416911,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864416911,MDEyOklzc3VlQ29tbWVudDg2NDQxNjkxMQ==,9599,simonw,2021-06-19T14:55:45Z,2021-06-19T14:55:45Z,OWNER,"https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810
So I can indeed detect a `rowid` table by looking for no `is_pk` columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416785,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864416785,MDEyOklzc3VlQ29tbWVudDg2NDQxNjc4NQ==,9599,simonw,2021-06-19T14:54:41Z,2021-06-19T14:54:41Z,OWNER,"```pycon
>>> db = sqlite_utils.Database(memory=True)
>>> db[""rowid_table""].insert({""name"": ""Cleo""})
>>> db[""regular_table""].insert({""id"": 1, ""name"": ""Cleo""}, pk=""id"")
>>> db[""rowid_table""].pks
['rowid']
>>> db[""regular_table""].pks
['id']
```
I think I need an introspection property for working out if a table is a `rowid` table or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/283#issuecomment-864416086,https://api.github.com/repos/simonw/sqlite-utils/issues/283,864416086,MDEyOklzc3VlQ29tbWVudDg2NDQxNjA4Ng==,9599,simonw,2021-06-19T14:49:06Z,2021-06-19T14:49:13Z,OWNER,"Once again, this is difficult because of the use of a generator here - `rows_from_file()` only yields rows, so there is no obvious mechanism for it to communicate back to the wrapping code that the detected format was CSV or TSV as opposed to JSON.
I'm going to change `rows_from_file()` to return a `(generator, detected_format)` tuple.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925319214,memory: Shouldn't detect types for JSON,
https://github.com/simonw/datasette/issues/1382#issuecomment-864480051,https://api.github.com/repos/simonw/datasette/issues/1382,864480051,MDEyOklzc3VlQ29tbWVudDg2NDQ4MDA1MQ==,9599,simonw,2021-06-20T00:20:06Z,2021-06-20T00:21:02Z,OWNER,"Yes you can - thanks for pointing this out, I've added a comment to the `install.sh` script in the `datasette-csvs` Glitch project:
```bash
pip3 install -U --no-cache-dir -r requirements.txt --user && \
mkdir -p .data && \
rm .data/data.db || true && \
for f in *.csv
do
# Add --encoding=latin-1 to the following if your CSVs use a different encoding:
sqlite-utils insert .data/data.db ${f%.*} $f --csv
done
```
So if you edit that file in your own project and change the line to this:
sqlite-utils insert .data/data.db ${f%.*} $f --csv --encoding=iso-8859-1
It should fix this for you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925406964,Datasette with Glitch - is it possible to use CSV with ISO-8859-1 encoding?,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864476167,https://api.github.com/repos/simonw/sqlite-utils/issues/272,864476167,MDEyOklzc3VlQ29tbWVudDg2NDQ3NjE2Nw==,9599,simonw,2021-06-19T23:36:48Z,2021-06-19T23:36:48Z,OWNER,Wrote this up on my blog here: https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/ - with a video demo here: https://www.youtube.com/watch?v=OUjd0rkc678,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864330508,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864330508,MDEyOklzc3VlQ29tbWVudDg2NDMzMDUwOA==,9599,simonw,2021-06-19T00:34:24Z,2021-06-19T00:34:24Z,OWNER,"Got this working:
% curl 'https://api.github.com/repos/simonw/datasette/issues' | sqlite-utils memory - 'select id from stdin' ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864328927,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864328927,MDEyOklzc3VlQ29tbWVudDg2NDMyODkyNw==,9599,simonw,2021-06-19T00:25:08Z,2021-06-19T00:25:17Z,OWNER,"I tried writing this function with type hints, but eventually gave up:
```python
def rows_from_file(
fp: BinaryIO,
format: Optional[Format] = None,
dialect: Optional[Type[csv.Dialect]] = None,
encoding: Optional[str] = None,
) -> Generator[dict, None, None]:
if format == Format.JSON:
decoded = json.load(fp)
if isinstance(decoded, dict):
decoded = [decoded]
if not isinstance(decoded, list):
raise RowsFromFileBadJSON(""JSON must be a list or a dictionary"")
yield from decoded
elif format == Format.CSV:
decoded_fp = io.TextIOWrapper(fp, encoding=encoding or ""utf-8-sig"")
yield from csv.DictReader(decoded_fp)
elif format == Format.TSV:
yield from rows_from_file(
fp, format=Format.CSV, dialect=csv.excel_tab, encoding=encoding
)
elif format is None:
# Detect the format, then call this recursively
buffered = io.BufferedReader(fp, buffer_size=4096)
first_bytes = buffered.peek(2048).strip()
if first_bytes[0] in (b""["", b""{""):
# TODO: Detect newline-JSON
yield from rows_from_file(fp, format=Format.JSON)
else:
dialect = csv.Sniffer().sniff(first_bytes.decode(encoding, ""ignore""))
yield from rows_from_file(
fp, format=Format.CSV, dialect=dialect, encoding=encoding
)
else:
raise RowsFromFileError(""Bad format"")
```
mypy said:
```
sqlite_utils/utils.py:157: error: Argument 1 to ""BufferedReader"" has incompatible type ""BinaryIO""; expected ""RawIOBase""
sqlite_utils/utils.py:163: error: Argument 1 to ""decode"" of ""bytes"" has incompatible type ""Optional[str]""; expected ""str""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/281#issuecomment-864323438,https://api.github.com/repos/simonw/sqlite-utils/issues/281,864323438,MDEyOklzc3VlQ29tbWVudDg2NDMyMzQzOA==,9599,simonw,2021-06-18T23:55:06Z,2021-06-18T23:55:06Z,OWNER,"The `-:json` idea is flawed: Click thinks that's the syntax for an option called `:json`.
I'm going to do `stdin:json` - which means you can't open a file called `stdin` - but you could use `cat stdin | sqlite-utils memory stdin:json ...` instead which is an OK workaround.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924992318,Mechanism for explicitly stating CSV or JSON or TSV for sqlite-utils memory,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358951,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864358951,MDEyOklzc3VlQ29tbWVudDg2NDM1ODk1MQ==,9599,simonw,2021-06-19T05:30:00Z,2021-06-19T05:30:00Z,OWNER,If this can be fixed it will be in the `transform_sql()` method.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358680,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864358680,MDEyOklzc3VlQ29tbWVudDg2NDM1ODY4MA==,9599,simonw,2021-06-19T05:27:13Z,2021-06-19T05:27:13Z,OWNER,How easy is it to detect a `rowid` table? Is it as simple as `.pks` returning `None`? If so the documentation should mention that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864354627,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864354627,MDEyOklzc3VlQ29tbWVudDg2NDM1NDYyNw==,9599,simonw,2021-06-19T04:42:03Z,2021-06-19T04:42:03Z,OWNER,"Demo:
curl -s 'https://api.github.com/users/simonw/repos?per_page=100' | \
sqlite-utils memory - 'select sum(size), sum(stargazers_count) from stdin limit 1'
[{""sum(size)"": 2042547, ""sum(stargazers_count)"": 6769}]
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data,
https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864350407,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864350407,MDEyOklzc3VlQ29tbWVudDg2NDM1MDQwNw==,9599,simonw,2021-06-19T03:52:20Z,2021-06-19T03:52:20Z,OWNER,I'll have an environment variable for `--detect-types` so users who really want that as the default option can turn it on.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data,
https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864349123,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864349123,MDEyOklzc3VlQ29tbWVudDg2NDM0OTEyMw==,9599,simonw,2021-06-19T03:36:54Z,2021-06-19T03:36:54Z,OWNER,"I may change the default for `sqlite-utils insert` to detect types if I release `sqlite-utils` 4.0, as a backwards-incompatible change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data,
https://github.com/simonw/sqlite-utils/issues/179#issuecomment-864349066,https://api.github.com/repos/simonw/sqlite-utils/issues/179,864349066,MDEyOklzc3VlQ29tbWVudDg2NDM0OTA2Ng==,9599,simonw,2021-06-19T03:36:04Z,2021-06-19T03:36:04Z,OWNER,This work is going to happen in #282.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types,
https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864348954,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864348954,MDEyOklzc3VlQ29tbWVudDg2NDM0ODk1NA==,9599,simonw,2021-06-19T03:34:42Z,2021-06-19T03:35:46Z,OWNER,"I built some prototype code here for something which looks at every row in a CSV import and records the likely types: https://gist.github.com/simonw/465f9356f175d1cf86957947dff501d4
This could be used by the command-line tools to figure out what `table.transform(types=...)` method to use at the end.
This is a different approach to the pure SQL version I tried building in https://github.com/simonw/sqlite-utils/issues/179 - I think this is a better approach though, it's less prone to weird idiosyncrasies of SQLite types, and it's also easy for us to add on to the existing CSV import code in a way that won't require scanning the data twice.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864208476,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864208476,MDEyOklzc3VlQ29tbWVudDg2NDIwODQ3Ng==,9599,simonw,2021-06-18T18:30:08Z,2021-06-18T23:30:19Z,OWNER,"So maybe this is a function which can either be told the format or, if none is provided, it detects one for itself.
```python
def rows_from_file(fp, format=None):
# ...
yield from rows
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864207841,MDEyOklzc3VlQ29tbWVudDg2NDIwNzg0MQ==,9599,simonw,2021-06-18T18:28:40Z,2021-06-18T18:28:46Z,OWNER,"```python
def detect_format(fp):
# ...
return ""csv"", fp, dialect
# or
return ""json"", fp, parsed_data
# or
return ""json-nl"", fp, docs
```
The mixed return types here are ugly. In all of these cases what we really want is to return a generator of `{...}` objects. So maybe it returns that instead.
```python
def filepointer_to_documents(fp):
# ...
yield from documents
```
I can refactor `sqlite-utils insert` to use this new code too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864206308,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864206308,MDEyOklzc3VlQ29tbWVudDg2NDIwNjMwOA==,9599,simonw,2021-06-18T18:25:04Z,2021-06-18T18:25:04Z,OWNER,"Or... since I'm not using a streaming JSON parser at the moment, if I think something is JSON I can load the entire thing into memory to validate it.
I still need to detect newline-delimited JSON. For that I can consume the first line of the input to see if it's a valid JSON object, then maybe sniff the second line too?
This does mean that if the input is a single line of GIANT JSON it will all be consumed into memory at once, but that's going to happen anyway.
So I need a function which, given a file pointer, consumes from it, detects the type, then returns that type AND a file pointer to the beginning of the file again. I can use `io.BufferedReader` for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864129273,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864129273,MDEyOklzc3VlQ29tbWVudDg2NDEyOTI3Mw==,9599,simonw,2021-06-18T15:47:47Z,2021-06-18T15:47:47Z,OWNER,"Detecting valid JSON is tricky - just because a stream starts with `[` or `{` doesn't mean the entire stream is valid JSON. You need to parse the entire stream to determine that for sure.
One way to solve this would be with a custom state machine. Another would be to use the `ijson` streaming parser - annoyingly it throws the same exception class for invalid JSON for different reasons, but the `e.args[0]` for that exception includes human-readable text about the error - if it's anything other than `parse error: premature EOF` then it probably means the JSON was invalid.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864128489,https://api.github.com/repos/simonw/sqlite-utils/issues/278,864128489,MDEyOklzc3VlQ29tbWVudDg2NDEyODQ4OQ==,9599,simonw,2021-06-18T15:46:24Z,2021-06-18T15:46:24Z,OWNER,A workaround could be to define a bash or zsh alias of some sort.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923697888,"Support db as first parameter before subcommand, or as environment variable",
https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864126781,https://api.github.com/repos/simonw/sqlite-utils/issues/278,864126781,MDEyOklzc3VlQ29tbWVudDg2NDEyNjc4MQ==,9599,simonw,2021-06-18T15:43:19Z,2021-06-18T15:43:19Z,OWNER,"I don't think it's possible to do this without breaking backwards compatibility, unfortunately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923697888,"Support db as first parameter before subcommand, or as environment variable",
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864103005,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864103005,MDEyOklzc3VlQ29tbWVudDg2NDEwMzAwNQ==,9599,simonw,2021-06-18T15:04:15Z,2021-06-18T15:04:15Z,OWNER,"To detect JSON, check to see if the stream starts with `[` or `{` - maybe do something more sophisticated than that.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864101267,https://api.github.com/repos/simonw/sqlite-utils/issues/272,864101267,MDEyOklzc3VlQ29tbWVudDg2NDEwMTI2Nw==,9599,simonw,2021-06-18T15:01:41Z,2021-06-18T15:01:41Z,OWNER,I'll split the remaining work out into separate issues.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-864099764,https://api.github.com/repos/simonw/sqlite-utils/issues/273,864099764,MDEyOklzc3VlQ29tbWVudDg2NDA5OTc2NA==,9599,simonw,2021-06-18T14:59:27Z,2021-06-18T14:59:27Z,OWNER,I'm going to merge this as-is and work on the JSON/TSV support in a separate issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/277#issuecomment-864092515,https://api.github.com/repos/simonw/sqlite-utils/issues/277,864092515,MDEyOklzc3VlQ29tbWVudDg2NDA5MjUxNQ==,9599,simonw,2021-06-18T14:47:57Z,2021-06-18T14:47:57Z,OWNER,This is a neat improvement.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",923612361,add -h support closes #276,
https://github.com/simonw/datasette/pull/1378#issuecomment-863230355,https://api.github.com/repos/simonw/datasette/issues/1378,863230355,MDEyOklzc3VlQ29tbWVudDg2MzIzMDM1NQ==,22429695,codecov[bot],2021-06-17T13:16:17Z,2021-06-17T13:16:17Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1378](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0c132d1) into [main](https://codecov.io/gh/simonw/datasette/commit/83e9c8bc7585dcc62f200e37c2daefcd669ee05e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (83e9c8b) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #1378 +/- ##
=======================================
Coverage 91.68% 91.68%
=======================================
Files 34 34
Lines 4340 4340
=======================================
Hits 3979 3979
Misses 361 361
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [83e9c8b...0c132d1](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923910375,"Update pytest-xdist requirement from <2.3,>=2.2.1 to >=2.2.1,<2.4",
https://github.com/simonw/sqlite-utils/pull/277#issuecomment-863205049,https://api.github.com/repos/simonw/sqlite-utils/issues/277,863205049,MDEyOklzc3VlQ29tbWVudDg2MzIwNTA0OQ==,22429695,codecov[bot],2021-06-17T12:40:49Z,2021-06-17T12:40:49Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#277](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (abbd324) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a19ce1a4d0048d389411cfe11a5dbe4c503720e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a19ce1a) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #277 +/- ##
=======================================
Coverage 96.06% 96.06%
=======================================
Files 4 4
Lines 1828 1829 +1
=======================================
+ Hits 1756 1757 +1
Misses 72 72
```
| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/277/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `94.03% <100.00%> (+<0.01%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a19ce1a...abbd324](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923612361,add -h support closes #276,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862817185,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862817185,MDEyOklzc3VlQ29tbWVudDg2MjgxNzE4NQ==,22429695,codecov[bot],2021-06-17T00:15:34Z,2021-06-17T00:15:34Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> :exclamation: No coverage uploaded for pull request base (`main@78aebb6`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #273 +/- ##
=======================================
Coverage ? 96.10%
=======================================
Files ? 4
Lines ? 1873
Branches ? 0
=======================================
Hits ? 1800
Misses ? 73
Partials ? 0
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [78aebb6...df7a37b](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/issues/275#issuecomment-862617165,https://api.github.com/repos/simonw/sqlite-utils/issues/275,862617165,MDEyOklzc3VlQ29tbWVudDg2MjYxNzE2NQ==,9599,simonw,2021-06-16T18:34:51Z,2021-06-16T18:34:51Z,OWNER,"Also use this: https://github.com/simonw/datasette/blob/83e9c8bc7585dcc62f200e37c2daefcd669ee05e/codecov.yml
And add a badge, as seen on https://github.com/simonw/asgi-csrf","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922955697,Enable code coverage,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862605436,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862605436,MDEyOklzc3VlQ29tbWVudDg2MjYwNTQzNg==,9599,simonw,2021-06-16T18:19:05Z,2021-06-16T18:19:05Z,OWNER,`--attach` documentation: https://github.com/simonw/sqlite-utils/blob/192dc2c5b73bd836ab8e2e5fed4b36c6ea02f250/docs/cli.rst#joining-in-memory-data-against-existing-databases-using-attach,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862574390,https://api.github.com/repos/simonw/sqlite-utils/issues/270,862574390,MDEyOklzc3VlQ29tbWVudDg2MjU3NDM5MA==,4068,frafra,2021-06-16T17:34:49Z,2021-06-16T17:34:49Z,NONE,"Sorry, I got confused because SQLite has a JSON column type, even if it is treated as TEXT, and I though automatic facets were available for JSON arrays stored as JSON only :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON,
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-862495803,https://api.github.com/repos/simonw/sqlite-utils/issues/131,862495803,MDEyOklzc3VlQ29tbWVudDg2MjQ5NTgwMw==,9599,simonw,2021-06-16T15:52:33Z,2021-06-16T15:52:33Z,OWNER,I like `-t` or `--type` for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042,sqlite-utils insert: options for column types,
https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862494864,https://api.github.com/repos/simonw/sqlite-utils/issues/267,862494864,MDEyOklzc3VlQ29tbWVudDg2MjQ5NDg2NA==,9599,simonw,2021-06-16T15:51:28Z,2021-06-16T16:26:15Z,OWNER,"I did add a slightly clumsy mechanism recently to help a bit here though: the `pks_and_rows_where()` method: https://sqlite-utils.datasette.io/en/stable/python-api.html#listing-rows-with-their-primary-keys
More details in the issue for that feature: #240
The idea here is that if you want to call update you need the primary key for the row - so you can do this:
```python
for pk, row in db[""sometable""].pks_and_rows_where():
db[""sometable""].update(pk, {""modified"": 1}"")
```
The `pk` may end up as a single value or a tuple depending on if the table has a compound primary key - but you don't need to worry about that if you use this method as it will return the correct primary key value for you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk,
https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862493179,https://api.github.com/repos/simonw/sqlite-utils/issues/267,862493179,MDEyOklzc3VlQ29tbWVudDg2MjQ5MzE3OQ==,9599,simonw,2021-06-16T15:49:13Z,2021-06-16T15:49:13Z,OWNER,"The big challenge here is that the rows returned by this library aren't objects, they are Python dictionaries - so adding methods to them isn't possible without changing the type that is returned by these methods.
Part of the philosophy of the library is that it should make it as easy as possible to round-trip between Python dictionaries and SQLite table data, so I don't think adding methods like this is going to fit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk,
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862491721,https://api.github.com/repos/simonw/sqlite-utils/issues/270,862491721,MDEyOklzc3VlQ29tbWVudDg2MjQ5MTcyMQ==,9599,simonw,2021-06-16T15:47:06Z,2021-06-16T15:47:06Z,OWNER,"SQLite doesn't have a JSON column type - it has JSON processing functions, but they operate against TEXT columns - so there's nothing I can do here unfortunately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862491016,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862491016,MDEyOklzc3VlQ29tbWVudDg2MjQ5MTAxNg==,9599,simonw,2021-06-16T15:46:13Z,2021-06-16T15:46:13Z,OWNER,"Columns from data imported from CSV in this way is currently treated as `TEXT`, which means numeric sorts and suchlike won't work as people might expect. It would be good to do automatic type detection here, see #179.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862485408,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862485408,MDEyOklzc3VlQ29tbWVudDg2MjQ4NTQwOA==,9599,simonw,2021-06-16T15:38:58Z,2021-06-16T15:39:28Z,OWNER,"Also `sqlite-utils memory` reflects the existing `sqlite-utils :memory:` mechanism, which is a point in its favour.
And it helps emphasize that the file you are querying will be loaded into memory, so probably don't try this against a 1GB CSV file.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862484557,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862484557,MDEyOklzc3VlQ29tbWVudDg2MjQ4NDU1Nw==,9599,simonw,2021-06-16T15:37:51Z,2021-06-16T15:38:34Z,OWNER,"I wonder if there's a better name for this than `sqlite-utils memory`?
- `sqlite-utils memory hello.csv ""select * from hello""`
- `sqlite-utils mem hello.csv ""select * from hello""`
- `sqlite-utils temp hello.csv ""select * from hello""`
- `sqlite-utils adhoc hello.csv ""select * from hello""`
- `sqlite-utils scratch hello.csv ""select * from hello""`
I think `memory` is best. I don't like the others, except for `scratch` which is OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862479704,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862479704,MDEyOklzc3VlQ29tbWVudDg2MjQ3OTcwNA==,9599,simonw,2021-06-16T15:31:31Z,2021-06-16T15:31:31Z,OWNER,"Plus, could I make this change to `sqlite-utils query` without breaking backwards compatibility? Adding a new `sqlite-utils memory` command is completely safe from that perspective.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862478881,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862478881,MDEyOklzc3VlQ29tbWVudDg2MjQ3ODg4MQ==,9599,simonw,2021-06-16T15:30:24Z,2021-06-16T15:30:24Z,OWNER,"But... `sqlite-utils my.csv ""select * from my""` is a much more compelling initial experience than `sqlite-utils memory my.csv ""select * from my""`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862475685,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862475685,MDEyOklzc3VlQ29tbWVudDg2MjQ3NTY4NQ==,9599,simonw,2021-06-16T15:26:19Z,2021-06-16T15:29:38Z,OWNER,"Here's a radical idea: what if I combined `sqlite-utils memory` into `sqlite-utils query`?
The trick here would be to detect if the arguments passed on the command-line refer to SQLite databases or if they refer to CSV/JSON data that should be imported into temporary tables.
Detecting a SQLite database file is actually really easy - they all start with the same binary string:
```pycon
>>> open(""my.db"", ""rb"").read(100)
b'SQLite format 3\x00...
```
(Need to carefully check that a CSV file with`SQLite format 3` as the first column name doesn't accidentally get interpreted as a SQLite DB though).
So then what would the semantics of `sqlite-utils query` (which is also the default command) be?
- `sqlite-utils mydb.db ""select * from x""`
- `sqlite-utils my.csv ""select * from my""`
- `sqlite-utils mydb.db my.csv ""select * from mydb.x join my on ...""` - this is where it gets weird. We can't import the CSV data directly into `mpdb.db` - it's suppose to go into the in-memory database - so now we need to start using database aliases like `mydb.x` because we passed at least one other file?
The complexity here is definitely in the handling of a combination of SQLite database files and CSV filenames. Also, `sqlite-utils query` doesn't accept multiple filenames at the moment, so that will change.
I'm not 100% sold on this as being better than having a separate `sqlite-utils memory` command, as seen in #273.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862018937,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862018937,MDEyOklzc3VlQ29tbWVudDg2MjAxODkzNw==,9599,simonw,2021-06-16T03:59:28Z,2021-06-16T04:00:05Z,OWNER,"Mainly for debugging purposes it would be useful to be able to save the created in-memory database back to a file again later. This could be done with:
sqlite-utils memory blah.csv --save saved.db
Can use `.iterdump()` to implement this: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump
Maybe instead (or as-well-as) offer `--dump` which dumps out the SQL from that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862046009,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862046009,MDEyOklzc3VlQ29tbWVudDg2MjA0NjAwOQ==,9599,simonw,2021-06-16T05:15:38Z,2021-06-16T05:15:38Z,OWNER,"I'm going to add a `--encoding` option - it will affect ALL CSV input files, so if you have CSV files with different encodings you'll need to sort that mess out yourself (likely by importing each CSV file separately into a database using `sqlite-utils insert` with different `--encoding` values).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045639,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862045639,MDEyOklzc3VlQ29tbWVudDg2MjA0NTYzOQ==,9599,simonw,2021-06-16T05:14:38Z,2021-06-16T05:14:38Z,OWNER,"Can't share much code though since a bunch of that `insert` stuff is specific to that command - showing progress bars, returning errors on illegal option combinations etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045438,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862045438,MDEyOklzc3VlQ29tbWVudDg2MjA0NTQzOA==,9599,simonw,2021-06-16T05:14:00Z,2021-06-16T05:14:00Z,OWNER,I should probably refactor the CSV/JSON/loading stuff into a function in `utils.py` in order to share some of the implementation with the existing `sqlite-utils insert` code: https://github.com/simonw/sqlite-utils/blob/287cdcae8908916687f2ecccc87c38549d004ac6/sqlite_utils/cli.py#L691-L734,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862043974,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862043974,MDEyOklzc3VlQ29tbWVudDg2MjA0Mzk3NA==,9599,simonw,2021-06-16T05:10:12Z,2021-06-16T05:10:12Z,OWNER,"I can stop promoting `:memory:` here and promote `memory` instead:
https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#L83-L86","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862042110,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862042110,MDEyOklzc3VlQ29tbWVudDg2MjA0MjExMA==,9599,simonw,2021-06-16T05:05:51Z,2021-06-16T05:06:11Z,OWNER,"Initial documentation is here: https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#running-queries-directly-against-csv-data
It only talks about CSV at the moment - needs to be updated to mention JSON too once that is implemented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040906,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862040906,MDEyOklzc3VlQ29tbWVudDg2MjA0MDkwNg==,9599,simonw,2021-06-16T05:02:47Z,2021-06-16T05:02:47Z,OWNER,"Got a prototype working!
```
% curl -s 'https://fivethirtyeight.datasettes.com/polls/president_approval_polls.csv?_size=max&_stream=1' | sqlite-utils memory - 'select * from t limit 5' --nl
{""rowid"": ""1"", ""question_id"": ""139304"", ""poll_id"": ""74225"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""568"", ""pollster"": ""YouGov"", ""sponsor_ids"": ""352"", ""sponsors"": ""Economist"", ""display_name"": ""YouGov"", ""pollster_rating_id"": ""391"", ""pollster_rating_name"": ""YouGov"", ""fte_grade"": ""B"", ""sample_size"": ""1500"", ""population"": ""a"", ""population_full"": ""a"", ""methodology"": ""Online"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://docs.cdn.yougov.com/y9zsit5bzd/weeklytrackingreport.pdf"", ""source"": ""538"", ""yes"": ""42.0"", ""no"": ""53.0""}
{""rowid"": ""2"", ""question_id"": ""139305"", ""poll_id"": ""74225"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""568"", ""pollster"": ""YouGov"", ""sponsor_ids"": ""352"", ""sponsors"": ""Economist"", ""display_name"": ""YouGov"", ""pollster_rating_id"": ""391"", ""pollster_rating_name"": ""YouGov"", ""fte_grade"": ""B"", ""sample_size"": ""1155"", ""population"": ""rv"", ""population_full"": ""rv"", ""methodology"": ""Online"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://docs.cdn.yougov.com/y9zsit5bzd/weeklytrackingreport.pdf"", ""source"": ""538"", ""yes"": ""44.0"", ""no"": ""55.0""}
{""rowid"": ""3"", ""question_id"": ""139306"", ""poll_id"": ""74226"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""23"", ""pollster"": ""American Research Group"", ""sponsor_ids"": """", ""sponsors"": """", ""display_name"": ""American Research Group"", ""pollster_rating_id"": ""9"", ""pollster_rating_name"": ""American Research Group"", ""fte_grade"": ""B"", ""sample_size"": ""1100"", ""population"": ""a"", ""population_full"": ""a"", ""methodology"": ""Live Phone"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://americanresearchgroup.com/economy/"", ""source"": ""538"", ""yes"": ""30.0"", ""no"": ""66.0""}
{""rowid"": ""4"", ""question_id"": ""139307"", ""poll_id"": ""74226"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""23"", ""pollster"": ""American Research Group"", ""sponsor_ids"": """", ""sponsors"": """", ""display_name"": ""American Research Group"", ""pollster_rating_id"": ""9"", ""pollster_rating_name"": ""American Research Group"", ""fte_grade"": ""B"", ""sample_size"": ""990"", ""population"": ""rv"", ""population_full"": ""rv"", ""methodology"": ""Live Phone"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://americanresearchgroup.com/economy/"", ""source"": ""538"", ""yes"": ""29.0"", ""no"": ""67.0""}
{""rowid"": ""5"", ""question_id"": ""139298"", ""poll_id"": ""74224"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""1528"", ""pollster"": ""AtlasIntel"", ""sponsor_ids"": """", ""sponsors"": """", ""display_name"": ""AtlasIntel"", ""pollster_rating_id"": ""546"", ""pollster_rating_name"": ""AtlasIntel"", ""fte_grade"": ""B/C"", ""sample_size"": ""5188"", ""population"": ""a"", ""population_full"": ""a"", ""methodology"": ""Online"", ""start_date"": ""1/15/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/19/21 21:52"", ""notes"": """", ""url"": ""https://projects.fivethirtyeight.com/polls/20210119_US_Atlas2.pdf"", ""source"": ""538"", ""yes"": ""44.6"", ""no"": ""53.9""}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040971,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862040971,MDEyOklzc3VlQ29tbWVudDg2MjA0MDk3MQ==,9599,simonw,2021-06-16T05:02:56Z,2021-06-16T05:02:56Z,OWNER,Moving this to a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861989987,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861989987,MDEyOklzc3VlQ29tbWVudDg2MTk4OTk4Nw==,9599,simonw,2021-06-16T02:34:21Z,2021-06-16T02:34:21Z,OWNER,"The documentation already covers this
```
$ sqlite-utils :memory: ""select sqlite_version()""
[{""sqlite_version()"": ""3.29.0""}]
```
https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-and-returning-json
`sqlite-utils memory ""select sqlite_version()""` is a little bit more intuitive than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861987651,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861987651,MDEyOklzc3VlQ29tbWVudDg2MTk4NzY1MQ==,9599,simonw,2021-06-16T02:27:20Z,2021-06-16T02:27:20Z,OWNER,Solution: `sqlite-utils memory -` attempts to detect the input based on if it starts with a `{` or `[` (likely JSON) or if it doesn't use the `csv.Sniffer()` mechanism. Or you can use `sqlite-utils memory -:csv` to specifically indicate the type of input.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861985944,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861985944,MDEyOklzc3VlQ29tbWVudDg2MTk4NTk0NA==,9599,simonw,2021-06-16T02:22:52Z,2021-06-16T02:22:52Z,OWNER,"Another option: allow an optional `:suffix` specifying the type of the file. If this is missing we detect based on the filename.
sqlite-utils memory somefile:csv ""select * from somefile""
One catch: how to treat `-` for standard input?
cat blah.csv | sqlite-utils memory - ""select * from stdin""
That's fine for CSV, but what about TSV or JSON or nl-JSON? Maybe this:
cat blah.csv | sqlite-utils memory -:json ""select * from stdin""
Bit weird though. The alternative would be to support this:
cat blah.csv | sqlite-utils memory --load-csv -
But that's verbose compared to the version without the long `--load-x` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861984707,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861984707,MDEyOklzc3VlQ29tbWVudDg2MTk4NDcwNw==,9599,simonw,2021-06-16T02:19:48Z,2021-06-16T02:19:48Z,OWNER,"This is going to need to be a separate command, for relatively non-obvious reasons.
sqlite-utils blah.db ""select * from x""
Is equivalent to this, because `query` is the default sub-command:
sqlite-utils query blah.db ""select * from x""
But... this means that making the filename optional doesn't actually work - because then this is ambiguous:
sqlite-utils --load-csv blah.csv ""select * from blah""
So instead, I'm going to add a new sub-command. I'm currently thinking `memory` to reflect that this command operates on an in-memory database:
sqlite-utils memory --load-csv blah.csv ""select * from blah""
I still think I need to use `--load-csv` rather than `--csv` because one interesting use-case for this is loading in CSV and converting it to JSON, or vice-versa.
Another option: allow multiple arguments which are filenames, and use the extension (or sniff the content) to decide what to do with them:
sqlite-utils memory blah.csv foo.csv ""select * from foo join blah on ...""
This would require the last positional argument to always be a SQL query, and would treat all other positional arguments as files that should be imported into memory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861944202,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861944202,MDEyOklzc3VlQ29tbWVudDg2MTk0NDIwMg==,25778,eyeseast,2021-06-16T01:41:03Z,2021-06-16T01:41:03Z,CONTRIBUTOR,"So, I do things like this a lot, too. I like the idea of piping in from stdin. Something like this would be nice to do in a makefile:
```sh
cat file.csv | sqlite-utils --csv --table data - 'SELECT * FROM data WHERE col=""whatever""' > filtered.csv
```
If you assumed that you're always piping out the same format you're piping in, the option names don't have to change. Depends how much you want to change formats.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891835,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861891835,MDEyOklzc3VlQ29tbWVudDg2MTg5MTgzNQ==,9599,simonw,2021-06-15T23:09:31Z,2021-06-15T23:09:31Z,OWNER,`--load-csv` and `--load-json` and `--load-nl` and `--load-tsv` are unambiguous.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891693,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861891693,MDEyOklzc3VlQ29tbWVudDg2MTg5MTY5Mw==,9599,simonw,2021-06-15T23:09:08Z,2021-06-15T23:09:08Z,OWNER,Problem: `--csv` and `--json` and `--nl` are already options for `sqlite-utils query` - need new non-conflicting names.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891272,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861891272,MDEyOklzc3VlQ29tbWVudDg2MTg5MTI3Mg==,9599,simonw,2021-06-15T23:08:02Z,2021-06-15T23:08:02Z,OWNER,"`--csv -` should work though, for reading from stdin. The table can be called `stdin`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891110,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861891110,MDEyOklzc3VlQ29tbWVudDg2MTg5MTExMA==,9599,simonw,2021-06-15T23:07:38Z,2021-06-15T23:07:38Z,OWNER,`--csvt` seems unnecessary to me: if people want to load different CSV files with the same filename (but in different directories) they will get an error unless they rename the files first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861890689,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861890689,MDEyOklzc3VlQ29tbWVudDg2MTg5MDY4OQ==,9599,simonw,2021-06-15T23:06:37Z,2021-06-15T23:06:37Z,OWNER,"How about `--json` and `--nl` and `--tsv` too? Imitating the format options for `sqlite-utils insert`.
And what happens if you provide a filename too? I'm tempted to say that the `--csv` stuff still gets loaded into an in-memory database but it's given a name and can then be joined against using SQLite `memory.blah` syntax.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861889437,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861889437,MDEyOklzc3VlQ29tbWVudDg2MTg4OTQzNw==,9599,simonw,2021-06-15T23:03:26Z,2021-06-15T23:03:26Z,OWNER,Maybe also support `--csvt` as an alternative option which takes two arguments: the CSV path and the name of the table that should be created from it (rather than auto-detecting from the filename).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/datasette/pull/1130#issuecomment-861497548,https://api.github.com/repos/simonw/datasette/issues/1130,861497548,MDEyOklzc3VlQ29tbWVudDg2MTQ5NzU0OA==,3243482,abdusco,2021-06-15T13:27:48Z,2021-06-15T13:27:48Z,CONTRIBUTOR,"There's a workaround: https://css-tricks.com/css-fix-for-100vh-in-mobile-webkit/
and a future fix: https://css-tricks.com/safari-15-new-ui-theme-colors-and-a-css-tricks-cameo/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756876238,Fix footer not sticking to bottom in short pages,
https://github.com/simonw/sqlite-utils/issues/269#issuecomment-861103967,https://api.github.com/repos/simonw/sqlite-utils/issues/269,861103967,MDEyOklzc3VlQ29tbWVudDg2MTEwMzk2Nw==,9599,simonw,2021-06-15T01:34:10Z,2021-06-15T01:34:10Z,OWNER,"SQLite doesn't have the concept of a boolean column, so there's not much I can do here: https://www.sqlite.org/datatype3.html#boolean_datatype","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919250621,bool type not supported,
https://github.com/simonw/sqlite-utils/issues/266#issuecomment-861103684,https://api.github.com/repos/simonw/sqlite-utils/issues/266,861103684,MDEyOklzc3VlQ29tbWVudDg2MTEwMzY4NA==,9599,simonw,2021-06-15T01:33:13Z,2021-06-15T01:33:13Z,OWNER,Dupe of #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913135723,"Add some types, enforce with mypy",
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861035862,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,861035862,MDEyOklzc3VlQ29tbWVudDg2MTAzNTg2Mg==,231498,khimaros,2021-06-14T22:29:20Z,2021-06-14T22:29:20Z,NONE,"it looks like the v4 GraphQL API is the only way to get data beyond 90 days from GitHub.
this is significant change, but may be worth considering in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/simonw/datasette/issues/1377#issuecomment-861089794,https://api.github.com/repos/simonw/datasette/issues/1377,861089794,MDEyOklzc3VlQ29tbWVudDg2MTA4OTc5NA==,9599,simonw,2021-06-15T00:53:29Z,2021-06-15T00:53:29Z,OWNER,"Potential hook names:
- `skip_csrf(scope, datasette)`
- ... I can't think of any other ones I would tolerate to be honest","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920884085,Mechanism for plugins to exclude certain paths from CSRF checks,
https://github.com/simonw/datasette/issues/1377#issuecomment-861087949,https://api.github.com/repos/simonw/datasette/issues/1377,861087949,MDEyOklzc3VlQ29tbWVudDg2MTA4Nzk0OQ==,9599,simonw,2021-06-15T00:49:19Z,2021-06-15T00:49:19Z,OWNER,"The new `skip_if_scope` mechanism in `asgi-csrf` https://github.com/simonw/asgi-csrf/issues/20 is designed to help here.
Now I need to design a plugin hook that allows plugins to have an opinion on whether a specific `scope` should have CSRF skipped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920884085,Mechanism for plugins to exclude certain paths from CSRF checks,
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861087651,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,861087651,MDEyOklzc3VlQ29tbWVudDg2MTA4NzY1MQ==,231498,khimaros,2021-06-15T00:48:37Z,2021-06-15T00:48:37Z,NONE,"@simonw -- i've created an omega-query that fetched most of what was interesting to me for a single user.
found by poking around in the ""Explorer"" tab in https://docs.github.com/en/graphql/overview/explorer
note: pagination is still required via `first` and `last` but it seems to allow unlimited history.
```
query MyQuery {
__typename
user(login: """") {
id
pinnedItems(first: 100) {
edges {
node
}
}
pullRequests(first: 100) {
nodes {
body
title
state
createdAt
}
}
createdAt
issues(first: 100) {
pageInfo {
endCursor
startCursor
}
nodes {
title
url
createdAt
body
}
}
issueComments(first: 100) {
edges {
node {
id
updatedAt
url
body
}
}
}
repositories(first: 100) {
nodes {
createdAt
description
parent {
name
}
pinnedIssues(first: 100) {
edges {
node {
id
}
}
}
pinnedDiscussions(first: 100) {
edges {
node {
id
}
}
}
}
}
starredRepositories(first: 100) {
edges {
node {
id
}
}
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861042050,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,861042050,MDEyOklzc3VlQ29tbWVudDg2MTA0MjA1MA==,9599,simonw,2021-06-14T22:45:42Z,2021-06-14T22:45:42Z,MEMBER,I'm definitely interested in supporting events in this tool - see #14.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861041597,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,861041597,MDEyOklzc3VlQ29tbWVudDg2MTA0MTU5Nw==,9599,simonw,2021-06-14T22:44:54Z,2021-06-14T22:44:54Z,MEMBER,Have you found a way to access events in GraphQL? I can only see way to access a timeline of events for a single issue or a single pull request. See also https://github.community/t/get-event-equivalent-for-v4/13600/2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-860895838,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,860895838,MDEyOklzc3VlQ29tbWVudDg2MDg5NTgzOA==,231498,khimaros,2021-06-14T18:23:21Z,2021-06-14T21:37:35Z,NONE,"i have a basic working version at https://github.com/khimaros/github-to-sqlite
this can be tested with `github-to-sqlite events.db khimaros/events`
caveat: the GitHub API doesn't seem to provide a complete history of events.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/simonw/datasette/issues/1375#issuecomment-860548546,https://api.github.com/repos/simonw/datasette/issues/1375,860548546,MDEyOklzc3VlQ29tbWVudDg2MDU0ODU0Ng==,4068,frafra,2021-06-14T09:41:59Z,2021-06-14T09:41:59Z,NONE,"> There is a feature for this at the moment, but it's a little bit hidden: you can use `?_json=col` to tell
> Datasette that you would like a specific column to be exported as nested JSON: https://docs.datasette.io/en/stable/json_api.html#special-json-arguments
Thanks :)
> I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning string instead of the expected dictionary and breaking your code.
If a developer is not sure if the JSON fields are valid, but then retrieves and parse them, it should handle errors too. Handling inconsistent data is necessary due to the nature of SQLite. A global or dataset option to render the data as they have been defined (JSON, boolean, etc.) when requesting JSON could allow the user to download a regular JSON from the browser without having to rely on APIs. I would guess someone could just make a custom template with an extra JSON-parsed download button otherwise :)","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919508498,JSON export dumps JSON fields as TEXT,
https://github.com/simonw/datasette/issues/1376#issuecomment-860230663,https://api.github.com/repos/simonw/datasette/issues/1376,860230663,MDEyOklzc3VlQ29tbWVudDg2MDIzMDY2Mw==,9599,simonw,2021-06-13T15:39:37Z,2021-06-13T15:39:37Z,OWNER,Actually it looks like there is a PR open already that addresses this: #1296 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919822817,Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns),
https://github.com/simonw/datasette/issues/1375#issuecomment-860230385,https://api.github.com/repos/simonw/datasette/issues/1375,860230385,MDEyOklzc3VlQ29tbWVudDg2MDIzMDM4NQ==,9599,simonw,2021-06-13T15:37:49Z,2021-06-13T15:37:49Z,OWNER,"There is a feature for this at the moment, but it's a little bit hidden: you can use `?_json=col` to tell
Datasette that you would like a specific column to be exported as nested JSON: https://docs.datasette.io/en/stable/json_api.html#special-json-arguments
I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning string instead of the expected dictionary and breaking your code.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919508498,JSON export dumps JSON fields as TEXT,
https://github.com/simonw/datasette/issues/1376#issuecomment-860229397,https://api.github.com/repos/simonw/datasette/issues/1376,860229397,MDEyOklzc3VlQ29tbWVudDg2MDIyOTM5Nw==,9599,simonw,2021-06-13T15:31:02Z,2021-06-13T15:31:02Z,OWNER,Alternative fix would be to update that section of the documentation - if the container upgrade proves tricky I can fall back on that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919822817,Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns),
https://github.com/simonw/datasette/issues/1376#issuecomment-860229226,https://api.github.com/repos/simonw/datasette/issues/1376,860229226,MDEyOklzc3VlQ29tbWVudDg2MDIyOTIyNg==,9599,simonw,2021-06-13T15:29:45Z,2021-06-13T15:29:45Z,OWNER,"Oh good catch - this is a SQLite version issue.
The `fixtures.db` file used on https://latest.datasette.io/ includes a generated column (for testing purposes) which is a feature added in SQLite 3.31.0 on 2020-01-22.
https://latest.datasette.io/-/versions
But... it looks like the packaged Datasette Docker container doesn't have that SQLite version!
I should fix that. I'm renaming this issue.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919822817,Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns),
https://github.com/simonw/sqlite-utils/issues/271#issuecomment-860142489,https://api.github.com/repos/simonw/sqlite-utils/issues/271,860142489,MDEyOklzc3VlQ29tbWVudDg2MDE0MjQ4OQ==,9599,simonw,2021-06-13T02:53:06Z,2021-06-13T02:53:06Z,OWNER,"Looks like this is the problem: https://github.com/simonw/sqlite-utils/blob/b0f9d1e494c9891ce407e27b0f5c6deeea361d30/sqlite_utils/db.py#L1724-L1742
Note how `set_cols = [col for col in all_columns if col not in pks] ` can potentially return an empty list if ALL of the columns are primary keys - but the next line of code that assigns `sql2` continues regardless, when it should instead be skipped if there are no columns in `set_cols`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919702451,table.upsert_all() fails if input has a single column that should be a primary key,
https://github.com/simonw/sqlite-utils/issues/269#issuecomment-860031217,https://api.github.com/repos/simonw/sqlite-utils/issues/269,860031217,MDEyOklzc3VlQ29tbWVudDg2MDAzMTIxNw==,4068,frafra,2021-06-12T10:01:53Z,2021-06-12T10:01:53Z,NONE,"`sqlite-utils transform` does not allow setting the column type to boolean:
```
Error: Invalid value for '--type': 'bool' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919250621,bool type not supported,
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-860031071,https://api.github.com/repos/simonw/sqlite-utils/issues/270,860031071,MDEyOklzc3VlQ29tbWVudDg2MDAzMTA3MQ==,4068,frafra,2021-06-12T10:00:24Z,2021-06-12T10:00:24Z,NONE,"Sure, I am sorry if my message hasn't been clear enough. I am also a new user :)
At the beginning, I just call `sqlite-utils insert ""$db"" ""$table"" ""$jsonfile""` to create the database. sqlite-utils convert JSON values into `TEXT`, when it tries to determine the schema automatically. I then try to transform the table to set `JSON` as type:
```
sqlite-utils transform species.sqlite species --type criteria json
Usage: sqlite-utils transform [OPTIONS] PATH TABLE
Try 'sqlite-utils transform --help' for help.
Error: Invalid value for '--type': 'json' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON,
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-860063190,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,860063190,MDEyOklzc3VlQ29tbWVudDg2MDA2MzE5MA==,232237,stiles,2021-06-12T14:46:44Z,2021-06-12T14:46:44Z,NONE,I'm having the same issue (same versions of python and twitter-to-sqlite). It's the `user-timeline` command. Other commands are working. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/simonw/datasette/issues/1286#issuecomment-860047794,https://api.github.com/repos/simonw/datasette/issues/1286,860047794,MDEyOklzc3VlQ29tbWVudDg2MDA0Nzc5NA==,4068,frafra,2021-06-12T12:36:15Z,2021-06-12T12:36:15Z,NONE,"@mroswell That is a very nice solution. I wonder if custom classes, like `col-columnName-value` could be automatically added to cells when facets on such column are enabled, to allow custom styling without having to modify templates or add custom JavaScript code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849220154,Better default display of arrays of items,
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-859986489,https://api.github.com/repos/simonw/sqlite-utils/issues/270,859986489,MDEyOklzc3VlQ29tbWVudDg1OTk4NjQ4OQ==,9599,simonw,2021-06-12T02:47:12Z,2021-06-12T02:47:12Z,OWNER,"Can you expand on what you'd like to change here? The library and CLI tool already allow JSON data to be stored in columns:
- https://sqlite-utils.datasette.io/en/stable/cli.html#nested-json-values
- https://sqlite-utils.datasette.io/en/stable/python-api.html#storing-json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON,
https://github.com/simonw/sqlite-utils/issues/269#issuecomment-859940977,https://api.github.com/repos/simonw/sqlite-utils/issues/269,859940977,MDEyOklzc3VlQ29tbWVudDg1OTk0MDk3Nw==,4068,frafra,2021-06-11T22:33:08Z,2021-06-11T22:33:08Z,NONE,"`true` and `false` json values are cast to integer, which is not optimal.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919250621,bool type not supported,
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859898736,https://api.github.com/repos/simonw/sqlite-utils/issues/268,859898736,MDEyOklzc3VlQ29tbWVudDg1OTg5ODczNg==,9599,simonw,2021-06-11T20:37:44Z,2021-06-11T20:37:44Z,OWNER,"From the prototype:
```
% sqlite-utils schema 24ways.db
CREATE TABLE [articles] (
[title] TEXT ,
[contents] TEXT ,
[year] TEXT ,
[author] TEXT ,
[author_slug] TEXT ,
[published] TEXT ,
[url] TEXT ,
[topic] TEXT
);
CREATE VIRTUAL TABLE ""articles_fts"" USING FTS5 (
title, author, contents,
content=""articles""
);
CREATE TABLE 'articles_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'articles_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE 'articles_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE 'articles_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
% sqlite-utils schema 24ways.db | sqlite3 /tmp/boo.db
Error: near line 15: table 'articles_fts_data' already exists
Error: near line 16: table 'articles_fts_idx' already exists
Error: near line 17: table 'articles_fts_docsize' already exists
Error: near line 18: table 'articles_fts_config' already exists
```
The problem here is that the `CREATE VIRTUAL TABLE ""articles_fts""...` line causes those next four tables to be created - but that means that piping the output of this command into `sqlite3` in order to re-create those tables throws errors.
I don't think this matters. I see this tool as more for introspection than for recreating table structures.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919181559,db.schema property and sqlite-utils schema command,
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859895540,https://api.github.com/repos/simonw/sqlite-utils/issues/268,859895540,MDEyOklzc3VlQ29tbWVudDg1OTg5NTU0MA==,9599,simonw,2021-06-11T20:30:34Z,2021-06-11T20:30:34Z,OWNER,"You can currently see the `sql` on the CLI using:
% sqlite-utils rows fixtures.db sqlite_master -c name -c sql
name sql
-------------------------------------------------------------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------
simple_primary_key CREATE TABLE simple_primary_key (
id varchar(30) primary key,
content text
)
sqlite_autoindex_simple_primary_key_1
primary_key_multiple_columns CREATE TABLE primary_key_multiple_columns (
id varchar(30) primary key,
content text,
content2 text
)
sqlite_autoindex_primary_key_multiple_columns_1
primary_key_multiple_columns_explicit_label CREATE TABLE primary_key_multiple_columns_explicit_label (
id varchar(30) primary key,
content text,
content2 text
)
sqlite_autoindex_primary_key_multiple_columns_explicit_label_1
compound_primary_key CREATE TABLE compound_primary_key (
pk1 varchar(30),
pk2 varchar(30),
content text,
PRIMARY KEY (pk1, pk2)
)
sqlite_autoindex_compound_primary_key_1
compound_three_primary_keys CREATE TABLE compound_three_primary_keys (
pk1 varchar(30),
pk2 varchar(30),
pk3 varchar(30),
content text,
PRIMARY KEY (pk1, pk2, pk3)
)
sqlite_autoindex_compound_three_primary_keys_1
foreign_key_references CREATE TABLE foreign_key_references (
pk varchar(30) primary key,
foreign_key_with_label varchar(30),
foreign_key_with_no_label varchar(30),
FOREIGN KEY (foreign_key_with_label) REFERENCES simple_primary_key(id),
FOREIGN KEY (foreign_key_with_no_label) REFERENCES primary_key_multiple_columns(id)
)
sqlite_autoindex_foreign_key_references_1
sortable CREATE TABLE sortable (
pk1 varchar(30),
pk2 varchar(30),
content text,
sortable integer,
sortable_with_nulls real,
sortable_with_nulls_2 real,
text text,
PRIMARY KEY (pk1, pk2)
)
sqlite_autoindex_sortable_1
no_primary_key CREATE TABLE no_primary_key (
content text,
a text,
b text,
c text
)
123_starts_with_digits CREATE TABLE [123_starts_with_digits] (
content text
)
paginated_view CREATE VIEW paginated_view AS
SELECT
content,
'- ' || content || ' -' AS content_extra
FROM no_primary_key
Table With Space In Name CREATE TABLE ""Table With Space In Name"" (
pk varchar(30) primary key,
content text
)
sqlite_autoindex_Table With Space In Name_1
table/with/slashes.csv CREATE TABLE ""table/with/slashes.csv"" (
pk varchar(30) primary key,
content text
)
sqlite_autoindex_table/with/slashes.csv_1
complex_foreign_keys CREATE TABLE ""complex_foreign_keys"" (
pk varchar(30) primary key,
f1 text,
f2 text,
f3 text,
FOREIGN KEY (""f1"") REFERENCES [simple_primary_key](id),
FOREIGN KEY (""f2"") REFERENCES [simple_primary_key](id),
FOREIGN KEY (""f3"") REFERENCES [simple_primary_key](id)
)
sqlite_autoindex_complex_foreign_keys_1
custom_foreign_key_label CREATE TABLE ""custom_foreign_key_label"" (
pk varchar(30) primary key,
foreign_key_with_custom_label text,
FOREIGN KEY (""foreign_key_with_custom_label"") REFERENCES [primary_key_multiple_columns_explicit_label](id)
)
sqlite_autoindex_custom_foreign_key_label_1
units CREATE TABLE units (
pk integer primary key,
distance int,
frequency int
)
searchable CREATE TABLE searchable (
pk integer primary key,
text1 text,
text2 text,
[name with . and spaces] text
)
searchable_fts CREATE VIRTUAL TABLE ""searchable_fts""
USING FTS3 (text1, text2, [name with . and spaces], content=""searchable"")
searchable_fts_content CREATE TABLE 'searchable_fts_content'(docid INTEGER PRIMARY KEY, 'c0text1', 'c1text2', 'c2name with . and spaces', 'c3content')
searchable_fts_segments CREATE TABLE 'searchable_fts_segments'(blockid INTEGER PRIMARY KEY, block BLOB)
searchable_fts_segdir CREATE TABLE 'searchable_fts_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx))
sqlite_autoindex_searchable_fts_segdir_1
select CREATE TABLE [select] (
[group] text,
[having] text,
[and] text
)
facet_cities CREATE TABLE facet_cities (
id integer primary key,
name text
)
simple_view CREATE VIEW simple_view AS
SELECT content, upper(content) AS upper_content FROM simple_primary_key
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919181559,db.schema property and sqlite-utils schema command,
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859894105,https://api.github.com/repos/simonw/sqlite-utils/issues/268,859894105,MDEyOklzc3VlQ29tbWVudDg1OTg5NDEwNQ==,9599,simonw,2021-06-11T20:28:52Z,2021-06-11T20:28:52Z,OWNER,"Out of interest, here are the rows from that table where `sql` is `null`: https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++sqlite_master%0D%0Awhere%0D%0A++sql+is+null
```csv
type,name,tbl_name,rootpage,sql
index,sqlite_autoindex_simple_primary_key_1,simple_primary_key,3,
index,sqlite_autoindex_primary_key_multiple_columns_1,primary_key_multiple_columns,5,
index,sqlite_autoindex_primary_key_multiple_columns_explicit_label_1,primary_key_multiple_columns_explicit_label,7,
index,sqlite_autoindex_compound_primary_key_1,compound_primary_key,9,
index,sqlite_autoindex_compound_three_primary_keys_1,compound_three_primary_keys,11,
index,sqlite_autoindex_foreign_key_references_1,foreign_key_references,14,
index,sqlite_autoindex_sortable_1,sortable,16,
index,sqlite_autoindex_Table With Space In Name_1,Table With Space In Name,20,
index,sqlite_autoindex_table/with/slashes.csv_1,table/with/slashes.csv,22,
index,sqlite_autoindex_complex_foreign_keys_1,complex_foreign_keys,24,
index,sqlite_autoindex_custom_foreign_key_label_1,custom_foreign_key_label,26,
index,sqlite_autoindex_tags_1,tags,31,
index,sqlite_autoindex_searchable_tags_1,searchable_tags,34,
index,sqlite_autoindex_searchable_fts_segdir_1,searchable_fts_segdir,37,
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919181559,db.schema property and sqlite-utils schema command,
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859888469,https://api.github.com/repos/simonw/sqlite-utils/issues/268,859888469,MDEyOklzc3VlQ29tbWVudDg1OTg4ODQ2OQ==,9599,simonw,2021-06-11T20:26:20Z,2021-06-11T20:26:20Z,OWNER,`sqlite-utils schema data.db` could output the same thing to the console.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919181559,db.schema property and sqlite-utils schema command,
https://github.com/simonw/datasette/pull/1374#issuecomment-859572791,https://api.github.com/repos/simonw/datasette/issues/1374,859572791,MDEyOklzc3VlQ29tbWVudDg1OTU3Mjc5MQ==,22429695,codecov[bot],2021-06-11T13:12:58Z,2021-06-11T13:12:58Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1374](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0ef0dd5) into [main](https://codecov.io/gh/simonw/datasette/commit/cd7678fde65319d7b6955ce9f4678ba4b9e64b66?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (cd7678f) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #1374 +/- ##
=======================================
Coverage 91.68% 91.68%
=======================================
Files 34 34
Lines 4340 4340
=======================================
Hits 3979 3979
Misses 361 361
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [cd7678f...0ef0dd5](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",918730335,Bump black from 21.5b2 to 21.6b0,
https://github.com/simonw/datasette/issues/858#issuecomment-858831895,https://api.github.com/repos/simonw/datasette/issues/858,858831895,MDEyOklzc3VlQ29tbWVudDg1ODgzMTg5NQ==,56045588,rachelll4,2021-06-10T17:44:09Z,2021-06-10T17:44:09Z,NONE,"any fixes for that recursive issue with temp file? I get it using both heroku and cloudrun, although it seems to still publish and deploy fine","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642388564,publish heroku does not work on Windows 10,
https://github.com/simonw/datasette/issues/858#issuecomment-858813675,https://api.github.com/repos/simonw/datasette/issues/858,858813675,MDEyOklzc3VlQ29tbWVudDg1ODgxMzY3NQ==,56045588,rachelll4,2021-06-10T17:27:46Z,2021-06-10T17:27:46Z,NONE,shell=True is added to line 56 (I guess it used to be 54) of heroku.py as detailed in the original issue. (for posterity),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642388564,publish heroku does not work on Windows 10,
https://github.com/simonw/datasette/issues/1371#issuecomment-858099514,https://api.github.com/repos/simonw/datasette/issues/1371,858099514,MDEyOklzc3VlQ29tbWVudDg1ODA5OTUxNA==,9599,simonw,2021-06-09T21:03:49Z,2021-06-09T21:03:49Z,OWNER,I'll release these as an alpha straight away - it makes sense to have plugin hook changes available for people to test as alpha dependencies ASAP.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915455228,Menu plugin hooks should include the request,
https://github.com/simonw/datasette/pull/1373#issuecomment-857684605,https://api.github.com/repos/simonw/datasette/issues/1373,857684605,MDEyOklzc3VlQ29tbWVudDg1NzY4NDYwNQ==,22429695,codecov[bot],2021-06-09T13:15:31Z,2021-06-13T15:34:07Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1373](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (d117ba7) into [main](https://codecov.io/gh/simonw/datasette/commit/03418ee037057aa85204f5a3feb2066cbb6a9b3e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (03418ee) will **increase** coverage by `7.65%`.
> The diff coverage is `93.29%`.
> :exclamation: Current head d117ba7 differs from pull request most recent head 51ff366. Consider uploading reports for the commit 51ff366 to get more accurate results
[](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #1373 +/- ##
==========================================
+ Coverage 84.02% 91.68% +7.65%
==========================================
Files 28 34 +6
Lines 3774 4340 +566
==========================================
+ Hits 3171 3979 +808
+ Misses 603 361 -242
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/inspect.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2luc3BlY3QucHk=) | `36.11% <ø> (ø)` | |
| [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | |
| [datasette/publish/common.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3B1Ymxpc2gvY29tbW9uLnB5) | `94.73% <ø> (ø)` | |
| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `76.33% <70.64%> (+1.19%)` | :arrow_up: |
| [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `94.35% <77.77%> (ø)` | |
| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <89.28%> (ø)` | |
| [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `90.98% <92.00%> (-0.95%)` | :arrow_down: |
| [datasette/publish/heroku.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3B1Ymxpc2gvaGVyb2t1LnB5) | `87.73% <94.44%> (+1.13%)` | :arrow_up: |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.73% <94.52%> (-0.27%)` | :arrow_down: |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.36% <94.53%> (+0.55%)` | :arrow_up: |
| ... and [29 more](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [e797565...51ff366](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",916183914,"Update trustme requirement from <0.8,>=0.7 to >=0.7,<0.9",
https://github.com/simonw/datasette/pull/1370#issuecomment-857298526,https://api.github.com/repos/simonw/datasette/issues/1370,857298526,MDEyOklzc3VlQ29tbWVudDg1NzI5ODUyNg==,25778,eyeseast,2021-06-09T01:18:59Z,2021-06-09T01:18:59Z,CONTRIBUTOR,"I'm happy to grab some or all of these in this PR, if you want. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",914130834,Ensure db.path is a string before trying to insert into internal database,
https://github.com/simonw/datasette/pull/1370#issuecomment-857139881,https://api.github.com/repos/simonw/datasette/issues/1370,857139881,MDEyOklzc3VlQ29tbWVudDg1NzEzOTg4MQ==,9599,simonw,2021-06-08T20:58:41Z,2021-06-08T20:58:41Z,OWNER,We can remove a bunch of unnecessary `str(path)` calls too - this search finds a bunch of possible candidates: https://ripgrep.datasette.io/-/ripgrep?pattern=str%5C%28.*%28db%7Cpath%29&glob=datasette%2F**%2F*.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",914130834,Ensure db.path is a string before trying to insert into internal database,
https://github.com/simonw/sqlite-utils/issues/266#issuecomment-856231119,https://api.github.com/repos/simonw/sqlite-utils/issues/266,856231119,MDEyOklzc3VlQ29tbWVudDg1NjIzMTExOQ==,9599,simonw,2021-06-07T20:26:05Z,2021-06-07T20:26:05Z,OWNER,"https://github.com/python/cpython/blob/2ab27c4af4ddf7528e1375e77c787c7fbb09b5e6/Lib/typing.py#L2173-L2195
In Python 3.6 or higher can do this:
```python
class Employee(NamedTuple):
name: str
id: int
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913135723,"Add some types, enforce with mypy",
https://github.com/simonw/datasette/issues/1365#issuecomment-856212136,https://api.github.com/repos/simonw/datasette/issues/1365,856212136,MDEyOklzc3VlQ29tbWVudDg1NjIxMjEzNg==,9599,simonw,2021-06-07T19:54:04Z,2021-06-07T19:54:04Z,OWNER,"I've hit this one too. I agree, fixing this in Datasette itself is better than fixing it in the tests across multiple other projects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913017577,pathlib.Path breaks internal schema,
https://github.com/simonw/datasette/issues/1369#issuecomment-856208637,https://api.github.com/repos/simonw/datasette/issues/1369,856208637,MDEyOklzc3VlQ29tbWVudDg1NjIwODYzNw==,9599,simonw,2021-06-07T19:47:23Z,2021-06-07T19:47:23Z,OWNER,No point in showing the IDs twice if the blue label doesn't differ from the gray ID,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913900374,Don't show foreign key IDs twice if no label,
https://github.com/simonw/datasette/pull/1368#issuecomment-856182547,https://api.github.com/repos/simonw/datasette/issues/1368,856182547,MDEyOklzc3VlQ29tbWVudDg1NjE4MjU0Nw==,2670795,brandonrobertz,2021-06-07T18:59:47Z,2021-06-07T23:04:25Z,CONTRIBUTOR,"Note that if we went with a ""update_metadata"" hook, the hook signature would look something like this (it would return nothing):
```
update_metadata(
datasette=self, metadata=metadata, key=key, database=database, table=table,
fallback=fallback
)
```
The Datasette function `_metadata_recursive_update(self, orig, updated)` would disappear into the plugins. Doing this, though, we'd lose the easy ability to make the local metadata.yaml immutable (since we'd no longer have the recursive update).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1367#issuecomment-856160770,https://api.github.com/repos/simonw/datasette/issues/1367,856160770,MDEyOklzc3VlQ29tbWVudDg1NjE2MDc3MA==,9599,simonw,2021-06-07T18:22:33Z,2021-06-07T18:22:33Z,OWNER,Here's why: https://github.com/simonw/datasette/blob/03ec71193b9545536898a4bc7493274fec48bdd7/datasette/static/app.css#L455-L458,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913823889,Navigation menu display bug,
https://github.com/simonw/datasette/issues/1366#issuecomment-856147969,https://api.github.com/repos/simonw/datasette/issues/1366,856147969,MDEyOklzc3VlQ29tbWVudDg1NjE0Nzk2OQ==,9599,simonw,2021-06-07T18:03:03Z,2021-06-07T18:03:03Z,OWNER,"Here's an example of a test that uses it. It's necessary because sometimes fixtures that create temporary directories break in unexpected ways:
https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_plugins.py#L658-L666","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913809802,Get rid of this `restore_working_directory` hack entirely,
https://github.com/simonw/datasette/issues/1366#issuecomment-856147450,https://api.github.com/repos/simonw/datasette/issues/1366,856147450,MDEyOklzc3VlQ29tbWVudDg1NjE0NzQ1MA==,9599,simonw,2021-06-07T18:02:13Z,2021-06-07T18:02:13Z,OWNER,"The hack in question is this fixture, which I've been using in an ad-hoc manner to work around errors while running the tests: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/tests/conftest.py#L62-L75
I don't understand the underlying issue well enough to know how to get rid of it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913809802,Get rid of this `restore_working_directory` hack entirely,
https://github.com/simonw/sqlite-utils/issues/266#issuecomment-855611939,https://api.github.com/repos/simonw/sqlite-utils/issues/266,855611939,MDEyOklzc3VlQ29tbWVudDg1NTYxMTkzOQ==,9599,simonw,2021-06-07T06:07:41Z,2021-06-07T06:07:41Z,OWNER,"Looks like this is the way to do this:
```python
Point = typing.NamedTuple(
""Point"", [('x', int), ('y', int)]
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913135723,"Add some types, enforce with mypy",
https://github.com/simonw/datasette/issues/1362#issuecomment-855430317,https://api.github.com/repos/simonw/datasette/issues/1362,855430317,MDEyOklzc3VlQ29tbWVudDg1NTQzMDMxNw==,9599,simonw,2021-06-06T17:07:48Z,2021-06-06T17:07:48Z,OWNER,"I guess I can offer a `disable_csp` setting so that people with complex custom templates aren't completely blocked from using them with Datasette, but maybe it would be better not to offer that? Or to offer it as a `datasette-insecure-csp` plugin instead?
I like the idea of very actively encouraging CSP across all Datasette projects, but I'm nervous about making the software unusable for certain edge cases.
Maybe require CSP and wait for someone to complain?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855429111,https://api.github.com/repos/simonw/datasette/issues/1362,855429111,MDEyOklzc3VlQ29tbWVudDg1NTQyOTExMQ==,9599,simonw,2021-06-06T16:59:05Z,2021-06-06T17:00:15Z,OWNER,"Twitter conversation: https://twitter.com/simonw/status/1401565566045806594
@dracos provided some really useful code examples there:
> We generate it here: https://github.com/mysociety/fixmystreet/blob/e9fec4e567e7148ed128816e5770c2963be51af6/perllib/FixMyStreet/Cobrand/Default.pm#L89-L90
And use it e.g. https://github.com/mysociety/fixmystreet/blob/ba6788cd25d8f471a4e3308403607627b4d2f4f6/templates/web/base/common_header_tags.html or https://github.com/mysociety/fixmystreet/blob/cb4f2b96364d151988b5c664888468b25cc62240/templates/web/fixmystreet.com/header/css.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855428601,https://api.github.com/repos/simonw/datasette/issues/1362,855428601,MDEyOklzc3VlQ29tbWVudDg1NTQyODYwMQ==,9599,simonw,2021-06-06T16:55:33Z,2021-06-06T16:55:33Z,OWNER,"> No, because Vary header is about _request_ headers that cause the response to vary, not response headers.
Hah, of course! Thanks for the correction. So the nonce mechanism would actually be pretty great here, especially for the `extra_body_script()` hook.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855428296,https://api.github.com/repos/simonw/datasette/issues/1362,855428296,MDEyOklzc3VlQ29tbWVudDg1NTQyODI5Ng==,154364,dracos,2021-06-06T16:53:20Z,2021-06-06T16:53:20Z,NONE,"> Presumably this would also require adding Content-Security-Policy to the Vary header though, which will have a nasty effect on Cloudflare and Fastly and such like.
No, because Vary header is about *request* headers that cause the response to vary, not response headers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855427396,https://api.github.com/repos/simonw/datasette/issues/1362,855427396,MDEyOklzc3VlQ29tbWVudDg1NTQyNzM5Ng==,9599,simonw,2021-06-06T16:46:17Z,2021-06-06T16:46:17Z,OWNER,"Mind you, since that plugin hook looks like this:
```python
@hookimpl
def extra_body_script():
return {
""module"": True,
""script"": ""console.log('Your JavaScript goes here...')""
}
```
Having it calculate a sha256 hash wouldn't be difficult.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855426750,https://api.github.com/repos/simonw/datasette/issues/1362,855426750,MDEyOklzc3VlQ29tbWVudDg1NTQyNjc1MA==,9599,simonw,2021-06-06T16:41:30Z,2021-06-06T16:44:49Z,OWNER,"This is from the current `base.html` template: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/base.html#L62-L66
Which includes this: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/_close_open_menus.html#L1-L16
The `body_scripts` bit is for this `extra_body_script` plugin hook, which is the thing that will be the most affected by implementing CSP: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855426516,https://api.github.com/repos/simonw/datasette/issues/1362,855426516,MDEyOklzc3VlQ29tbWVudDg1NTQyNjUxNg==,9599,simonw,2021-06-06T16:39:34Z,2021-06-06T16:39:34Z,OWNER,The reason Datasette uses small inline scripts right now is to avoid the overhead of an extra HTTP request for a JavaScript file - but these are both inherently cachable and perform much better under HTTP/2 so that's likely a false optimization.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855426314,https://api.github.com/repos/simonw/datasette/issues/1362,855426314,MDEyOklzc3VlQ29tbWVudDg1NTQyNjMxNA==,9599,simonw,2021-06-06T16:38:04Z,2021-06-06T16:38:04Z,OWNER,"The other option for inline scripts is the CSP nonce:
Content-Security-Policy: script-src 'nonce-2726c7f26c'
Then:
Since an attacker can't guess what the nonce will be it prevents them from injecting their own script block - this seems easier to make available to plugins than a full hashing mechanism, just make `{{ csp_nonce() }}` available to the template.
That template function can then be smart enough to set a flag which Datasette uses to decide if the `script-src 'nonce-2726c7f26c'` policy should be sent or not.
Presumably this would also require adding `Content-Security-Policy` to the `Vary` header though, which will have a nasty effect on Cloudflare and Fastly and such like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418899,https://api.github.com/repos/simonw/datasette/issues/1362,855418899,MDEyOklzc3VlQ29tbWVudDg1NTQxODg5OQ==,9599,simonw,2021-06-06T15:42:55Z,2021-06-06T15:42:55Z,OWNER,Another consideration: testing that this works correctly could require adoption of a real browser test environment (probably Cypress or maybe Playwright) to execute tests that will fail if CSP is violated.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418698,https://api.github.com/repos/simonw/datasette/issues/1362,855418698,MDEyOklzc3VlQ29tbWVudDg1NTQxODY5OA==,9599,simonw,2021-06-06T15:41:24Z,2021-06-06T15:41:24Z,OWNER,"I think the best way to answer these questions is with some prototyping - of both Datasette and some of the existing JavaScript plugins.
I can start with a `datasette-experimental-csp` plugin that sets the header (and could even run an optional report URI mechanism).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418401,https://api.github.com/repos/simonw/datasette/issues/1362,855418401,MDEyOklzc3VlQ29tbWVudDg1NTQxODQwMQ==,9599,simonw,2021-06-06T15:39:38Z,2021-06-06T15:39:38Z,OWNER,"The security benefit of forcing all JavaScript plugins to be written as CSP-friendly external scripts is very compelling though.
Other plugin-heavy ecosystems such as WordPress have suffered greatly from insecurely written plugins - could this be a huge security win for the Datasette ecosystem generally?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418065,https://api.github.com/repos/simonw/datasette/issues/1362,855418065,MDEyOklzc3VlQ29tbWVudDg1NTQxODA2NQ==,9599,simonw,2021-06-06T15:37:11Z,2021-06-06T15:37:11Z,OWNER,"The easiest way to apply CSP is to remove all inline `
{% endblock %}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729017519,Add template block prior to extra URL loaders,
https://github.com/simonw/datasette/pull/1049#issuecomment-718340847,https://api.github.com/repos/simonw/datasette/issues/1049,718340847,MDEyOklzc3VlQ29tbWVudDcxODM0MDg0Nw==,9599,simonw,2020-10-29T03:45:47Z,2020-10-29T03:48:26Z,OWNER,"[thebe](https://thebelab.readthedocs.io/en/latest/examples/minimal_example.html) is the first time I've seen a library that requires you to set up some global JavaScript configuration before loading the script itself.
I'm hesitant to add an extra template block just to cover that one case since it's such a rare pattern. But it's important that `thebelab` can be used with Datasette.
Would this pattern work for you instead?
```html+jinja
{% block extra_head %}
{% endblock %}
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729017519,Add template block prior to extra URL loaders,
https://github.com/simonw/datasette/pull/1060#issuecomment-718243062,https://api.github.com/repos/simonw/datasette/issues/1060,718243062,MDEyOklzc3VlQ29tbWVudDcxODI0MzA2Mg==,22429695,codecov[bot],2020-10-28T22:23:33Z,2020-10-28T22:23:33Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=h1) Report
> Merging [#1060](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/abcf0222496d8148b2e585ffa0ff192270a04b06?el=desc) will **increase** coverage by `6.42%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #1060 +/- ##
==========================================
+ Coverage 84.71% 91.13% +6.42%
==========================================
Files 28 27 -1
Lines 3957 3677 -280
==========================================
- Hits 3352 3351 -1
+ Misses 605 326 -279
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1060/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `73.63% <100.00%> (+0.13%)` | :arrow_up: |
| [datasette/version.py](https://codecov.io/gh/simonw/datasette/pull/1060/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZlcnNpb24ucHk=) | `100.00% <100.00%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=footer). Last update [abcf022...4725d46](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731827081,New explicit versioning mechanism,
https://github.com/simonw/sqlite-utils/issues/191#issuecomment-718170295,https://api.github.com/repos/simonw/sqlite-utils/issues/191,718170295,MDEyOklzc3VlQ29tbWVudDcxODE3MDI5NQ==,9599,simonw,2020-10-28T19:50:16Z,2020-10-28T19:50:16Z,OWNER,"I think I made a mistake when I designed the initial decorator. I should have had it work like this:
```python
@db.register_function()
def reverse_string(s):
return """".join(reversed(list(s)))
```
As this leaves open the option to add new parameters in the future.
To avoid breaking backwards compatibility I'll use the hack that detects the argument this time, but in the future I'll try to remember to always design decorators to be called like `@decorator()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731740458,Idea: @db.register_function(deterministic=True),
https://github.com/simonw/sqlite-utils/issues/191#issuecomment-718168730,https://api.github.com/repos/simonw/sqlite-utils/issues/191,718168730,MDEyOklzc3VlQ29tbWVudDcxODE2ODczMA==,9599,simonw,2020-10-28T19:47:20Z,2020-10-28T19:47:20Z,OWNER,"https://stackoverflow.com/a/3931903 looks useful:
```python
def trace(*args):
def _trace(func):
def wrapper(*args, **kwargs):
print enter_string
func(*args, **kwargs)
print exit_string
return wrapper
if len(args) == 1 and callable(args[0]):
# No arguments, this is the decorator
# Set default values for the arguments
enter_string = 'entering'
exit_string = 'exiting'
return _trace(args[0])
else:
# This is just returning the decorator
enter_string, exit_string = args
return _trace
```
Can improve that code with `functools.wraps`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731740458,Idea: @db.register_function(deterministic=True),
https://github.com/simonw/datasette/pull/1059#issuecomment-718078447,https://api.github.com/repos/simonw/datasette/issues/1059,718078447,MDEyOklzc3VlQ29tbWVudDcxODA3ODQ0Nw==,9599,simonw,2020-10-28T17:07:59Z,2020-10-28T17:08:14Z,OWNER,"> #### 0.6.0 (2020-10-27)
>
> - aiofiles is now tested on ppc64le.
> - Added name and mode properties to async file objects. [#82](https://github.com/Tinche/aiofiles/pull/82)
> - Fixed a DeprecationWarning internally. [#75](https://github.com/Tinche/aiofiles/pull/75)
> - Python 3.9 support and tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731445447,"Update aiofiles requirement from <0.6,>=0.4 to >=0.4,<0.7",
https://github.com/simonw/datasette/pull/1059#issuecomment-717938992,https://api.github.com/repos/simonw/datasette/issues/1059,717938992,MDEyOklzc3VlQ29tbWVudDcxNzkzODk5Mg==,22429695,codecov[bot],2020-10-28T13:38:46Z,2020-10-28T13:38:46Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=h1) Report
> Merging [#1059](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/7d9fedc176717a7e3d22a96575ae0aada5a65440?el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #1059 +/- ##
=======================================
Coverage 84.71% 84.71%
=======================================
Files 28 28
Lines 3957 3957
=======================================
Hits 3352 3352
Misses 605 605
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=footer). Last update [7d9fedc...e46327a](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731445447,"Update aiofiles requirement from <0.6,>=0.4 to >=0.4,<0.7",
https://github.com/simonw/datasette/issues/1057#issuecomment-717531272,https://api.github.com/repos/simonw/datasette/issues/1057,717531272,MDEyOklzc3VlQ29tbWVudDcxNzUzMTI3Mg==,9599,simonw,2020-10-27T20:51:09Z,2020-10-27T20:51:09Z,OWNER,"That works!
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730797787,--cors should enable /fixtures.db CORS access,
https://github.com/simonw/datasette/issues/1058#issuecomment-717527606,https://api.github.com/repos/simonw/datasette/issues/1058,717527606,MDEyOklzc3VlQ29tbWVudDcxNzUyNzYwNg==,9599,simonw,2020-10-27T20:44:06Z,2020-10-27T20:44:06Z,OWNER,Example: https://github.com/simonw/datasette/blob/5a1519796037105bc20bcf2f91a76e022926c204/datasette/views/database.py#L26-L32,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730802994,Database download should implement cascading permissions,
https://github.com/simonw/datasette/pull/1056#issuecomment-717489501,https://api.github.com/repos/simonw/datasette/issues/1056,717489501,MDEyOklzc3VlQ29tbWVudDcxNzQ4OTUwMQ==,22429695,codecov[bot],2020-10-27T19:39:41Z,2020-10-27T19:39:41Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=h1) Report
> Merging [#1056](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/26bb4a268127da2c38f4241abe45444b2a6f7874?el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #1056 +/- ##
=======================================
Coverage 84.70% 84.70%
=======================================
Files 28 28
Lines 3955 3955
=======================================
Hits 3350 3350
Misses 605 605
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=footer). Last update [26bb4a2...a7b2aab](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730752399,"Radical new colour scheme and base styles, courtesy of @natbat",
https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717361487,https://api.github.com/repos/simonw/sqlite-utils/issues/189,717361487,MDEyOklzc3VlQ29tbWVudDcxNzM2MTQ4Nw==,9599,simonw,2020-10-27T16:24:04Z,2020-10-27T16:24:04Z,OWNER,"This is great, thank you very much.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records,
https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717359145,https://api.github.com/repos/simonw/sqlite-utils/issues/189,717359145,MDEyOklzc3VlQ29tbWVudDcxNzM1OTE0NQ==,35681,adamwolf,2020-10-27T16:20:32Z,2020-10-27T16:20:32Z,CONTRIBUTOR,"No problem. I added a test. Let me know if it looks sufficient or if you want me to to tweak something!
If you don't mind, would you tag this PR as ""hacktoberfest-accepted""? If you do mind, no problem and I'm sorry for asking :) My kiddos like the shirts.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records,
https://github.com/simonw/datasette/issues/1054#issuecomment-717051707,https://api.github.com/repos/simonw/datasette/issues/1054,717051707,MDEyOklzc3VlQ29tbWVudDcxNzA1MTcwNw==,9599,simonw,2020-10-27T07:41:21Z,2020-10-27T07:41:21Z,OWNER,Essentially it's this problem: https://github.com/python-versioneer/python-versioneer/issues/140,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730199464,Switch from versioneer to concrete version in setup.py,
https://github.com/simonw/datasette/issues/1054#issuecomment-717050585,https://api.github.com/repos/simonw/datasette/issues/1054,717050585,MDEyOklzc3VlQ29tbWVudDcxNzA1MDU4NQ==,9599,simonw,2020-10-27T07:38:50Z,2020-10-27T07:38:50Z,OWNER,"Maybe imitate how Django does this, e.g. https://github.com/django/django/commit/6b9b2af7352908d40ca4d31bdb1b80c013cab29a","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730199464,Switch from versioneer to concrete version in setup.py,
https://github.com/simonw/sqlite-utils/pull/189#issuecomment-716756103,https://api.github.com/repos/simonw/sqlite-utils/issues/189,716756103,MDEyOklzc3VlQ29tbWVudDcxNjc1NjEwMw==,9599,simonw,2020-10-26T18:56:19Z,2020-10-26T18:56:19Z,OWNER,"This is a great fix, thanks! If you add a unit test somewhere in here I'll merge the PR: https://github.com/simonw/sqlite-utils/blob/main/tests/test_m2m.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records,
https://github.com/simonw/datasette/issues/1051#issuecomment-716681602,https://api.github.com/repos/simonw/datasette/issues/1051,716681602,MDEyOklzc3VlQ29tbWVudDcxNjY4MTYwMg==,9599,simonw,2020-10-26T16:51:58Z,2020-10-26T16:51:58Z,OWNER,"I still need to improve the current binary display on the query page though, where it outputs a Python `b'...'` literal.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1051#issuecomment-716681167,https://api.github.com/repos/simonw/datasette/issues/1051,716681167,MDEyOklzc3VlQ29tbWVudDcxNjY4MTE2Nw==,9599,simonw,2020-10-26T16:51:15Z,2020-10-26T16:51:15Z,OWNER,"Crazy idea: generate a signed URL containing a base64 of the gzip of the binary content (to try and reduce size).
No: this will blow through URL limits in various hosting providers and possibly even browsers. It could be made to work a little bit more reliably with some extra JavaScript that turns it into a download on the browser-side, but that would be hideously complicated.
Also the signed bit doesn't prevent people from generating SQL queries that generate nasty binary blobs for download.
I'm beginning to think that restricting this feature to just table view, not query view, is a better idea. Query view can still get at the binary using JSON and base64.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/976#issuecomment-716305890,https://api.github.com/repos/simonw/datasette/issues/976,716305890,MDEyOklzc3VlQ29tbWVudDcxNjMwNTg5MA==,9599,simonw,2020-10-26T05:07:10Z,2020-10-26T05:07:10Z,OWNER,"I used the new `datasette.urls` methods to handle escaping table names.
https://github.com/simonw/datasette/blob/f5dbe61a4568c0915ec6be820095c2960cf0857c/datasette/utils/__init__.py#L996-L1008","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708289783,Idea: -o could open to a more convenient location,
https://github.com/simonw/datasette/pull/1043#issuecomment-716237524,https://api.github.com/repos/simonw/datasette/issues/1043,716237524,MDEyOklzc3VlQ29tbWVudDcxNjIzNzUyNA==,45380,bollwyvl,2020-10-26T00:14:57Z,2020-10-26T00:14:57Z,CONTRIBUTOR,"Sorry, I was out of the loop this weekend. The missing sdists were in some the `datasette-*` plugins... i'll capture my findings more concretely in one spot when i have a chance...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727915394,Include LICENSE in sdist,