```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",986829194,xml.etree.ElementTree.Parse Error - mismatched tag,
https://github.com/simonw/datasette/issues/633#issuecomment-620841496,https://api.github.com/repos/simonw/datasette/issues/633,620841496,MDEyOklzc3VlQ29tbWVudDYyMDg0MTQ5Ng==,46165,nryberg,2020-04-28T20:37:50Z,2020-04-28T20:37:50Z,NONE,"Using the Heroku web interface, you can set the WEB_CONCURRENCY = 1
![image](https://user-images.githubusercontent.com/46165/80535319-352c8100-8966-11ea-9d4f-df2622ec8bff.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",522334771,"Publish to Heroku is broken: ""WARNING: You must pass the application as an import string to enable 'reload' or 'workers""",
https://github.com/simonw/datasette/issues/187#issuecomment-489353316,https://api.github.com/repos/simonw/datasette/issues/187,489353316,MDEyOklzc3VlQ29tbWVudDQ4OTM1MzMxNg==,46059,carsonyl,2019-05-04T18:36:36Z,2019-05-04T18:36:36Z,NONE,"Hi @simonw - I just hit this issue when trying out Datasette after your PyCon talk today. Datasette is pinned to Sanic 0.7.0, but it looks like 0.8.0 added the option to remove the uvloop dependency for Windows by having an environment variable `SANIC_NO_UVLOOP` at install time. Maybe that'll be sufficient before a port to Starlette?","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",309033998,Windows installation error,
https://github.com/simonw/datasette/issues/123#issuecomment-698174957,https://api.github.com/repos/simonw/datasette/issues/123,698174957,MDEyOklzc3VlQ29tbWVudDY5ODE3NDk1Nw==,45416,obra,2020-09-24T07:42:05Z,2020-09-24T07:42:05Z,NONE,"
Oh. Awesome.
On Thu, Sep 24, 2020 at 12:28:53AM -0700, Simon Willison wrote:
> @obra there's a plugin for that! https://github.com/simonw/
> datasette-upload-csvs
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.*
>
--
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats,
https://github.com/simonw/datasette/issues/123#issuecomment-698110186,https://api.github.com/repos/simonw/datasette/issues/123,698110186,MDEyOklzc3VlQ29tbWVudDY5ODExMDE4Ng==,45416,obra,2020-09-24T04:49:51Z,2020-09-24T04:49:51Z,NONE,"As a half-measure, I'd get value out of being able to upload a CSV and have datasette run csv-to-sqlite on it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats,
https://github.com/simonw/datasette/issues/619#issuecomment-697973420,https://api.github.com/repos/simonw/datasette/issues/619,697973420,MDEyOklzc3VlQ29tbWVudDY5Nzk3MzQyMA==,45416,obra,2020-09-23T21:07:58Z,2020-09-23T21:07:58Z,NONE,"I've just run into this after crafting a complex query and discovered that hitting back loses my query.
Even showing me the whole bad query would be a huge improvement over the current status quo.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/1886#issuecomment-1316289392,https://api.github.com/repos/simonw/datasette/issues/1886,1316289392,IC_kwDOBm6k_c5OdPtw,45195,rtanglao,2022-11-16T03:54:17Z,2022-11-16T03:58:56Z,NONE,"Happy Birthday Datasette!
Thanks Simon!!
I use datasette on everything most notably [my flickr metadata SQLite DB](https://www.dropbox.com/s/6j10e2vohp2j5kf/roland2019-2020.db?dl=0) to make art.
Datasette lite on my 2019 flickr metadata is super helpful too:
https://lite.datasette.io/?csv=https%3A%2F%2Fraw.githubusercontent.com%2Frtanglao%2Frt-flickr-sqlite-csv%2Fmain%2F2019-roland-flickr-metadata.csv
Even better datasette lite on all firefox support questions from 2021: https://lite.datasette.io/?url=https%3A%2F%2Fraw.githubusercontent.com%2Frtanglao%2Frt-kits-api3%2Fmain%2FYEARLY_CSV_FILES%2F2021-firefox-sumo-questions.db
Thanks again Simon! So great! What a gift to the world!!!!!!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here",
https://github.com/simonw/datasette/issues/409#issuecomment-472844001,https://api.github.com/repos/simonw/datasette/issues/409,472844001,MDEyOklzc3VlQ29tbWVudDQ3Mjg0NDAwMQ==,43100,Uninen,2019-03-14T13:04:20Z,2019-03-14T13:04:42Z,NONE,It seems this affects the Datasette Publish -site as well: https://github.com/simonw/datasette-publish-support/issues/3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",408376825,Zeit API v1 does not work for new users - need to migrate to v2,
https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537744000,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537744000,IC_kwDOCGYnMM5bqByA,42327,pquentin,2023-05-08T04:56:12Z,2023-05-08T04:56:12Z,NONE,"Hey @simonw, urllib3 maintainer here :wave:
Sorry for breaking your CI. I understand you may prefer to pin the Python version, but note that specifying just `python: ""3""` will get you the latest. We use that in urllib3: https://github.com/urllib3/urllib3/blob/main/.readthedocs.yml
I can open PRs to sqlite-utils / datasette if you're interested","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633234781,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633234781,MDEyOklzc3VlQ29tbWVudDYzMzIzNDc4MQ==,41439,dmd,2020-05-24T13:56:13Z,2020-05-24T13:56:13Z,NONE,"As that seems to be closed, can you give a hint on how to make this work?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/simonw/datasette/issues/1624#issuecomment-1261194164,https://api.github.com/repos/simonw/datasette/issues/1624,1261194164,IC_kwDOBm6k_c5LLEu0,38532,palfrey,2022-09-28T16:54:22Z,2022-09-28T16:54:22Z,NONE,https://github.com/simonw/datasette-cors seems to workaround this,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122427321,Index page `/` has no CORS headers,
https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803502424,https://api.github.com/repos/simonw/sqlite-utils/issues/249,803502424,MDEyOklzc3VlQ29tbWVudDgwMzUwMjQyNA==,36287,prabhur,2021-03-21T02:43:32Z,2021-03-21T02:43:32Z,NONE,"> Did you run `enable-fts` before you inserted the data?
>
> If so you'll need to run `populate-fts` after the insert to populate the FTS index.
>
> A better solution may be to add `--create-triggers` to the `enable-fts` command to add triggers that will automatically keep the index updated as you insert new records.
Wow. Wasn't expecting a response this quick, especially during a weekend. :-) Sincerely appreciate it.
I tried the `populate-fts` and that did the trick. My bad for not consulting the docs again. I think I forgot to add that step when I automated the workflow.
Thanks for the suggestion. I'll close this issue. Have a great weekend and many many thanks for creating these suite of tools around sqlite.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836963850,Full text search possibly broken?,
https://github.com/simonw/sqlite-utils/issues/89#issuecomment-593122605,https://api.github.com/repos/simonw/sqlite-utils/issues/89,593122605,MDEyOklzc3VlQ29tbWVudDU5MzEyMjYwNQ==,35075,chrishas35,2020-03-01T17:33:11Z,2020-03-01T17:33:11Z,NONE,"If you're happy with the proposed implementation, I have code & tests written that I'll get ready for a PR.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548,Ability to customize columns used by extracts= feature,
https://github.com/simonw/sqlite-utils/issues/46#issuecomment-592999503,https://api.github.com/repos/simonw/sqlite-utils/issues/46,592999503,MDEyOklzc3VlQ29tbWVudDU5Mjk5OTUwMw==,35075,chrishas35,2020-02-29T22:08:20Z,2020-02-29T22:08:20Z,NONE,"@simonw any thoughts on allow extracts to specify the lookup column name? If I'm understanding the documentation right, `.lookup()` allows you to define the ""value"" column (the documentation uses name), but when you use `extracts` keyword as part of `.insert()`, `.upsert()` etc. the lookup must be done against a column named ""value"". I have an existing lookup table that I've populated with columns ""id"" and ""name"" as opposed to ""id"" and ""value"", and seems I can't use `extracts=`, unless I'm missing something...
Initial thought on how to do this would be to allow the dictionary value to be a tuple of table name column pair... so:
```
table = db.table(""trees"", extracts={""species_id"": (""Species"", ""name""})
```
I haven't dug too much into the existing code yet, but does this make sense? Worth doing?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471780443,extracts= option for insert/update/etc,
https://github.com/simonw/datasette/pull/1574#issuecomment-1214765672,https://api.github.com/repos/simonw/datasette/issues/1574,1214765672,IC_kwDOBm6k_c5IZ9po,33631,fs111,2022-08-15T08:49:31Z,2022-08-15T08:49:31Z,NONE,closing as this is now the default,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084193403,introduce new option for datasette package to use a slim base image,
https://github.com/simonw/datasette/pull/1574#issuecomment-1084216224,https://api.github.com/repos/simonw/datasette/issues/1574,1084216224,IC_kwDOBm6k_c5An9Og,33631,fs111,2022-03-31T07:45:25Z,2022-03-31T07:45:25Z,NONE,"@simonw I like that you want to go ""slim by default"". Do you want another PR for that or should I just wait?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084193403,introduce new option for datasette package to use a slim base image,
https://github.com/simonw/datasette/pull/1574#issuecomment-1008279307,https://api.github.com/repos/simonw/datasette/issues/1574,1008279307,IC_kwDOBm6k_c48GR8L,33631,fs111,2022-01-09T11:26:06Z,2022-01-09T11:26:06Z,NONE,"@fgregg my thinking was backwards compatibility. I don't know what people do to their builds, I just wanted a smaller image for my use case.
@simonw any chance to take a look at this? If there is no interest, feel free to close the PR","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084193403,introduce new option for datasette package to use a slim base image,
https://github.com/dogsheep/swarm-to-sqlite/issues/12#issuecomment-941274088,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/12,941274088,IC_kwDODD6af844GrPo,33631,fs111,2021-10-12T18:31:57Z,2021-10-12T18:31:57Z,NONE,I am running into the same problem. Is there any workaround?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",951817328,403 when getting token,
https://github.com/simonw/datasette/issues/1528#issuecomment-988468238,https://api.github.com/repos/simonw/datasette/issues/1528,988468238,IC_kwDOBm6k_c466tQO,30934,20after4,2021-12-08T03:35:45Z,2021-12-08T03:35:45Z,NONE,"FWIW I implemented something similar with a bit of plugin code:
```python
@hookimpl
def canned_queries(datasette: Datasette, database: str) -> Mapping[str, str]:
# load ""canned queries"" from the filesystem under
# www/sql/db/query_name.sql
queries = {}
sqldir = Path(__file__).parent.parent / ""sql""
if database:
sqldir = sqldir / database
if not sqldir.is_dir():
return queries
for f in sqldir.glob('*.sql'):
try:
sql = f.read_text('utf8').strip()
if not len(sql):
log(f""Skipping empty canned query file: {f}"")
continue
queries[f.stem] = { ""sql"": sql }
except OSError as err:
log(err)
return queries
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1060631257,"Add new `""sql_file""` key to Canned Queries in metadata?",
https://github.com/simonw/datasette/issues/1304#issuecomment-988463455,https://api.github.com/repos/simonw/datasette/issues/1304,988463455,IC_kwDOBm6k_c466sFf,30934,20after4,2021-12-08T03:23:14Z,2021-12-08T03:23:14Z,NONE,I actually think it would be a useful thing to add support for in datasette. It wouldn't be difficult to unwind an array of params and add the placeholders automatically.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",863884805,"Document how to send multiple values for ""Named parameters"" ",
https://github.com/simonw/datasette/issues/1304#issuecomment-988461884,https://api.github.com/repos/simonw/datasette/issues/1304,988461884,IC_kwDOBm6k_c466rs8,30934,20after4,2021-12-08T03:20:26Z,2021-12-08T03:20:26Z,NONE,"The easiest or most straightforward thing to do is to use named parameters like:
```sql
select * where key IN (:p1, :p2, :p3)
```
And simply construct the list of placeholders dynamically based on the number of values.
Doing this is possible with datasette if you forgo ""canned queries"" and just use the raw query endpoint and pass the query sql, along with p1, p2 ... in the request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",863884805,"Document how to send multiple values for ""Named parameters"" ",
https://github.com/simonw/datasette/issues/1532#issuecomment-982745406,https://api.github.com/repos/simonw/datasette/issues/1532,982745406,IC_kwDOBm6k_c46k4E-,30934,20after4,2021-11-30T15:28:57Z,2021-11-30T15:28:57Z,NONE,"It's a really great API and the documentation is really great too. Honestly, in more than 20 years of professional experience, I haven't worked with any software API that was more of a joy to use. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936,Use datasette-table Web Component to guide the design of the JSON API for 1.0,
https://github.com/simonw/datasette/issues/1304#issuecomment-981980048,https://api.github.com/repos/simonw/datasette/issues/1304,981980048,IC_kwDOBm6k_c46h9OQ,30934,20after4,2021-11-29T20:13:53Z,2021-11-29T20:14:11Z,NONE,There isn't any way to do this with sqlite as far as I know. The only option is to insert the right number of ? placeholders into the sql template and then provide an array of values.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",863884805,"Document how to send multiple values for ""Named parameters"" ",
https://github.com/simonw/datasette/issues/1532#issuecomment-981966693,https://api.github.com/repos/simonw/datasette/issues/1532,981966693,IC_kwDOBm6k_c46h59l,30934,20after4,2021-11-29T19:56:52Z,2021-11-29T19:56:52Z,NONE,FWIW I've written some web components that consume the json api and I think it's a really nice way to work with datasette. I like the combination with datasette+sqlite as a back-end feeding data to a front-end that's entirely javascript + html.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936,Use datasette-table Web Component to guide the design of the JSON API for 1.0,
https://github.com/simonw/datasette/issues/878#issuecomment-951740637,https://api.github.com/repos/simonw/datasette/issues/878,951740637,IC_kwDOBm6k_c44umjd,30934,20after4,2021-10-26T09:12:15Z,2021-10-26T09:12:15Z,NONE,"This sounds really ambitious but also really awesome. I like the idea that basically any piece of a page could be selectively replaced.
It sort of sounds like a python asyncio version of https://github.com/observablehq/runtime","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/pull/1204#issuecomment-951731255,https://api.github.com/repos/simonw/datasette/issues/1204,951731255,IC_kwDOBm6k_c44ukQ3,30934,20after4,2021-10-26T09:01:28Z,2021-10-26T09:01:28Z,NONE,"> Writing the tests will be a bit tricky since we need to confirm that the `include_table_top(datasette, database, actor, table)` arguments were all passed correctly but the only thing we get back from the plugin is a list of templates. Maybe encode those values into the template names somehow?
Why not return a data structure instead of just a template name?
I've already done some custom hacking to modify datasette but the plugin mechanism you are building here would be much cleaner than what I've built. I'd be happy to help with testing this PR and fleshing it out further if you are still considering merging this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793002853,WIP: Plugin includes,
https://github.com/simonw/datasette/issues/782#issuecomment-783265830,https://api.github.com/repos/simonw/datasette/issues/782,783265830,MDEyOklzc3VlQ29tbWVudDc4MzI2NTgzMA==,30665,frankieroberto,2021-02-22T10:21:14Z,2021-02-22T10:21:14Z,NONE,"@simonw:
> The problem there is that ?_size=x isn't actually doing the same thing as the SQL limit keyword.
Interesting! Although I don't think it matters too much what the underlying implementation is - I more meant that `limit` is familiar to developers conceptually as ""up to and including this number, if they exist"", whereas ""size"" is potentially more ambiguous. However, it's probably no big deal either way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782746755,https://api.github.com/repos/simonw/datasette/issues/782,782746755,MDEyOklzc3VlQ29tbWVudDc4Mjc0Njc1NQ==,30665,frankieroberto,2021-02-20T20:44:05Z,2021-02-20T20:44:05Z,NONE,"Minor suggestion: rename `size` query param to `limit`, to better reflect that it’s a maximum number of rows returned rather than a guarantee of getting that number, and also for consistency with the SQL keyword?
I like the idea of specifying a limit of 0 if you don’t want any rows data - and returning an empty array under the `rows` key seems fine.
Have you given any thought as to whether to pretty print (format with spaces) the output or not? Can be useful for debugging/exploring in a browser or other basic tools which don’t parse the JSON. Could be default (can’t be much bigger with gzip?) or opt-in.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782745199,https://api.github.com/repos/simonw/datasette/issues/782,782745199,MDEyOklzc3VlQ29tbWVudDc4Mjc0NTE5OQ==,30665,frankieroberto,2021-02-20T20:32:03Z,2021-02-20T20:32:03Z,NONE,"I think it’s a good idea if the top level item of the response JSON is always an object, rather than an array, at least as the default. Mainly because it allows you to add extra keys in a backwards-compatible way. Also just seems more expected somehow.
The API design guidance for the UK government also recommends this: https://www.gov.uk/guidance/gds-api-technical-and-data-standards#use-json
I also strongly dislike having versioned APIs (eg with a `/v1/` path prefix, as it invariably means that old versions stop working at some point, even though the bit of the API you’re using might not have changed at all.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1845#issuecomment-1279924827,https://api.github.com/repos/simonw/datasette/issues/1845,1279924827,IC_kwDOBm6k_c5MShpb,30636,kindly,2022-10-16T08:54:53Z,2022-10-16T08:54:53Z,NONE,"> It was part of a larger idea I was exploring around ensuring Datasette could be used to start interacting with CSV/JSON data out-of-the-box, without needing to first convert that data into SQLite using separate tools.
This would be great. My organization deals with very nested JSON open data and I have been wanting to find a way to hook into datasette so that the analysts do not have to first convert to sqlite first.
This can kind of be done with datasette-lite.
From this random nested JSON API:
https://api.nobelprize.org/v1/prize.json
You can use the API of https://flatterer.herokuapp.com to return a multi table sqlite database:
https://lite.datasette.io/?url=https://flatterer.herokuapp.com/api/convert?output_format=sqlite%26file_url=https://api.nobelprize.org/v1/prize.json
This is great and fun, but it would be great if there was some plugin mechanism that you could feed a local datasette a nested JSON file directly, possibly hooking into other flattening tools for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1410305897,Reconsider the Datasette first-run experience,
https://github.com/simonw/datasette/issues/1220#issuecomment-778467759,https://api.github.com/repos/simonw/datasette/issues/1220,778467759,MDEyOklzc3VlQ29tbWVudDc3ODQ2Nzc1OQ==,30607,aborruso,2021-02-12T21:35:17Z,2021-02-12T21:35:17Z,NONE,Thank you,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806743116,Installing datasette via docker: Path 'fixtures.db' does not exist,
https://github.com/simonw/datasette/issues/1220#issuecomment-778008752,https://api.github.com/repos/simonw/datasette/issues/1220,778008752,MDEyOklzc3VlQ29tbWVudDc3ODAwODc1Mg==,30607,aborruso,2021-02-12T06:37:34Z,2021-02-12T06:37:34Z,NONE,"I have used my path, I'm running it from the folder in wich I have the db.
Do I must an absolute path?
Do I must create exactly that folder?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806743116,Installing datasette via docker: Path 'fixtures.db' does not exist,
https://github.com/simonw/sqlite-utils/issues/188#issuecomment-710778368,https://api.github.com/repos/simonw/sqlite-utils/issues/188,710778368,MDEyOklzc3VlQ29tbWVudDcxMDc3ODM2OA==,30607,aborruso,2020-10-17T08:52:58Z,2020-10-17T08:52:58Z,NONE,"I have done a stupid question.
If I run
```
sqlite-utils :memory: ""select spatialite_version()"" --load-extension=/usr/local/lib/mod_spatialite.so
```
I have `[{""spatialite_version()"": ""5.0.0""}]`
Thank you for this great tool","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723708310,About loading spatialite,
https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710768396,https://api.github.com/repos/simonw/sqlite-utils/issues/69,710768396,MDEyOklzc3VlQ29tbWVudDcxMDc2ODM5Ng==,30607,aborruso,2020-10-17T07:46:59Z,2020-10-17T07:46:59Z,NONE,Great @simonw thank you very much,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534507142,Feature request: enable extensions loading,
https://github.com/simonw/datasette/issues/744#issuecomment-635386935,https://api.github.com/repos/simonw/datasette/issues/744,635386935,MDEyOklzc3VlQ29tbWVudDYzNTM4NjkzNQ==,30607,aborruso,2020-05-28T14:32:53Z,2020-05-28T14:32:53Z,NONE,"Wow, I'm in some way very proud!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-634446887,https://api.github.com/repos/simonw/datasette/issues/744,634446887,MDEyOklzc3VlQ29tbWVudDYzNDQ0Njg4Nw==,30607,aborruso,2020-05-27T06:01:28Z,2020-05-27T06:01:28Z,NONE,"Dear @simonw thank you for your time, now IT WORKS!!!
I hope that this edit to datasette code is not for an exceptional case (my PC configuration) and that it will be useful to other users.
Thank you again!!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-634283355,https://api.github.com/repos/simonw/datasette/issues/744,634283355,MDEyOklzc3VlQ29tbWVudDYzNDI4MzM1NQ==,30607,aborruso,2020-05-26T21:15:34Z,2020-05-26T21:15:34Z,NONE,"> Oh no! It looks like `dirs_exist_ok` is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.
Thank you very much","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632305868,https://api.github.com/repos/simonw/datasette/issues/744,632305868,MDEyOklzc3VlQ29tbWVudDYzMjMwNTg2OA==,30607,aborruso,2020-05-21T19:43:23Z,2020-05-21T19:43:23Z,NONE,"@simonw now I have
```
Traceback (most recent call last):
File ""/home/aborruso/.local/bin/datasette"", line 8, in
sys.exit(cli())
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py"", line 103, in heroku
extra_metadata,
File ""/usr/lib/python3.7/contextlib.py"", line 112, in __enter__
return next(self.gen)
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py"", line 191, in temporary_heroku_directory
os.path.join(tmp.name, ""templates""),
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py"", line 605, in link_or_copy_directory
shutil.copytree(src, dst, copy_function=os.link, dirs_exist_ok=True)
TypeError: copytree() got an unexpected keyword argument 'dirs_exist_ok'
```
Do I must open a new issue?
Thank you","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632255088,https://api.github.com/repos/simonw/datasette/issues/744,632255088,MDEyOklzc3VlQ29tbWVudDYzMjI1NTA4OA==,30607,aborruso,2020-05-21T17:58:51Z,2020-05-21T17:58:51Z,NONE,"Thank you very much!!
I will try and I write you here","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632249565,https://api.github.com/repos/simonw/datasette/issues/744,632249565,MDEyOklzc3VlQ29tbWVudDYzMjI0OTU2NQ==,30607,aborruso,2020-05-21T17:47:40Z,2020-05-21T17:47:40Z,NONE,"@simonw can I test it know? What I must do to update it?
Thank you","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-625091976,https://api.github.com/repos/simonw/datasette/issues/744,625091976,MDEyOklzc3VlQ29tbWVudDYyNTA5MTk3Ng==,30607,aborruso,2020-05-07T07:51:25Z,2020-05-07T07:51:25Z,NONE,"I have installed `heroku plugins:install heroku-builds`, but I have the same error.
Then I have removed from `datasette\publish\heroku.py`
```python
# Check for heroku-builds plugin
plugins = [
line.split()[0] for line in check_output([""heroku"", ""plugins""]).splitlines()
]
if b""heroku-builds"" not in plugins:
click.echo(
""Publishing to Heroku requires the heroku-builds plugin to be installed.""
)
click.confirm(
""Install it? (this will run `heroku plugins:install heroku-builds`)"",
abort=True,
)
call([""heroku"", ""plugins:install"", ""heroku-builds""])
```
And now I have
```
Traceback (most recent call last):
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py"", line 210, in temporary_heroku_directory
yield
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py"", line 96, in heroku
list_output = check_output([""heroku"", ""apps:list"", ""--json""]).decode(
File ""c:\python37\lib\subprocess.py"", line 395, in check_output
**kwargs).stdout
File ""c:\python37\lib\subprocess.py"", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File ""c:\python37\lib\subprocess.py"", line 775, in __init__
restore_signals, start_new_session)
File ""c:\python37\lib\subprocess.py"", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The specified file could not be found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ""c:\python37\lib\runpy.py"", line 193, in _run_module_as_main
""__main__"", mod_spec)
File ""c:\python37\lib\runpy.py"", line 85, in _run_code
exec(code, run_globals)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\Scripts\datasette.exe\__main__.py"", line 9, in
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 782, in main
rv = self.invoke(ctx)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py"", line 120, in heroku
call([""heroku"", ""builds:create"", ""-a"", app_name, ""--include-vcs-ignore""])
File ""c:\python37\lib\contextlib.py"", line 130, in __exit__
self.gen.throw(type, value, traceback)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py"", line 213, in temporary_heroku_directory
tmp.cleanup()
File ""c:\python37\lib\tempfile.py"", line 809, in cleanup
_shutil.rmtree(self.name)
File ""c:\python37\lib\shutil.py"", line 513, in rmtree
return _rmtree_unsafe(path, onerror)
File ""c:\python37\lib\shutil.py"", line 401, in _rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File ""c:\python37\lib\shutil.py"", line 399, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 32] Unable to access file. The file is being used by another process: 'C:\\Users\\aborr\\AppData\\Local\\Temp\\tmpkcxy8i_q'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-625083715,https://api.github.com/repos/simonw/datasette/issues/744,625083715,MDEyOklzc3VlQ29tbWVudDYyNTA4MzcxNQ==,30607,aborruso,2020-05-07T07:34:18Z,2020-05-07T07:34:18Z,NONE,"In Windows I'm not very strong. I use debian (inside WSL).
However these are the possible steps:
- I have installed Python 3 for win (I have 3.7.3);
- I have installed heroku cli for win64 and logged in;
- I have installed datasette running `python -m pip install --upgrade --user datasette`.
It's a very basic Python env that I do not use. This time only to reach my goal: try to publish using custom template","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-625066073,https://api.github.com/repos/simonw/datasette/issues/744,625066073,MDEyOklzc3VlQ29tbWVudDYyNTA2NjA3Mw==,30607,aborruso,2020-05-07T06:53:09Z,2020-05-07T06:53:09Z,NONE,"@simonw another error starting from Windows.
I run
```
datasette publish heroku -n comunepa --template-dir template commissioniComunePalermo.db
```
And I have
```
Traceback (most recent call last):
File ""c:\python37\lib\runpy.py"", line 193, in _run_module_as_main
""__main__"", mod_spec)
File ""c:\python37\lib\runpy.py"", line 85, in _run_code
exec(code, run_globals)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\Scripts\datasette.exe\__main__.py"", line 9, in
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 782, in main
rv = self.invoke(ctx)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py"", line 53, in heroku
line.split()[0] for line in check_output([""heroku"", ""plugins""]).splitlines()
File ""c:\python37\lib\subprocess.py"", line 395, in check_output
**kwargs).stdout
File ""c:\python37\lib\subprocess.py"", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File ""c:\python37\lib\subprocess.py"", line 775, in __init__
restore_signals, start_new_session)
File ""c:\python37\lib\subprocess.py"", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The specified file could not be found
```
[files.zip](https://github.com/simonw/datasette/files/4591173/files.zip)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-625060561,https://api.github.com/repos/simonw/datasette/issues/744,625060561,MDEyOklzc3VlQ29tbWVudDYyNTA2MDU2MQ==,30607,aborruso,2020-05-07T06:38:24Z,2020-05-07T06:38:24Z,NONE,"Hi @simonw probably I could try to do it in Python for windows. I do not like to do these things in win enviroment.
Because probably WSL Linux env (in which I do a lot of great things) is not an environment that will be tested for datasette.
In win I shouldn't have any problems. Am I right?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-621030783,https://api.github.com/repos/simonw/datasette/issues/744,621030783,MDEyOklzc3VlQ29tbWVudDYyMTAzMDc4Mw==,30607,aborruso,2020-04-29T07:16:27Z,2020-04-29T07:16:27Z,NONE,"Hi @simonw it's debian as Windows Subsystem for Linux
```
PRETTY_NAME=""Pengwin""
NAME=""Pengwin""
VERSION_ID=""10""
VERSION=""10 (buster)""
ID=debian
ID_LIKE=debian
HOME_URL=""https://github.com/whitewaterfoundry/Pengwin""
SUPPORT_URL=""https://github.com/whitewaterfoundry/Pengwin""
BUG_REPORT_URL=""https://github.com/whitewaterfoundry/Pengwin""
VERSION_CODENAME=buster
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-621011554,https://api.github.com/repos/simonw/datasette/issues/744,621011554,MDEyOklzc3VlQ29tbWVudDYyMTAxMTU1NA==,30607,aborruso,2020-04-29T06:17:26Z,2020-04-29T06:17:26Z,NONE,"A stupid note: I have no `tmpcqv_1i5d` folder in in `/tmp`.
It seems to me that it does not create any `/tmp/tmpcqv_1i5d/templates` folder (or other name folder, inside /tmp)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-621008152,https://api.github.com/repos/simonw/datasette/issues/744,621008152,MDEyOklzc3VlQ29tbWVudDYyMTAwODE1Mg==,30607,aborruso,2020-04-29T06:05:02Z,2020-04-29T06:05:02Z,NONE,"Hi @simonw , I have installed it and I have the below errors.
> Is it possible that your /tmp directory is on a different volume from the template folder? That could cause a problem with the symlinks.
No, /tmp folder is in the same volume.
Thank you
```
Traceback (most recent call last):
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py"", line 607, in link_or_copy_directory
shutil.copytree(src, dst, copy_function=os.link)
File ""/usr/lib/python3.7/shutil.py"", line 365, in copytree
raise Error(errors)
shutil.Error: [('/var/youtubeComunePalermo/processing/./template/base.html', '/tmp/tmpcqv_1i5d/templates/base.html', ""[Errno 18] Invalid cross-device link: '/var/youtubeComunePalermo/processing/./template/base.html' -> '/tmp/tmpcqv_1i5d/templates/base.html'""), ('/var/youtubeComunePalermo/processing/./template/index.html', '/tmp/tmpcqv_1i5d/templates/index.html', ""[Errno 18] Invalid cross-device link: '/var/youtubeComunePalermo/processing/./template/index.html' -> '/tmp/tmpcqv_1i5d/templates/index.html'"")]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ""/home/aborruso/.local/bin/datasette"", line 8, in
sys.exit(cli())
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 610, in invoke return callback(*args, **kwargs)
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py"", line 103, in heroku
extra_metadata,
File ""/usr/lib/python3.7/contextlib.py"", line 112, in __enter__
return next(self.gen)
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py"", line 191, in temporary_heroku_directory
os.path.join(tmp.name, ""templates""),
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py"", line 609, in link_or_copy_directory
shutil.copytree(src, dst)
File ""/usr/lib/python3.7/shutil.py"", line 321, in copytree
os.makedirs(dst)
File ""/usr/lib/python3.7/os.py"", line 221, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/tmp/tmpcqv_1i5d/templates'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/735#issuecomment-620401443,https://api.github.com/repos/simonw/datasette/issues/735,620401443,MDEyOklzc3VlQ29tbWVudDYyMDQwMTQ0Mw==,30607,aborruso,2020-04-28T06:10:20Z,2020-04-28T06:10:20Z,NONE,"It works in heroku, than might be a bug with datasette-publish-now.
Thank you","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605806386,"Error when I click on ""View and edit SQL""",
https://github.com/simonw/datasette/issues/736#issuecomment-620401172,https://api.github.com/repos/simonw/datasette/issues/736,620401172,MDEyOklzc3VlQ29tbWVudDYyMDQwMTE3Mg==,30607,aborruso,2020-04-28T06:09:28Z,2020-04-28T06:09:28Z,NONE,"> Would you mind trying publishing your database using one of the other options - Heroku, Cloud Run or https://fly.io/ - and see if you have the same bug there?
It works in heroku, than might be a bug with datasette-publish-now.
Thank you","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",606720674,strange behavior using accented characters,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1003437288,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,1003437288,IC_kwDODFE5qs47zzzo,28565,maxhawkins,2021-12-31T19:06:20Z,2021-12-31T19:06:20Z,NONE,"> @maxhawkins how hard would it be to add an entry to the table that includes the HTML version of the email, if it exists? I just attempted your the PR branch on a very small mbox file, and it worked great. My use case is a research project and I need to access more than just the body plain text.
Shouldn't be hard. The easiest way is probably to remove the `if body.content_type == ""text/html""` clause from [utils.py:254](https://github.com/dogsheep/google-takeout-to-sqlite/pull/8/commits/8e6d487b697ce2e8ad885acf613a157bfba84c59#diff-25ad9dd1ced1b8bfc37fda8444819c803232c08891e4af3d4064aa205d8174eaR254) and just return content directly without parsing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-896378525,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,896378525,IC_kwDODFE5qs41baad,28565,maxhawkins,2021-08-10T23:28:45Z,2021-08-10T23:28:45Z,NONE,"I added parsing of text/html emails using BeautifulSoup.
Around half of the emails in my archive don't include a text/plain payload so adding html parsing makes a good chunk of them searchable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-894581223,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,894581223,IC_kwDODFE5qs41Ujnn,28565,maxhawkins,2021-08-07T00:57:48Z,2021-08-07T00:57:48Z,NONE,"Just added two more fixes:
* Added parsing for rfc 2047 encoded unicode headers
* Body is now stored as TEXT rather than a BLOB regardless of what order the messages are parsed in.
I was able to run this on my Takeout export and everything seems to work fine. @simonw let me know if this looks good to merge.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-888075098,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,888075098,IC_kwDODFE5qs407vNa,28565,maxhawkins,2021-07-28T07:18:56Z,2021-07-28T07:18:56Z,NONE,"> I'm not sure why but my most recent import, when displayed in Datasette, looks like this:
>
>
I did some investigation into this issue and made a fix [here](https://github.com/dogsheep/google-takeout-to-sqlite/pull/8/commits/8ee555c2889a38ff42b95664ee074b4a01a82f06). The problem was that some messages (like gchat logs) don't have a `Message-Id` and we need to use `X-GM-THRID` as the pkey instead.
@simonw While looking into this I found something unexpected about how sqlite_utils handles upserts if the pkey column is `None`. When the pkey is NULL I'd expect the function to either use rowid or throw an exception. Instead, it seems upsert_all creates a row where all columns are NULL instead of using the values provided as parameters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885094284,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,885094284,IC_kwDODFE5qs40wXeM,28565,maxhawkins,2021-07-22T17:41:32Z,2021-07-22T17:41:32Z,NONE,I added a follow-up commit that deals with emails that don't have a `Date` header: https://github.com/maxhawkins/google-takeout-to-sqlite/commit/4bc70103582c10802c85a523ef1e99a8a2154aa9,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885022230,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,885022230,IC_kwDODFE5qs40wF4W,28565,maxhawkins,2021-07-22T15:51:46Z,2021-07-22T15:51:46Z,NONE,One thing I noticed is this importer doesn't save attachments along with the body of the emails. It would be nice if those got stored as blobs in a separate attachments table so attachments can be included while fetching search results.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-884672647,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,884672647,IC_kwDODFE5qs40uwiH,28565,maxhawkins,2021-07-22T05:56:31Z,2021-07-22T14:03:08Z,NONE,"How does this commit look? https://github.com/maxhawkins/google-takeout-to-sqlite/commit/72802a83fee282eb5d02d388567731ba4301050d
It seems that Takeout's mbox format is pretty simple, so we can get away with just splitting the file on lines begining with `From `. My commit just splits the file every time a line starts with `From ` and uses `email.message_from_bytes` to parse each chunk.
I was able to load a 12GB takeout mbox without the program using more than a couple hundred MB of memory during the import process. It does make us lose the progress bar, but maybe I can add that back in a later commit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-849708617,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,849708617,MDEyOklzc3VlQ29tbWVudDg0OTcwODYxNw==,28565,maxhawkins,2021-05-27T15:01:42Z,2021-05-27T15:01:42Z,NONE,Any updates?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791089881,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,791089881,MDEyOklzc3VlQ29tbWVudDc5MTA4OTg4MQ==,28565,maxhawkins,2021-03-05T02:03:19Z,2021-03-05T02:03:19Z,NONE,"I just tried to run this on a small VPS instance with 2GB of memory and it crashed out of memory while processing a 12GB mbox from Takeout.
Is it possible to stream the emails to sqlite instead of loading it all into memory and upserting at once?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/simonw/datasette/issues/991#issuecomment-712855389,https://api.github.com/repos/simonw/datasette/issues/991,712855389,MDEyOklzc3VlQ29tbWVudDcxMjg1NTM4OQ==,24740,furilo,2020-10-20T13:36:41Z,2020-10-20T13:36:41Z,NONE,"Here is one quick sketch (done in Figma :P) for an idea: a possible filter to switch between showing all tables from all databases, or grouping tables by database.
(the switch is interactive)
All tables: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=1%3A2&viewport=536%2C348%2C0.5&scaling=min-zoom
Grouped: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=3%3A974&viewport=536%2C348%2C0.5&scaling=min-zoom
When only 1 database: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=1%3A162&viewport=536%2C348%2C0.5&scaling=min-zoom
Is this is useful, I can send some more suggestions/sketches.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/sqlite-utils/issues/54#issuecomment-524300388,https://api.github.com/repos/simonw/sqlite-utils/issues/54,524300388,MDEyOklzc3VlQ29tbWVudDUyNDMwMDM4OA==,20264,ftrain,2019-08-23T12:41:09Z,2019-08-23T12:41:09Z,NONE,Extremely cool and easy to understand. Thank you!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",480961330,"Ability to list views, and to access db[""view_name""].rows / rows_where / etc",
https://github.com/simonw/datasette/issues/153#issuecomment-348252037,https://api.github.com/repos/simonw/datasette/issues/153,348252037,MDEyOklzc3VlQ29tbWVudDM0ODI1MjAzNw==,20264,ftrain,2017-11-30T16:59:00Z,2017-11-30T16:59:00Z,NONE,"WOW!
--
Paul Ford // (646) 369-7128 // @ftrain
On Thu, Nov 30, 2017 at 11:47 AM, Simon Willison
wrote:
> Remaining work on this now lives in a milestone:
> https://github.com/simonw/datasette/milestone/6
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
>
> .
>
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",276842536,Ability to customize presentation of specific columns in HTML view,
https://github.com/simonw/sqlite-utils/issues/26#issuecomment-1141711418,https://api.github.com/repos/simonw/sqlite-utils/issues/26,1141711418,IC_kwDOCGYnMM5EDSI6,19304,nileshtrivedi,2022-05-31T06:21:15Z,2022-05-31T06:21:15Z,NONE,"I ran into this. My use case has a JSON file with array of `book` objects with a key called `reviews` which is also an array of objects. My JSON is human-edited and does not specify IDs for either books or reviews. Because sqlite-utils does not support inserting nested objects, I instead have to maintain two separate CSV files with `id` column in `books.csv` and `book_id` column in reviews.csv.
I think the right way to declare the relationship while inserting a JSON might be to describe the relationship:
`sqlite-utils insert data.db books mydata.json --hasmany reviews --hasone author --manytomany tags`
This is relying on the assumption that foreign keys can point to `rowid` primary key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455486286,Mechanism for turning nested JSON into foreign keys / many-to-many,
https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1035717429,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31,1035717429,IC_kwDOD079W849u8s1,18504,harperreed,2022-02-11T01:55:38Z,2022-02-11T01:55:38Z,NONE,I would love this merged! ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771511344,Update for Big Sur,
https://github.com/dogsheep/dogsheep-photos/issues/7#issuecomment-906015471,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7,906015471,IC_kwDOD079W842ALLv,18232,dkam,2021-08-26T02:01:01Z,2021-08-26T02:01:01Z,NONE,Perceptual hashes might be what you're after : http://phash.org,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602585497,Integrate image content hashing,
https://github.com/simonw/datasette/issues/1522#issuecomment-974607456,https://api.github.com/repos/simonw/datasette/issues/1522,974607456,IC_kwDOBm6k_c46F1Rg,17906,mrchrisadams,2021-11-20T07:10:11Z,2021-11-20T07:10:11Z,NONE,"As a a sanity check, would it be worth looking at trying to push the multi-process container on another provider of a knative / cloud run / tekton ? I have a somewhat similar use case for a future proejct, so i'm been very grateful to you sharing all the progress in this issue.
As I understand it, Scaleway also offer a very similar offering using what appear to be many similar components that might at least see if it's an issue with more than one knative based FaaS provider
https://www.scaleway.com/en/serverless-containers/
https://developers.scaleway.com/en/products/containers/api/#main-features
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy,
https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-602136481,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16,602136481,MDEyOklzc3VlQ29tbWVudDYwMjEzNjQ4MQ==,15092,jayvdb,2020-03-22T02:08:57Z,2020-03-22T02:08:57Z,NONE,"I'd love to be using your library as a better cached gh layer for a new library I have built, replacing large parts of the very ugly https://github.com/jayvdb/pypidb/blob/master/pypidb/_github.py , and then probably being able to rebuild the setuppy chunk as a feature here at a later stage.
I would also need tokenless and netrc support, but I would be happy to add those bits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546051181,Exception running first command: IndexError: list index out of range,
https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-571412923,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16,571412923,MDEyOklzc3VlQ29tbWVudDU3MTQxMjkyMw==,15092,jayvdb,2020-01-07T03:06:46Z,2020-01-07T03:06:46Z,NONE,"I re-tried after doing `auth`, and I get the same result.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546051181,Exception running first command: IndexError: list index out of range,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-1236200834,https://api.github.com/repos/simonw/sqlite-utils/issues/239,1236200834,IC_kwDOCGYnMM5Jru2C,14294,hubgit,2022-09-03T21:26:32Z,2022-09-03T21:26:32Z,NONE,"I was looking for something like this today, for extracting columns containing objects (and arrays of objects) into separate tables.
Would it make sense (especially for the fields containing arrays of objects) to create a one-to-many relationship, where each row of the newly created table would contain the id of the row that originally contained it?
If the extracted objects have a unique id and are repeated, it could even create a many-to-many relationship, with a third table for the joins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/474#issuecomment-1229449018,https://api.github.com/repos/simonw/sqlite-utils/issues/474,1229449018,IC_kwDOCGYnMM5JR-c6,14294,hubgit,2022-08-28T12:40:13Z,2022-08-28T12:40:13Z,NONE,"Creating the table before inserting is a useful workaround, thanks. It does require figuring out the `create table` syntax and listing all the fields manually, though, which loses some of the magic of sqlite-utils.
I was expecting to find an option like `--headers=foo,bar` (or `--header-row='foo\tbar'`, if that would be easier) - not necessarily that exact syntax, but something that would essentially be treated the same as having a header row in the file.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353074021,Add an option for specifying column names when inserting CSV data,
https://github.com/simonw/datasette/issues/176#issuecomment-617208503,https://api.github.com/repos/simonw/datasette/issues/176,617208503,MDEyOklzc3VlQ29tbWVudDYxNzIwODUwMw==,12976,nkirsch,2020-04-21T14:16:24Z,2020-04-21T14:16:24Z,NONE,"@eads I'm interested in helping, if there's still a need...","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",285168503,Add GraphQL endpoint,
https://github.com/simonw/sqlite-utils/issues/328#issuecomment-925300720,https://api.github.com/repos/simonw/sqlite-utils/issues/328,925300720,IC_kwDOCGYnMM43Jvfw,12752,gravis,2021-09-22T20:21:33Z,2021-09-22T20:21:33Z,NONE,"Wow, that was fast! Thank you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1004613267,Invalid JSON output when no rows,
https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1239516561,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10,1239516561,IC_kwDODLZ_YM5J4YWR,11887,ashanan,2022-09-07T15:07:38Z,2022-09-07T15:07:38Z,NONE,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1246826792,"When running `auth` command, don't overwrite an existing auth.json file",
https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074256603,https://api.github.com/repos/simonw/sqlite-utils/issues/417,1074256603,IC_kwDOCGYnMM5AB9rb,9954,blaine,2022-03-21T18:19:41Z,2022-03-21T18:19:41Z,NONE,"That makes sense; just a little hint that points folks towards doing the right thing might be helpful!
fwiw, the reason I was using jq in the first place was just a quick way to extract one attribute from an actual JSON array. When I initially imported it, I got a table with a bunch of embedded JSON values, rather than a native table, because each array entry had two attributes, one with the data I _actually_ wanted. Not sure how common a use-case this is, though (and easily fixed, aside from the jq weirdness!)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175744654,insert fails on JSONL with whitespace,
https://github.com/simonw/datasette/issues/176#issuecomment-359697938,https://api.github.com/repos/simonw/datasette/issues/176,359697938,MDEyOklzc3VlQ29tbWVudDM1OTY5NzkzOA==,7193,gijs,2018-01-23T07:17:56Z,2018-01-23T07:17:56Z,NONE,👍 I'd like this too! ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",285168503,Add GraphQL endpoint,
https://github.com/simonw/datasette/issues/1298#issuecomment-1125083348,https://api.github.com/repos/simonw/datasette/issues/1298,1125083348,IC_kwDOBm6k_c5DD2jU,7150,llimllib,2022-05-12T14:43:51Z,2022-05-12T14:43:51Z,NONE,"user report: I found this issue because the first time I tried to use datasette for real, I displayed a large table, and thought there was no horizontal scroll bar at all. I didn't even consider that I had to scroll all the way to the end of the page to find it.
Just chipping in to say that this confused me, and I didn't even find the scroll bar until after I saw this issue. I don't know what the right answer is, but IMO the UI should suggest to the user that there is a way to view the data that's hidden to the right.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",855476501,improve table horizontal scroll experience,
https://github.com/simonw/datasette/issues/2054#issuecomment-1499797384,https://api.github.com/repos/simonw/datasette/issues/2054,1499797384,IC_kwDOBm6k_c5ZZReI,6213,dsisnero,2023-04-07T00:46:50Z,2023-04-07T00:46:50Z,NONE,you should have a look at Roda written in ruby . ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-791053721,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32,791053721,MDEyOklzc3VlQ29tbWVudDc5MTA1MzcyMQ==,6213,dsisnero,2021-03-05T00:31:27Z,2021-03-05T00:31:27Z,NONE,I am getting the same thing for US West (N. California) us-west-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803333769,KeyError: 'Contents' on running upload,
https://github.com/simonw/datasette/issues/1818#issuecomment-1258738740,https://api.github.com/repos/simonw/datasette/issues/1818,1258738740,IC_kwDOBm6k_c5LBtQ0,5363,nelsonjchen,2022-09-26T22:52:45Z,2022-09-26T22:55:57Z,NONE,"thoughts on order of precedence to use:
* sqlite-utils count, if present. closest thing to a standard i guess.
* row(max_id) if like, the first and/or last x amount of rows ids are all contiguous. kind of a cheap/dumb/imperfect heuristic to see if the table is dump/not dump. if the check passes, still stick on `est.` after the display.
* count(*) if enabled in datasette ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1384549993,Setting to turn off table row counts entirely,
https://github.com/simonw/datasette/issues/1818#issuecomment-1257290709,https://api.github.com/repos/simonw/datasette/issues/1818,1257290709,IC_kwDOBm6k_c5K8LvV,5363,nelsonjchen,2022-09-25T22:17:06Z,2022-09-25T22:17:06Z,NONE,"I wonder if having an option for displaying the max row id might help too. Not accurate especially if something was deleted, but useful for DBs as a dump. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1384549993,Setting to turn off table row counts entirely,
https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1139484453,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1139484453,IC_kwDOCGYnMM5D6ycl,4068,frafra,2022-05-27T10:20:08Z,2022-05-27T10:20:08Z,NONE,I can confirm. This only happens with sqlite-utils. I am using gnome-terminal with bash.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor,
https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1139426398,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1139426398,IC_kwDOCGYnMM5D6kRe,4068,frafra,2022-05-27T09:04:05Z,2022-05-27T10:44:54Z,NONE,"This code works:
```python
import csv
import sqlite_utils
db = sqlite_utils.Database(""test.db"")
reader = csv.DictReader(open(""csv"", encoding=""utf-16-le"").read().split(""\r\n""), delimiter="";"")
db[""test""].insert_all(reader, pk=""Id"")
```
I used `iconv` to change the encoding; sqlite-utils can import the resulting file, even if it stops at 98 %:
```
sqlite-utils insert --csv test test.db clean
[------------------------------------] 0%
[###################################-] 98% 00:00:00
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input,
https://github.com/simonw/sqlite-utils/issues/438#issuecomment-1139392769,https://api.github.com/repos/simonw/sqlite-utils/issues/438,1139392769,IC_kwDOCGYnMM5D6cEB,4068,frafra,2022-05-27T08:21:53Z,2022-05-27T08:21:53Z,NONE,Argument were specified in the wrong order. `PATH TABLE FILE` can be misleading :),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250161887,illegal UTF-16 surrogate,
https://github.com/simonw/sqlite-utils/issues/438#issuecomment-1139379923,https://api.github.com/repos/simonw/sqlite-utils/issues/438,1139379923,IC_kwDOCGYnMM5D6Y7T,4068,frafra,2022-05-27T08:05:01Z,2022-05-27T08:05:01Z,NONE,"I tried to debug it using `pdb`, but it looks `sqlite-utils` catches the exception, so it is not quick to figure out where the failure is happening.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250161887,illegal UTF-16 surrogate,
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862574390,https://api.github.com/repos/simonw/sqlite-utils/issues/270,862574390,MDEyOklzc3VlQ29tbWVudDg2MjU3NDM5MA==,4068,frafra,2021-06-16T17:34:49Z,2021-06-16T17:34:49Z,NONE,"Sorry, I got confused because SQLite has a JSON column type, even if it is treated as TEXT, and I though automatic facets were available for JSON arrays stored as JSON only :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON,
https://github.com/simonw/datasette/issues/1375#issuecomment-860548546,https://api.github.com/repos/simonw/datasette/issues/1375,860548546,MDEyOklzc3VlQ29tbWVudDg2MDU0ODU0Ng==,4068,frafra,2021-06-14T09:41:59Z,2021-06-14T09:41:59Z,NONE,"> There is a feature for this at the moment, but it's a little bit hidden: you can use `?_json=col` to tell
> Datasette that you would like a specific column to be exported as nested JSON: https://docs.datasette.io/en/stable/json_api.html#special-json-arguments
Thanks :)
> I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning string instead of the expected dictionary and breaking your code.
If a developer is not sure if the JSON fields are valid, but then retrieves and parse them, it should handle errors too. Handling inconsistent data is necessary due to the nature of SQLite. A global or dataset option to render the data as they have been defined (JSON, boolean, etc.) when requesting JSON could allow the user to download a regular JSON from the browser without having to rely on APIs. I would guess someone could just make a custom template with an extra JSON-parsed download button otherwise :)","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919508498,JSON export dumps JSON fields as TEXT,
https://github.com/simonw/datasette/issues/1286#issuecomment-860047794,https://api.github.com/repos/simonw/datasette/issues/1286,860047794,MDEyOklzc3VlQ29tbWVudDg2MDA0Nzc5NA==,4068,frafra,2021-06-12T12:36:15Z,2021-06-12T12:36:15Z,NONE,"@mroswell That is a very nice solution. I wonder if custom classes, like `col-columnName-value` could be automatically added to cells when facets on such column are enabled, to allow custom styling without having to modify templates or add custom JavaScript code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849220154,Better default display of arrays of items,
https://github.com/simonw/sqlite-utils/issues/269#issuecomment-860031217,https://api.github.com/repos/simonw/sqlite-utils/issues/269,860031217,MDEyOklzc3VlQ29tbWVudDg2MDAzMTIxNw==,4068,frafra,2021-06-12T10:01:53Z,2021-06-12T10:01:53Z,NONE,"`sqlite-utils transform` does not allow setting the column type to boolean:
```
Error: Invalid value for '--type': 'bool' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919250621,bool type not supported,
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-860031071,https://api.github.com/repos/simonw/sqlite-utils/issues/270,860031071,MDEyOklzc3VlQ29tbWVudDg2MDAzMTA3MQ==,4068,frafra,2021-06-12T10:00:24Z,2021-06-12T10:00:24Z,NONE,"Sure, I am sorry if my message hasn't been clear enough. I am also a new user :)
At the beginning, I just call `sqlite-utils insert ""$db"" ""$table"" ""$jsonfile""` to create the database. sqlite-utils convert JSON values into `TEXT`, when it tries to determine the schema automatically. I then try to transform the table to set `JSON` as type:
```
sqlite-utils transform species.sqlite species --type criteria json
Usage: sqlite-utils transform [OPTIONS] PATH TABLE
Try 'sqlite-utils transform --help' for help.
Error: Invalid value for '--type': 'json' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON,
https://github.com/simonw/sqlite-utils/issues/269#issuecomment-859940977,https://api.github.com/repos/simonw/sqlite-utils/issues/269,859940977,MDEyOklzc3VlQ29tbWVudDg1OTk0MDk3Nw==,4068,frafra,2021-06-11T22:33:08Z,2021-06-11T22:33:08Z,NONE,"`true` and `false` json values are cast to integer, which is not optimal.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919250621,bool type not supported,
https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221521377,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11,1221521377,IC_kwDODLZ_YM5Izu_h,2467,fernand0,2022-08-21T10:51:37Z,2022-08-21T10:51:37Z,NONE,I didn't see there is a PR about this: https://github.com/dogsheep/pocket-to-sqlite/pull/7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1345452427,"-a option is used for ""--auth"" and for ""--all""",
https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-927312650,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54,927312650,IC_kwDODEm0Qs43RasK,2182,danp,2021-09-26T14:09:51Z,2021-09-26T14:09:51Z,NONE,"Similar trouble with ageinfo using 0.22. Here's what my ageinfo.js file looks like:
```
window.YTD.ageinfo.part0 = [
{
""ageMeta"" : { }
}
]
```
Commenting out the registration for ageinfo in archive.py gets my archive to import.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779088071,Archive import appears to be broken on recent exports,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622279374,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622279374,MDEyOklzc3VlQ29tbWVudDYyMjI3OTM3NA==,2029,garethr,2020-05-01T07:12:47Z,2020-05-01T07:12:47Z,NONE,"I also go it working with:
```yaml
run: echo ${{ secrets.github_token }} | github-to-sqlite auth
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,Fall back to authentication via ENV,
https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-605439685,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15,605439685,MDEyOklzc3VlQ29tbWVudDYwNTQzOTY4NQ==,2029,garethr,2020-03-28T12:17:01Z,2020-03-28T12:17:01Z,NONE,"That looks great, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",544571092,Assets table with downloads,
https://github.com/simonw/datasette/issues/526#issuecomment-810943882,https://api.github.com/repos/simonw/datasette/issues/526,810943882,MDEyOklzc3VlQ29tbWVudDgxMDk0Mzg4Mg==,701,jokull,2021-03-31T10:03:55Z,2021-03-31T10:03:55Z,NONE,"+1 on using nested queries to achieve this! Would be great as streaming CSV is an amazing feature.
Some UX/DX details:
I was expecting it to work to simply add `&_stream=on` to custom SQL queries because the docs say
> Any Datasette table, view or **custom SQL query** can be exported as CSV.
After a bit of testing back and forth I realized streaming only works for full tables.
Would love this feature because I'm using `pandas.read_csv` to paint graphs from custom queries and the graphs are cut off because of the 1000 row limit. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711083698,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711083698,MDEyOklzc3VlQ29tbWVudDcxMTA4MzY5OA==,572,jarib,2020-10-17T21:39:15Z,2020-10-17T21:39:15Z,NONE,Nice! Works perfectly. Thanks for the quick response and great tooling in general.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/twitter-to-sqlite/issues/53#issuecomment-748436453,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/53,748436453,MDEyOklzc3VlQ29tbWVudDc0ODQzNjQ1Mw==,27,anotherjesse,2020-12-19T07:47:01Z,2020-12-19T07:47:01Z,NONE,"I think this should probably be closed as won't fix.
Attempting to make a patch for this I realized that the since_id would limit to tweets posted since that since_id, not when it was favorited. So favoriting something in the older would be missed if you used `--since` with a cron script
Better to just use `--stop_after` set to a couple hundred","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771324837,--since support for favorites,