issue_comments
9,947 rows sorted by id descending
This data as json, CSV (advanced)
reactions 15 ✖
- {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 5,066
- {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 99
- {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 18
- {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0} 13
- {"total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 9
- {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0} 5
- {"total_count": 1, "+1": 0, "-1": 0, "laugh": 1, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 4
- {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 2, "rocket": 0, "eyes": 0} 3
- {"total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 3
- {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1} 2
- {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 2, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 2
- {"total_count": 15, "+1": 7, "-1": 0, "laugh": 1, "hooray": 1, "confused": 0, "heart": 5, "rocket": 1, "eyes": 0} 1
- {"total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0} 1
- {"total_count": 3, "+1": 0, "-1": 0, "laugh": 0, "hooray": 3, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 1
- {"total_count": 3, "+1": 3, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 1
id ▲ | html_url | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
784347646 | https://github.com/simonw/datasette/issues/1241#issuecomment-784347646 | https://api.github.com/repos/simonw/datasette/issues/1241 | MDEyOklzc3VlQ29tbWVudDc4NDM0NzY0Ng== | Kabouik 7107523 | 2021-02-23T16:55:26Z | 2021-02-23T16:57:39Z | NONE | > I think it's possible that many users these days no longer assume they can paste a URL from the browser address bar (if they ever understood that at all) because to many apps are SPAs with broken URLs. Absolutely, that's why I thought my corner case with `iframe` preventing access to the datasette URL could actually be relevant in more general situations. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Share button for copying current URL 814595021 | |
784334931 | https://github.com/simonw/datasette/issues/1241#issuecomment-784334931 | https://api.github.com/repos/simonw/datasette/issues/1241 | MDEyOklzc3VlQ29tbWVudDc4NDMzNDkzMQ== | simonw 9599 | 2021-02-23T16:37:26Z | 2021-02-23T16:37:26Z | OWNER | A "Share link" button would only be needed on the table page and the arbitrary query page I think - and maybe on the row page, especially as that page starts to grow more features in the future. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Share button for copying current URL 814595021 | |
784333768 | https://github.com/simonw/datasette/issues/1241#issuecomment-784333768 | https://api.github.com/repos/simonw/datasette/issues/1241 | MDEyOklzc3VlQ29tbWVudDc4NDMzMzc2OA== | simonw 9599 | 2021-02-23T16:35:51Z | 2021-02-23T16:35:51Z | OWNER | This can definitely be done with a plugin. Adding to Datasette itself is an interesting idea. I think it's possible that many users these days no longer assume they can paste a URL from the browser address bar (if they ever understood that at all) because to many apps are SPAs with broken URLs. The shareable URLs are actually a key feature of Datasette - so maybe they should be highlighted in the default UI? I built a "copy to clipboard" feature for `datasette-copyable` and wrote up how that works here: https://til.simonwillison.net/javascript/copy-button | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Share button for copying current URL 814595021 | |
784312460 | https://github.com/simonw/datasette/issues/1240#issuecomment-784312460 | https://api.github.com/repos/simonw/datasette/issues/1240 | MDEyOklzc3VlQ29tbWVudDc4NDMxMjQ2MA== | Kabouik 7107523 | 2021-02-23T16:07:10Z | 2021-02-23T16:08:28Z | NONE | Likewise, while answering to another issue regarding the Vega plugin, I realized that there is no such way of linking rows after a custom query, I only get this "Link" column with individual URLs for the default SQL view: ![ss-2021-02-23_170559](https://user-images.githubusercontent.com/7107523/108871491-1e3fd500-75f1-11eb-8f76-5d5a82cc14d7.png) Or is it there and I am just missing the option in my custom queries? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Allow facetting on custom queries 814591962 | |
784157345 | https://github.com/simonw/datasette/issues/1218#issuecomment-784157345 | https://api.github.com/repos/simonw/datasette/issues/1218 | MDEyOklzc3VlQ29tbWVudDc4NDE1NzM0NQ== | soobrosa 1244799 | 2021-02-23T12:12:17Z | 2021-02-23T12:12:17Z | NONE | Topline this fixed the same problem for me. ``` brew install python@3.7 ln -s /usr/local/opt/python@3.7/bin/python3.7 /usr/local/opt/python/bin/python3.7 pip3 uninstall -y numpy pip3 uninstall -y setuptools pip3 install setuptools pip3 install numpy pip3 install datasette-publish-fly ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | /usr/local/opt/python3/bin/python3.6: bad interpreter: No such file or directory 803356942 | |
783794520 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-783794520 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc4Mzc5NDUyMA== | UtahDave 306240 | 2021-02-23T01:13:54Z | 2021-02-23T01:13:54Z | NONE | Also, @simonw I created a test based off the existing tests. I think it's working correctly | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | WIP: Add Gmail takeout mbox import 813880401 | |
783774084 | https://github.com/simonw/datasette/issues/1239#issuecomment-783774084 | https://api.github.com/repos/simonw/datasette/issues/1239 | MDEyOklzc3VlQ29tbWVudDc4Mzc3NDA4NA== | simonw 9599 | 2021-02-23T00:18:56Z | 2021-02-23T00:19:18Z | OWNER | Bug is here: https://github.com/simonw/datasette/blob/42caabf7e9e6e4d69ef6dd7de16f2cd96bc79d5b/datasette/filters.py#L149-L165 Those `json_each` lines should be: select {t}.rowid from {t}, json_each([{t}].[{c}]) j | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | JSON filter fails if column contains spaces 813978858 | |
783688547 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-783688547 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDc4MzY4ODU0Nw== | UtahDave 306240 | 2021-02-22T21:31:28Z | 2021-02-22T21:31:28Z | NONE | @Btibert3 I've opened a PR with my initial attempt at this. Would you be willing to give this a try? https://github.com/dogsheep/google-takeout-to-sqlite/pull/5 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature Request: Gmail 778380836 | |
783676548 | https://github.com/simonw/datasette/issues/1237#issuecomment-783676548 | https://api.github.com/repos/simonw/datasette/issues/1237 | MDEyOklzc3VlQ29tbWVudDc4MzY3NjU0OA== | simonw 9599 | 2021-02-22T21:10:19Z | 2021-02-22T21:10:25Z | OWNER | This is another change which is a little bit hard to figure out because I haven't solved #878 yet. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ?_pretty=1 option for pretty-printing JSON output 812704869 | |
783674659 | https://github.com/simonw/datasette/issues/1234#issuecomment-783674659 | https://api.github.com/repos/simonw/datasette/issues/1234 | MDEyOklzc3VlQ29tbWVudDc4MzY3NDY1OQ== | simonw 9599 | 2021-02-22T21:06:28Z | 2021-02-22T21:06:28Z | OWNER | I'm not going to work on this for a while, but if anyone has needs or ideas around that they can add them to this issue. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Runtime support for ATTACHing multiple databases 811505638 | |
783674038 | https://github.com/simonw/datasette/issues/1236#issuecomment-783674038 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MzY3NDAzOA== | simonw 9599 | 2021-02-22T21:05:21Z | 2021-02-22T21:05:21Z | OWNER | It's good on mobile - iOS at least. Going to close this open new issues if anyone reports bugs. | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0} | Ability to increase size of the SQL editor window 812228314 | |
783662968 | https://github.com/simonw/sqlite-utils/issues/220#issuecomment-783662968 | https://api.github.com/repos/simonw/sqlite-utils/issues/220 | MDEyOklzc3VlQ29tbWVudDc4MzY2Mjk2OA== | mhalle 649467 | 2021-02-22T20:44:51Z | 2021-02-22T20:44:51Z | NONE | Actually, coming back to this, I have a clearer use case for enabling fts generation for views: making it easier to bring in text from lookup tables and other joins. The datasette documentation describes populating an fts table like so: ``` INSERT INTO "items_fts" (rowid, name, description, category_name) SELECT items. rowid, items.name, items.description, categories.name FROM items JOIN categories ON items.category_id=categories.id; ``` Alternatively if you have fts support in sqlite_utils for views (which sqlite and fts5 support), you can do the same thing just by creating a view that captures the above joins as columns, then creating an fts table from that view. Such an fts table can be created using sqlite_utils, where one created with your method can't. The resulting fts table can then be used by a whole family of related tables and views in the manner you described earlier in this issue. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Better error message for *_fts methods against views 783778672 | |
783560017 | https://github.com/simonw/datasette/issues/1166#issuecomment-783560017 | https://api.github.com/repos/simonw/datasette/issues/1166 | MDEyOklzc3VlQ29tbWVudDc4MzU2MDAxNw== | thorn0 94334 | 2021-02-22T18:00:57Z | 2021-02-22T18:13:11Z | NONE | Hi! I don't think Prettier supports this syntax for globs: `datasette/static/*[!.min].js` Are you sure that works? Prettier uses https://github.com/mrmlnc/fast-glob, which in turn uses https://github.com/micromatch/micromatch, and the docs for these packages don't mention this syntax. As per the docs, square brackets should work as in regexes (`foo-[1-5].js`). Tested it. Apparently, it works as a negated character class in regexes (like `[^.min]`). I wonder where this syntax comes from. Micromatch doesn't support that: ```js micromatch(['static/table.js', 'static/n.js'], ['static/*[!.min].js']); // result: ["static/n.js"] -- brackets are treated like [!.min] in regexes, without negation ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Adopt Prettier for JavaScript code formatting 777140799 | |
783265830 | https://github.com/simonw/datasette/issues/782#issuecomment-783265830 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4MzI2NTgzMA== | frankieroberto 30665 | 2021-02-22T10:21:14Z | 2021-02-22T10:21:14Z | NONE | @simonw: > The problem there is that ?_size=x isn't actually doing the same thing as the SQL limit keyword. Interesting! Although I don't think it matters too much what the underlying implementation is - I more meant that `limit` is familiar to developers conceptually as "up to and including this number, if they exist", whereas "size" is potentially more ambiguous. However, it's probably no big deal either way. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782789598 | https://github.com/simonw/datasette/issues/782#issuecomment-782789598 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc4OTU5OA== | simonw 9599 | 2021-02-21T03:30:02Z | 2021-02-21T03:30:02Z | OWNER | Another benefit to default:object - I could include a key that shows a list of available extras. I could then use that to power an interactive API explorer. | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782765665 | https://github.com/simonw/datasette/issues/782#issuecomment-782765665 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc2NTY2NQ== | simonw 9599 | 2021-02-20T23:34:41Z | 2021-02-20T23:34:41Z | OWNER | OK, I'm back to the "top level object as the default" side of things now - it's pretty much unanimous at this point, and it's certainly true that it's not a decision you'll even regret. | {"total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782756398 | https://github.com/simonw/datasette/issues/782#issuecomment-782756398 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc1NjM5OA== | simonrjones 601316 | 2021-02-20T22:05:48Z | 2021-02-20T22:05:48Z | NONE | > I think it’s a good idea if the top level item of the response JSON is always an object, rather than an array, at least as the default. I agree it is more predictable if the top level item is an object with a rows or data object that contains an array of data, which then allows for other top-level meta data. I can see the argument for removing this and just using an array for convenience - but I think that's OK as an option (as you have now). Rather than have lots of top-level keys you could have a "meta" object to contain non-data stuff. You could use something like "links" for API endpoint URLs (or use a standard like HAL). Which would then leave the top level a bit cleaner - if that's what you what. Have you had much feedback from users who use the Datasette API a lot? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782748501 | https://github.com/simonw/datasette/issues/782#issuecomment-782748501 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0ODUwMQ== | simonw 9599 | 2021-02-20T20:58:18Z | 2021-02-20T20:58:18Z | OWNER | Yet another option: support a `?_path=x` option which returns a nested path from the result. So you could do this: `/github/commits.json?_path=rows` - to get back a top-level array pulled from the `"rows"` key. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782748093 | https://github.com/simonw/datasette/issues/782#issuecomment-782748093 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0ODA5Mw== | simonw 9599 | 2021-02-20T20:54:52Z | 2021-02-20T20:54:52Z | OWNER | > Have you given any thought as to whether to pretty print (format with spaces) the output or not? Can be useful for debugging/exploring in a browser or other basic tools which don’t parse the JSON. Could be default (can’t be much bigger with gzip?) or opt-in. Adding a `?_pretty=1` option that does that is a great idea, I'm filing a ticket for it: #1237 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782747878 | https://github.com/simonw/datasette/issues/782#issuecomment-782747878 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0Nzg3OA== | simonw 9599 | 2021-02-20T20:53:11Z | 2021-02-20T20:53:11Z | OWNER | ... though thinking about this further, I could re-implement the `select * from commits` (but only return a max of 10 results) feature using a nested `select * from (select * from commits) limit 10` query. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782747743 | https://github.com/simonw/datasette/issues/782#issuecomment-782747743 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0Nzc0Mw== | simonw 9599 | 2021-02-20T20:52:10Z | 2021-02-20T20:52:10Z | OWNER | > Minor suggestion: rename `size` query param to `limit`, to better reflect that it’s a maximum number of rows returned rather than a guarantee of getting that number, and also for consistency with the SQL keyword? The problem there is that `?_size=x` isn't actually doing the same thing as the SQL `limit` keyword. Consider this query: https://latest-with-plugins.datasette.io/github?sql=select+*+from+commits - `select * from commits` Datasette returns 1,000 results, and shows a "Custom SQL query returning more than 1,000 rows" message at the top. That's the `size` kicking in - I only fetch the first 1,000 results from the cursor to avoid exhausting resources. In the JSON version of that at https://latest-with-plugins.datasette.io/github.json?sql=select+*+from+commits there's a `"truncated": true` key to let you know what happened. I find myself using `?_size=2` against Datasette occasionally if I know the rows being returned are really big and I don't want to load 10+MB of HTML. This is only really a concern for arbitrary SQL queries though - for table pages such as https://latest-with-plugins.datasette.io/github/commits?_size=10 adding `?_size=10` actually puts a `limit 10` on the underlying SQL query. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782747164 | https://github.com/simonw/datasette/issues/782#issuecomment-782747164 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0NzE2NA== | simonw 9599 | 2021-02-20T20:47:16Z | 2021-02-20T20:47:16Z | OWNER | (I started a thread on Twitter about this: https://twitter.com/simonw/status/1363220355318358016) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782746755 | https://github.com/simonw/datasette/issues/782#issuecomment-782746755 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0Njc1NQ== | frankieroberto 30665 | 2021-02-20T20:44:05Z | 2021-02-20T20:44:05Z | NONE | Minor suggestion: rename `size` query param to `limit`, to better reflect that it’s a maximum number of rows returned rather than a guarantee of getting that number, and also for consistency with the SQL keyword? I like the idea of specifying a limit of 0 if you don’t want any rows data - and returning an empty array under the `rows` key seems fine. Have you given any thought as to whether to pretty print (format with spaces) the output or not? Can be useful for debugging/exploring in a browser or other basic tools which don’t parse the JSON. Could be default (can’t be much bigger with gzip?) or opt-in. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782746633 | https://github.com/simonw/datasette/issues/782#issuecomment-782746633 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0NjYzMw== | simonw 9599 | 2021-02-20T20:43:07Z | 2021-02-20T20:43:07Z | OWNER | Another option: `.json` always returns an object with a list of keys that gets increased through adding `?_extra=` parameters. `.jsona` always returns a JSON array of objects I had something similar to this in Datasette a few years ago - a `.jsono` extension, which still redirects to the `shape=array` version. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782745199 | https://github.com/simonw/datasette/issues/782#issuecomment-782745199 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0NTE5OQ== | frankieroberto 30665 | 2021-02-20T20:32:03Z | 2021-02-20T20:32:03Z | NONE | I think it’s a good idea if the top level item of the response JSON is always an object, rather than an array, at least as the default. Mainly because it allows you to add extra keys in a backwards-compatible way. Also just seems more expected somehow. The API design guidance for the UK government also recommends this: https://www.gov.uk/guidance/gds-api-technical-and-data-standards#use-json I also strongly dislike having versioned APIs (eg with a `/v1/` path prefix, as it invariably means that old versions stop working at some point, even though the bit of the API you’re using might not have changed at all. | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1} | Redesign default .json format 627794879 | |
782742233 | https://github.com/simonw/datasette/issues/782#issuecomment-782742233 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0MjIzMw== | simonw 9599 | 2021-02-20T20:09:16Z | 2021-02-20T20:09:16Z | OWNER | I just noticed that https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total&_size=0&_trace=1 executes 35 SQL queries at the moment! A great reminder that a big improvement from this change will be a reduction in queries through not calculating things like suggested facets unless they are explicitly requested. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782741719 | https://github.com/simonw/datasette/issues/782#issuecomment-782741719 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0MTcxOQ== | simonw 9599 | 2021-02-20T20:05:04Z | 2021-02-20T20:05:04Z | OWNER | > The only advantage of headers is that you don’t need to do .rows, but that’s actually good as a data validation step anyway—if .rows is missing assume there’s an error and do your error handling path instead of parsing the rest. This is something I've not thought very hard about. If there's an error, I need to return a top-level object, not a top-level array, so I can provide details of the error. But this means that client code will have to handle this difference - it will have to know that the returned data can be array-shaped if nothing went wrong, and object-shaped if there's an error. The HTTP status code helps here - calling client code can know that a 200 status code means there will be an array, but an error status code means an object. If developers really hate that the shape could be different, they can always use `?_extra=next` to ensure that the top level item is an object whether or not an error occurred. So I think this is OK. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782741107 | https://github.com/simonw/datasette/issues/782#issuecomment-782741107 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0MTEwNw== | simonw 9599 | 2021-02-20T20:00:22Z | 2021-02-20T20:00:22Z | OWNER | A really exciting opportunity this opens up is for parallel execution - the `facets()` and `suggested_facets()` and `total()` async functions could be called in parallel, which could speed things up if I'm confident the SQLite thread pool can execute on multiple CPU cores (it should be able to because the Python `sqlite3` module releases the GIL while it's executing C code). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782740985 | https://github.com/simonw/datasette/issues/782#issuecomment-782740985 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0MDk4NQ== | simonw 9599 | 2021-02-20T19:59:21Z | 2021-02-20T19:59:21Z | OWNER | This design should be influenced by how it's implemented. One implementation that could be nice is that each of the keys that can be requested - `next_url`, `total` etc - maps to an `async def` function which can do the work. So that expensive `count(*)` will only be executed by the `async def total` function if it is requested. This raises more questions: Both `next` and `next_url` work off the same underlying data, so if they are both requested can we re-use the work that `next` does somehow? Maybe by letting these functions depend on each other (so `next_url()` knows to first call `next()`, but only if it hasn't been called already. I think I need to flesh out the full default collection of `?_extra=` parameters in order to design how they will work under the hood. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782740604 | https://github.com/simonw/datasette/issues/782#issuecomment-782740604 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0MDYwNA== | simonw 9599 | 2021-02-20T19:56:21Z | 2021-02-20T19:56:33Z | OWNER | I think I want to support `?_extra=next_url,total` in addition to `?_extra=next_url&_extra=total` - partly because it's less characters to type, and also because I know there exist URL handling library that don't know how to handle the same parameter multiple times (though they're going to break against Datasette already, so it's not a big deal). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782740488 | https://github.com/simonw/datasette/issues/782#issuecomment-782740488 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0MDQ4OA== | simonw 9599 | 2021-02-20T19:55:23Z | 2021-02-20T19:55:23Z | OWNER | Am I saying you won't get back a key in the response unless you explicitly request it, either by name or by specifying a bundle of extras (e.g. `all` or `paginated`)? The `"truncated": true` key that tells you that your arbitrary query returned more than X results but was truncated is pretty important, do I really want people to have to opt-in to that one? Also: having bundles like `all` or `paginated` live in the same namespace as single keys like `next_url` or `total` is a little odd - you can't tell by looking at them if they'll add a key called `all` or if they'll add a bunch of other stuff. Maybe bundles could be prefixed with something, perhaps an underscore? `?_extra=_all` and `?_extra=_paginated` for example. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782739926 | https://github.com/simonw/datasette/issues/782#issuecomment-782739926 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4MjczOTkyNg== | simonw 9599 | 2021-02-20T19:51:30Z | 2021-02-20T19:52:19Z | OWNER | Demos: - https://latest-with-plugins.datasette.io/github/commits.json-preview - https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=next_url - https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total - https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=next_url&_extra=total - https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total&_size=0 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782709425 | https://github.com/simonw/datasette/issues/782#issuecomment-782709425 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4MjcwOTQyNQ== | simonw 9599 | 2021-02-20T16:24:54Z | 2021-02-20T16:24:54Z | OWNER | Having shortcuts means I could support `?_extra=all` for returning ALL possible keys. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782709270 | https://github.com/simonw/datasette/issues/782#issuecomment-782709270 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4MjcwOTI3MA== | simonw 9599 | 2021-02-20T16:23:51Z | 2021-02-20T16:24:11Z | OWNER | Also how would you opt out of returning the `"rows"` key? I sometimes want to do this - if I want to get back just the count or just the facets for example. Some options: * `/fixtures/roadside_attractions.json?_extra=total&_extra=-rows` * `/fixtures/roadside_attractions.json?_extra=total&_skip=rows` * `/fixtures/roadside_attractions.json?_extra=total&_size=0` I quite like that last one with `?_size=0`. I think it would still return `"rows": []` but that's OK. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782708938 | https://github.com/simonw/datasette/issues/782#issuecomment-782708938 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4MjcwODkzOA== | simonw 9599 | 2021-02-20T16:22:14Z | 2021-02-20T16:22:14Z | OWNER | I'm leaning back in the direction of a flat JSON array of objects as the default - this: `/fixtures/roadside_attractions.json` Would return: ```json [ { "pk": 1, "name": "The Mystery Spot", "address": "465 Mystery Spot Road, Santa Cruz, CA 95065", "latitude": 37.0167, "longitude": -122.0024 }, { "pk": 2, "name": "Winchester Mystery House", "address": "525 South Winchester Boulevard, San Jose, CA 95128", "latitude": 37.3184, "longitude": -121.9511 }, { "pk": 3, "name": "Burlingame Museum of PEZ Memorabilia", "address": "214 California Drive, Burlingame, CA 94010", "latitude": 37.5793, "longitude": -122.3442 }, { "pk": 4, "name": "Bigfoot Discovery Museum", "address": "5497 Highway 9, Felton, CA 95018", "latitude": 37.0414, "longitude": -122.0725 } ] ``` To get the version that includes pagination information you would use the `?_extra=` parameter. For example: `/fixtures/roadside_attractions.json?_extra=total&_extra=next_url` ```json { "rows": [ { "pk": 1, "name": "The Mystery Spot", "address": "465 Mystery Spot Road, Santa Cruz, CA 95065", "latitude": 37.0167, "longitude": -122.0024 }, { "pk": 2, "name": "Winchester Mystery House", "address": "525 South Winchester Boulevard, San Jose, CA 95128", "latitude": 37.3184, "longitude": -121.9511 }, { "pk": 3, "name": "Burlingame Museum of PEZ Memorabilia", "address": "214 California Drive, Burlingame, CA 94010", "latitude": 37.5793, "longitude": -122.3442 }, { "pk": 4, "name": "Bigfoot Discovery Museum", "address": "5497 Highway 9, Felton, CA 95018", "latitude": 37.0414, "longitude": -122.0725 } ], "total": 4, "next_url": null } ``` ANY usage of the `?_extra=` parameter would turn t… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Redesign default .json format 627794879 | |
782464306 | https://github.com/simonw/datasette/issues/1236#issuecomment-782464306 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ2NDMwNg== | simonw 9599 | 2021-02-19T23:57:32Z | 2021-02-19T23:57:32Z | OWNER | Need to test this on mobile. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to increase size of the SQL editor window 812228314 | |
782464215 | https://github.com/simonw/datasette/issues/1236#issuecomment-782464215 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ2NDIxNQ== | simonw 9599 | 2021-02-19T23:57:13Z | 2021-02-19T23:57:13Z | OWNER | Now live on https://latest.datasette.io/_memory | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to increase size of the SQL editor window 812228314 | |
782462049 | https://github.com/simonw/datasette/issues/1236#issuecomment-782462049 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ2MjA0OQ== | simonw 9599 | 2021-02-19T23:51:12Z | 2021-02-19T23:51:12Z | OWNER | ![resize-demo](https://user-images.githubusercontent.com/9599/108573758-4914eb00-72ca-11eb-989c-e642eee68021.gif) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to increase size of the SQL editor window 812228314 | |
782459550 | https://github.com/simonw/datasette/issues/1236#issuecomment-782459550 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ1OTU1MA== | simonw 9599 | 2021-02-19T23:45:30Z | 2021-02-19T23:45:30Z | OWNER | Encoded using https://meyerweb.com/eric/tools/dencoder/ `%3Csvg%20aria-labelledby%3D%22cm-drag-to-resize%22%20role%3D%22img%22%20fill%3D%22%23ccc%22%20stroke%3D%22%23ccc%22%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2016%2016%22%20width%3D%2216%22%20height%3D%2216%22%3E%0A%20%20%3Ctitle%20id%3D%22cm-drag-to-resize%22%3EDrag%20to%20resize%3C%2Ftitle%3E%0A%20%20%3Cpath%20fill-rule%3D%22evenodd%22%20d%3D%22M1%202.75A.75.75%200%20011.75%202h12.5a.75.75%200%20110%201.5H1.75A.75.75%200%20011%202.75zm0%205A.75.75%200%20011.75%207h12.5a.75.75%200%20110%201.5H1.75A.75.75%200%20011%207.75zM1.75%2012a.75.75%200%20100%201.5h12.5a.75.75%200%20100-1.5H1.75z%22%3E%3C%2Fpath%3E%0A%3C%2Fsvg%3E` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to increase size of the SQL editor window 812228314 | |
782459405 | https://github.com/simonw/datasette/issues/1236#issuecomment-782459405 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ1OTQwNQ== | simonw 9599 | 2021-02-19T23:45:02Z | 2021-02-19T23:45:02Z | OWNER | I'm going to use a variant of the Datasette menu icon. Here it is in `#ccc` with an ARIA label: ```svg <svg aria-labelledby="cm-drag-to-resize" role="img" fill="#ccc" stroke="#ccc" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" width="16" height="16"> <title id="cm-drag-to-resize">Drag to resize</title> <path fill-rule="evenodd" d="M1 2.75A.75.75 0 011.75 2h12.5a.75.75 0 110 1.5H1.75A.75.75 0 011 2.75zm0 5A.75.75 0 011.75 7h12.5a.75.75 0 110 1.5H1.75A.75.75 0 011 7.75zM1.75 12a.75.75 0 100 1.5h12.5a.75.75 0 100-1.5H1.75z"></path> </svg> ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to increase size of the SQL editor window 812228314 | |
782458983 | https://github.com/simonw/datasette/issues/1236#issuecomment-782458983 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ1ODk4Mw== | simonw 9599 | 2021-02-19T23:43:34Z | 2021-02-19T23:43:34Z | OWNER | I only want it to resize up and down, not left to right - so I'm not keen on the default resize handle: <img width="722" alt="cm-resize_demo" src="https://user-images.githubusercontent.com/9599/108573363-364de680-72c9-11eb-8741-5112463ebfaa.png"> https://rawgit.com/Sphinxxxx/cm-resize/master/demo/index.html | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to increase size of the SQL editor window 812228314 | |
782458744 | https://github.com/simonw/datasette/issues/1236#issuecomment-782458744 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ1ODc0NA== | simonw 9599 | 2021-02-19T23:42:42Z | 2021-02-19T23:42:42Z | OWNER | I can use https://github.com/Sphinxxxx/cm-resize for this | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to increase size of the SQL editor window 812228314 | |
782430028 | https://github.com/simonw/datasette/issues/1212#issuecomment-782430028 | https://api.github.com/repos/simonw/datasette/issues/1212 | MDEyOklzc3VlQ29tbWVudDc4MjQzMDAyOA== | kbaikov 4488943 | 2021-02-19T22:54:13Z | 2021-02-19T22:54:13Z | CONTRIBUTOR | I will close this issue since it appears only in my particular setup. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tests are very slow. 797651831 | |
782246111 | https://github.com/simonw/datasette/issues/619#issuecomment-782246111 | https://api.github.com/repos/simonw/datasette/issues/619 | MDEyOklzc3VlQ29tbWVudDc4MjI0NjExMQ== | simonw 9599 | 2021-02-19T18:11:22Z | 2021-02-19T18:11:22Z | OWNER | Big usability improvement, see also #1236 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | "Invalid SQL" page should let you edit the SQL 520655983 | |
782053455 | https://github.com/simonw/datasette/pull/1229#issuecomment-782053455 | https://api.github.com/repos/simonw/datasette/issues/1229 | MDEyOklzc3VlQ29tbWVudDc4MjA1MzQ1NQ== | camallen 295329 | 2021-02-19T12:47:19Z | 2021-02-19T12:47:19Z | CONTRIBUTOR | I believe this pr and #1031 are related and fix the same issue. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ensure immutable databses when starting in configuration directory mode with 810507413 | |
781825726 | https://github.com/simonw/sqlite-utils/issues/236#issuecomment-781825726 | https://api.github.com/repos/simonw/sqlite-utils/issues/236 | MDEyOklzc3VlQ29tbWVudDc4MTgyNTcyNg== | simonw 9599 | 2021-02-19T05:10:41Z | 2021-02-19T05:10:41Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#attaching-additional-databases | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --attach command line option for attaching extra databases 811680502 | |
781825187 | https://github.com/simonw/sqlite-utils/issues/113#issuecomment-781825187 | https://api.github.com/repos/simonw/sqlite-utils/issues/113 | MDEyOklzc3VlQ29tbWVudDc4MTgyNTE4Nw== | simonw 9599 | 2021-02-19T05:09:12Z | 2021-02-19T05:09:12Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#attaching-additional-databases | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Syntactic sugar for ATTACH DATABASE 621286870 | |
781764561 | https://github.com/simonw/datasette/issues/283#issuecomment-781764561 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTc2NDU2MQ== | simonw 9599 | 2021-02-19T02:10:21Z | 2021-02-19T02:10:21Z | OWNER | This feature is now released! https://docs.datasette.io/en/stable/changelog.html#v0-55 | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0} | Support cross-database joins 325958506 | |
781736855 | https://github.com/simonw/datasette/issues/1235#issuecomment-781736855 | https://api.github.com/repos/simonw/datasette/issues/1235 | MDEyOklzc3VlQ29tbWVudDc4MTczNjg1NQ== | simonw 9599 | 2021-02-19T00:52:47Z | 2021-02-19T01:47:53Z | OWNER | I bumped the two lines in the `Dockerfile` to `FROM python:3.7.10-slim-stretch as build` and ran this to build it: docker build -f Dockerfile -t datasetteproject/datasette:python-3-7-10 . Then I ran it with: docker run -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette:python-3-7-10 datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db http://0.0.0.0:8001/-/versions confirmed that it was now running Python 3.7.10 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Upgrade Python version used by official Datasette Docker image 811589344 | |
781735887 | https://github.com/simonw/datasette/issues/1235#issuecomment-781735887 | https://api.github.com/repos/simonw/datasette/issues/1235 | MDEyOklzc3VlQ29tbWVudDc4MTczNTg4Nw== | simonw 9599 | 2021-02-19T00:50:21Z | 2021-02-19T00:50:55Z | OWNER | I'll bump to `3.7.10` for the moment - the fix for 3.8 isn't out until March 1st according to https://news.ycombinator.com/item?id=26186434 https://www.python.org/downloads/release/python-3710/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Upgrade Python version used by official Datasette Docker image 811589344 | |
781670827 | https://github.com/simonw/datasette/issues/283#issuecomment-781670827 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTY3MDgyNw== | simonw 9599 | 2021-02-18T22:16:46Z | 2021-02-18T22:16:46Z | OWNER | Demo is now live here: https://latest.datasette.io/_memory The documentation is at https://docs.datasette.io/en/latest/sql_queries.html#cross-database-queries - it links to this example query: https://latest.datasette.io/_memory?sql=select%0D%0A++%27fixtures%27+as+database%2C+*%0D%0Afrom%0D%0A++%5Bfixtures%5D.sqlite_master%0D%0Aunion%0D%0Aselect%0D%0A++%27extra_database%27+as+database%2C+*%0D%0Afrom%0D%0A++%5Bextra_database%5D.sqlite_master | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support cross-database joins 325958506 | |
781665560 | https://github.com/simonw/datasette/issues/283#issuecomment-781665560 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTY2NTU2MA== | simonw 9599 | 2021-02-18T22:06:14Z | 2021-02-18T22:06:14Z | OWNER | The implementation in #1232 is ready to land. It's the simplest-thing-that-could-possibly-work: you can run `datasette one.db two.db three.db --crossdb` and then use the `/_memory` page to run joins across tables from multiple databases. It only works on the first 10 databases that were passed to the command-line. This means that if you have a Datasette instance with hundreds of attached databases (see [Datasette Library](https://github.com/simonw/datasette/issues/417)) this won't be particularly useful for you. So... a better, future version of this feature would be one that lets you join across databases on command - maybe by hitting `/_memory?attach=db1&attach=db2` to get a special connection. Also worth noting: plugins that implement the [prepare_connection()](https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette) hook can attach additional databases - so if you need better, customized support for this one way to handle that would be with a custom plugin. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support cross-database joins 325958506 | |
781651283 | https://github.com/simonw/datasette/pull/1232#issuecomment-781651283 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTY1MTI4Mw== | simonw 9599 | 2021-02-18T21:37:55Z | 2021-02-18T21:37:55Z | OWNER | UI listing the attached tables: <img width="888" alt="_memory" src="https://user-images.githubusercontent.com/9599/108424809-8015ce80-71ee-11eb-8752-03c459f89320.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --crossdb option for joining across databases 811407131 | |
781641728 | https://github.com/simonw/datasette/pull/1232#issuecomment-781641728 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTY0MTcyOA== | simonw 9599 | 2021-02-18T21:19:34Z | 2021-02-18T21:19:34Z | OWNER | I tested the demo deployment like this: ``` datasette publish cloudrun fixtures.db extra_database.db \ -m fixtures.json \ --plugins-dir=plugins \ --branch=crossdb \ --extra-options="--setting template_debug 1 --crossdb" \ --install=pysqlite3-binary \ --service=datasette-latest-crossdb ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --crossdb option for joining across databases 811407131 | |
781637292 | https://github.com/simonw/datasette/pull/1232#issuecomment-781637292 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTYzNzI5Mg== | simonw 9599 | 2021-02-18T21:11:31Z | 2021-02-18T21:11:31Z | OWNER | Due to bug #1233 I'm going to publish the additional database as `extra_database.db` rather than `extra database.db` as it is used in the tests. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --crossdb option for joining across databases 811407131 | |
781636590 | https://github.com/simonw/datasette/issues/1233#issuecomment-781636590 | https://api.github.com/repos/simonw/datasette/issues/1233 | MDEyOklzc3VlQ29tbWVudDc4MTYzNjU5MA== | simonw 9599 | 2021-02-18T21:10:08Z | 2021-02-18T21:10:08Z | OWNER | I think the bug is here: https://github.com/simonw/datasette/blob/640ac7071b73111ba4423812cd683756e0e1936b/datasette/utils/__init__.py#L349-L373 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | "datasette publish cloudrun" cannot publish files with spaces in their name 811458446 | |
781634819 | https://github.com/simonw/datasette/pull/1232#issuecomment-781634819 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTYzNDgxOQ== | simonw 9599 | 2021-02-18T21:06:43Z | 2021-02-18T21:06:43Z | OWNER | I'll document this option on https://docs.datasette.io/en/stable/sql_queries.html | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --crossdb option for joining across databases 811407131 | |
781629841 | https://github.com/simonw/datasette/pull/1232#issuecomment-781629841 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTYyOTg0MQ== | simonw 9599 | 2021-02-18T20:57:23Z | 2021-02-18T20:57:23Z | OWNER | The new warning looks like this: <img width="514" alt="datasette_—_pipenv_shell_▸_Python_—_182×66" src="https://user-images.githubusercontent.com/9599/108420562-d41db480-71e8-11eb-87e3-1cddd65627c2.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --crossdb option for joining across databases 811407131 | |
781599929 | https://github.com/simonw/datasette/pull/1232#issuecomment-781599929 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTU5OTkyOQ== | codecov[bot] 22429695 | 2021-02-18T19:59:54Z | 2021-02-18T22:06:42Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=h1) Report > Merging [#1232](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=desc) (8876499) into [main](https://codecov.io/gh/simonw/datasette/commit/4df548e7668b5b21d64a267964951e67894f4712?el=desc) (4df548e) will **increase** coverage by `0.03%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1232/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1232 +/- ## ========================================== + Coverage 91.42% 91.46% +0.03% ========================================== Files 32 32 Lines 3955 3970 +15 ========================================== + Hits 3616 3631 +15 Misses 339 339 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1232/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.68% <100.00%> (+0.06%)` | :arrow_up: | | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1232/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `76.62% <100.00%> (+0.36%)` | :arrow_up: | | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1232/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.19% <100.00%> (+0.01%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=footer). Last update [4df5… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --crossdb option for joining across databases 811407131 | |
781598585 | https://github.com/simonw/datasette/pull/1232#issuecomment-781598585 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTU5ODU4NQ== | simonw 9599 | 2021-02-18T19:57:30Z | 2021-02-18T19:57:30Z | OWNER | It would also be neat if https://latest.datasette.io/ had multiple databases attached in order to demonstrate this feature. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --crossdb option for joining across databases 811407131 | |
781594632 | https://github.com/simonw/datasette/pull/1232#issuecomment-781594632 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTU5NDYzMg== | simonw 9599 | 2021-02-18T19:50:21Z | 2021-02-18T19:50:21Z | OWNER | It would be neat if the `/_memory` page showed a list of attached databases, to indicate that the `--crossdb` option is working and give people links to click to start running queries. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --crossdb option for joining across databases 811407131 | |
781593169 | https://github.com/simonw/datasette/issues/283#issuecomment-781593169 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTU5MzE2OQ== | simonw 9599 | 2021-02-18T19:47:34Z | 2021-02-18T19:47:34Z | OWNER | I have a working version now, moving development to a pull request. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support cross-database joins 325958506 | |
781591015 | https://github.com/simonw/datasette/issues/283#issuecomment-781591015 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTU5MTAxNQ== | simonw 9599 | 2021-02-18T19:44:02Z | 2021-02-18T19:44:02Z | OWNER | For the moment I'm going to hard-code a `SQLITE_LIMIT_ATTACHED=10` constant and only attach the first 10 databases to the `_memory` connection. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support cross-database joins 325958506 | |
781574786 | https://github.com/simonw/datasette/issues/283#issuecomment-781574786 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTU3NDc4Ng== | simonw 9599 | 2021-02-18T19:15:37Z | 2021-02-18T19:15:37Z | OWNER | `select * from pragma_database_list();` is useful - shows all attached databases for the current connection. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support cross-database joins 325958506 | |
781573676 | https://github.com/simonw/datasette/issues/283#issuecomment-781573676 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTU3MzY3Ng== | simonw 9599 | 2021-02-18T19:13:30Z | 2021-02-18T19:13:30Z | OWNER | It turns out SQLite defaults to a maximum of 10 attached databases. This can be increased using a compile-time constant, but even with that it cannot be more than 62: https://stackoverflow.com/questions/9845448/attach-limit-10 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support cross-database joins 325958506 | |
781560989 | https://github.com/simonw/datasette/issues/1231#issuecomment-781560989 | https://api.github.com/repos/simonw/datasette/issues/1231 | MDEyOklzc3VlQ29tbWVudDc4MTU2MDk4OQ== | simonw 9599 | 2021-02-18T18:50:53Z | 2021-02-18T18:50:53Z | OWNER | Ideally I'd figure out a way to replicate this error in a concurrent unit test. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Race condition errors in new refresh_schemas() mechanism 811367257 | |
781560865 | https://github.com/simonw/datasette/issues/1231#issuecomment-781560865 | https://api.github.com/repos/simonw/datasette/issues/1231 | MDEyOklzc3VlQ29tbWVudDc4MTU2MDg2NQ== | simonw 9599 | 2021-02-18T18:50:38Z | 2021-02-18T18:50:38Z | OWNER | I started trying to use locks to resolve this but I've not figured out the right way to do that yet - here's my first experiment: ```diff diff --git a/datasette/app.py b/datasette/app.py index 9e15a16..1681c9d 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -217,6 +217,7 @@ class Datasette: self.inspect_data = inspect_data self.immutables = set(immutables or []) self.databases = collections.OrderedDict() + self._refresh_schemas_lock = threading.Lock() if memory or not self.files: self.add_database(Database(self, is_memory=True), name="_memory") # memory_name is a random string so that each Datasette instance gets its own @@ -324,6 +325,13 @@ class Datasette: self.client = DatasetteClient(self) async def refresh_schemas(self): + return + if self._refresh_schemas_lock.locked(): + return + with self._refresh_schemas_lock: + await self._refresh_schemas() + + async def _refresh_schemas(self): internal_db = self.databases["_internal"] if not self.internal_db_created: await init_internal_db(internal_db) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Race condition errors in new refresh_schemas() mechanism 811367257 | |
781546512 | https://github.com/simonw/datasette/issues/1226#issuecomment-781546512 | https://api.github.com/repos/simonw/datasette/issues/1226 | MDEyOklzc3VlQ29tbWVudDc4MTU0NjUxMg== | simonw 9599 | 2021-02-18T18:26:19Z | 2021-02-18T18:26:19Z | OWNER | This broke CI: https://github.com/simonw/datasette/runs/1929355965?check_suite_focus=true | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --port option should validate port is between 0 and 65535 808843401 | |
781530157 | https://github.com/simonw/datasette/issues/1226#issuecomment-781530157 | https://api.github.com/repos/simonw/datasette/issues/1226 | MDEyOklzc3VlQ29tbWVudDc4MTUzMDE1Nw== | simonw 9599 | 2021-02-18T18:00:15Z | 2021-02-18T18:00:15Z | OWNER | I can use `click.IntRange(min=None, max=None)` for this. https://click.palletsprojects.com/en/7.x/options/#ranges - inclusive on both edges. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --port option should validate port is between 0 and 65535 808843401 | |
781451701 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-781451701 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDc4MTQ1MTcwMQ== | Btibert3 203343 | 2021-02-18T16:06:21Z | 2021-02-18T16:06:21Z | NONE | Awesome! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature Request: Gmail 778380836 | |
781330466 | https://github.com/simonw/datasette/issues/1230#issuecomment-781330466 | https://api.github.com/repos/simonw/datasette/issues/1230 | MDEyOklzc3VlQ29tbWVudDc4MTMzMDQ2Ng== | Kabouik 7107523 | 2021-02-18T13:06:22Z | 2021-02-18T15:22:15Z | NONE | [Edit] Oh, I just saw the "Load all" button under the cluster map as well as the [setting to alter the max number or results](https://docs.datasette.io/en/stable/settings.html#max-returned-rows). So I guess this issue only is about the Vega charts. <details> Note that datasette-cluster-map also seems to be limited to 998 displayed points: ![ss-2021-02-18_140548](https://user-images.githubusercontent.com/7107523/108361225-15fb2a80-71ea-11eb-9a19-d885e8513f55.png) </details> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Vega charts are plotted only for rows on the visible page, cluster maps only for rows in the remaining pages 811054000 | |
781077127 | https://github.com/simonw/datasette/issues/283#issuecomment-781077127 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTA3NzEyNw== | simonw 9599 | 2021-02-18T05:56:30Z | 2021-02-18T05:57:34Z | OWNER | I'm going to to try prototyping the `--crossdb` option that causes `/_memory` to connect to all databases as a starting point and see how well that works. | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0} | Support cross-database joins 325958506 | |
780991910 | https://github.com/simonw/datasette/issues/283#issuecomment-780991910 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MDk5MTkxMA== | rayvoelker 9308268 | 2021-02-18T02:13:56Z | 2021-02-18T02:13:56Z | NONE | I was going ask you about this issue when we talk during your office-hours schedule this Friday, but was there any support ever added for doing this cross-database joining? I have a use-case where could be pretty neat to do analysis using this tool on time-specific databases from snapshots https://ilsweb.cincinnatilibrary.org/collection-analysis/ ![image](https://user-images.githubusercontent.com/9308268/108294883-ba3a8e00-7164-11eb-9206-fcd5a8cdd883.png) and thanks again for such an amazing tool! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support cross-database joins 325958506 | |
780830464 | https://github.com/simonw/datasette/pull/1229#issuecomment-780830464 | https://api.github.com/repos/simonw/datasette/issues/1229 | MDEyOklzc3VlQ29tbWVudDc4MDgzMDQ2NA== | codecov[bot] 22429695 | 2021-02-17T20:24:30Z | 2021-03-29T00:17:21Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=h1) Report > Merging [#1229](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=desc) (a095248) into [main](https://codecov.io/gh/simonw/datasette/commit/8e18c7943181f228ce5ebcea48deb59ce50bee1f?el=desc) (8e18c79) will **not change** coverage. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1229/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1229 +/- ## ======================================= Coverage 91.51% 91.51% ======================================= Files 34 34 Lines 4255 4255 ======================================= Hits 3894 3894 Misses 361 361 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1229/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.85% <100.00%> (ø)` | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=footer). Last update [8e18c79...a095248](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ensure immutable databses when starting in configuration directory mode with 810507413 | |
780817596 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-780817596 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDc4MDgxNzU5Ng== | UtahDave 306240 | 2021-02-17T20:01:35Z | 2021-02-17T20:01:35Z | NONE | I've got this almost working. Just needs some polish | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature Request: Gmail 778380836 | |
779785638 | https://github.com/simonw/sqlite-utils/issues/227#issuecomment-779785638 | https://api.github.com/repos/simonw/sqlite-utils/issues/227 | MDEyOklzc3VlQ29tbWVudDc3OTc4NTYzOA== | camallen 295329 | 2021-02-16T11:48:03Z | 2021-02-16T11:48:03Z | NONE | Thank you @simonw | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Error reading csv files with large column data 807174161 | |
779467451 | https://github.com/simonw/datasette/issues/1226#issuecomment-779467451 | https://api.github.com/repos/simonw/datasette/issues/1226 | MDEyOklzc3VlQ29tbWVudDc3OTQ2NzQ1MQ== | simonw 9599 | 2021-02-15T22:02:46Z | 2021-02-15T22:02:46Z | OWNER | I'm OK with the current error message shown if you try to use too low a port: ``` datasette fivethirtyeight.db -p 800 INFO: Started server process [45511] INFO: Waiting for application startup. INFO: Application startup complete. ERROR: [Errno 13] error while attempting to bind on address ('127.0.0.1', 800): permission denied INFO: Waiting for application shutdown. INFO: Application shutdown complete. ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --port option should validate port is between 0 and 65535 808843401 | |
779467160 | https://github.com/simonw/datasette/issues/1226#issuecomment-779467160 | https://api.github.com/repos/simonw/datasette/issues/1226 | MDEyOklzc3VlQ29tbWVudDc3OTQ2NzE2MA== | simonw 9599 | 2021-02-15T22:01:53Z | 2021-02-15T22:01:53Z | OWNER | This check needs to happen in two places: https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L222-L227 https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L328-L333 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --port option should validate port is between 0 and 65535 808843401 | |
779448912 | https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779448912 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | MDEyOklzc3VlQ29tbWVudDc3OTQ0ODkxMg== | simonw 9599 | 2021-02-15T21:09:50Z | 2021-02-15T21:09:50Z | OWNER | I fiddled around and replaced that line with `batch_size = SQLITE_MAX_VARS // num_columns` - which evaluated to `10416` for this particular file. That got me this: 40.71s user 1.81s system 98% cpu 43.081 total 43s is definitely better than 56s, but it's still not as big as the ~26.5s to ~3.5s improvement described by @simonwiles at the top of this issue. I wonder what I'm missing here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | SQLITE_MAX_VARS maybe hard-coded too low 688670158 | |
779446652 | https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779446652 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | MDEyOklzc3VlQ29tbWVudDc3OTQ0NjY1Mg== | simonw 9599 | 2021-02-15T21:04:19Z | 2021-02-15T21:04:19Z | OWNER | ... but it looks like `batch_size` is hard-coded to 100, rather than `None` - which means it's not being calculated using that value: https://github.com/simonw/sqlite-utils/blob/1f49f32814a942fa076cfe5f504d1621188097ed/sqlite_utils/db.py#L704 And https://github.com/simonw/sqlite-utils/blob/1f49f32814a942fa076cfe5f504d1621188097ed/sqlite_utils/db.py#L1877 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | SQLITE_MAX_VARS maybe hard-coded too low 688670158 | |
779445423 | https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779445423 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | MDEyOklzc3VlQ29tbWVudDc3OTQ0NTQyMw== | simonw 9599 | 2021-02-15T21:00:44Z | 2021-02-15T21:01:09Z | OWNER | I tried changing the hard-coded value from 999 to 156_250 and running `sqlite-utils insert` against a 500MB CSV file, with these results: ``` (sqlite-utils) sqlite-utils % time sqlite-utils insert slow-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers [###################################-] 99% 00:00:00sqlite-utils insert slow-ethos.db ethos ../ethos-datasette/ethos.csv 44.74s user 7.61s system 92% cpu 56.601 total # Increased the setting here (sqlite-utils) sqlite-utils % time sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers [###################################-] 99% 00:00:00sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv 39.40s user 5.15s system 96% cpu 46.320 total ``` Not as big a difference as I was expecting. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | SQLITE_MAX_VARS maybe hard-coded too low 688670158 | |
779417723 | https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779417723 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | MDEyOklzc3VlQ29tbWVudDc3OTQxNzcyMw== | simonw 9599 | 2021-02-15T19:44:02Z | 2021-02-15T19:47:00Z | OWNER | `%timeit find_limit(max=1_000_000)` took 378ms on my laptop `%timeit find_limit(max=500_000)` took 197ms `%timeit find_limit(max=200_000)` reported 53ms per loop `%timeit find_limit(max=100_000)` reported 26.8ms per loop. All of these are still slow enough that I'm not comfortable running this search for every time the library is imported. Allowing users to opt-in to this as a performance enhancement might be better. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | SQLITE_MAX_VARS maybe hard-coded too low 688670158 | |
779416619 | https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779416619 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | MDEyOklzc3VlQ29tbWVudDc3OTQxNjYxOQ== | simonw 9599 | 2021-02-15T19:40:57Z | 2021-02-15T21:27:55Z | OWNER | Tried this experiment (not proper binary search, it only searches downwards): ```python import sqlite3 db = sqlite3.connect(":memory:") def tryit(n): sql = "select 1 where 1 in ({})".format(", ".join("?" for i in range(n))) db.execute(sql, [0 for i in range(n)]) def find_limit(min=0, max=5_000_000): value = max while True: print('Trying', value) try: tryit(value) return value except: value = value // 2 ``` Running `find_limit()` with those default parameters takes about 1.47s on my laptop: ``` In [9]: %timeit find_limit() Trying 5000000 Trying 2500000... 1.47 s ± 28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ``` Interestingly the value it suggested was 156250 - suggesting that the macOS `sqlite3` binary with a 500,000 limit isn't the same as whatever my Python is using here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | SQLITE_MAX_VARS maybe hard-coded too low 688670158 | |
779409770 | https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779409770 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | MDEyOklzc3VlQ29tbWVudDc3OTQwOTc3MA== | simonw 9599 | 2021-02-15T19:23:11Z | 2021-02-15T19:23:11Z | OWNER | On my Mac right now I'm seeing a limit of 500,000: ``` % sqlite3 -cmd ".limits variable_number" variable_number 500000 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | SQLITE_MAX_VARS maybe hard-coded too low 688670158 | |
778854808 | https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778854808 | https://api.github.com/repos/simonw/sqlite-utils/issues/227 | MDEyOklzc3VlQ29tbWVudDc3ODg1NDgwOA== | simonw 9599 | 2021-02-14T22:46:54Z | 2021-02-14T22:46:54Z | OWNER | Fix is released in 3.5. | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Error reading csv files with large column data 807174161 | |
778851721 | https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778851721 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | MDEyOklzc3VlQ29tbWVudDc3ODg1MTcyMQ== | simonw 9599 | 2021-02-14T22:23:46Z | 2021-02-14T22:23:46Z | OWNER | I called this `--no-headers` for consistency with the existing output option: https://github.com/simonw/sqlite-utils/blob/427dace184c7da57f4a04df07b1e84cdae3261e8/sqlite_utils/cli.py#L61-L64 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --no-headers option for CSV and TSV 807437089 | |
778849394 | https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778849394 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | MDEyOklzc3VlQ29tbWVudDc3ODg0OTM5NA== | simonw 9599 | 2021-02-14T22:06:53Z | 2021-02-14T22:06:53Z | OWNER | For the moment I think just adding `--no-header` - which causes column names "unknown1,unknown2,..." to be used - should be enough. Users can import with that option, then use `sqlite-utils transform --rename` to rename them. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --no-headers option for CSV and TSV 807437089 | |
778844016 | https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778844016 | https://api.github.com/repos/simonw/sqlite-utils/issues/229 | MDEyOklzc3VlQ29tbWVudDc3ODg0NDAxNg== | simonw 9599 | 2021-02-14T21:22:45Z | 2021-02-14T21:22:45Z | OWNER | I'm going to use this pattern from https://stackoverflow.com/a/15063941 ```python import sys import csv maxInt = sys.maxsize while True: # decrease the maxInt value by factor 10 # as long as the OverflowError occurs. try: csv.field_size_limit(maxInt) break except OverflowError: maxInt = int(maxInt/10) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Hitting `_csv.Error: field larger than field limit (131072)` 807817197 | |
778843503 | https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843503 | https://api.github.com/repos/simonw/sqlite-utils/issues/229 | MDEyOklzc3VlQ29tbWVudDc3ODg0MzUwMw== | simonw 9599 | 2021-02-14T21:18:51Z | 2021-02-14T21:18:51Z | OWNER | I want to set this to the maximum allowed limit, which seems to be surprisingly hard! That StackOverflow thread is full of ideas for that, many of them involving `ctypes`. I'm a bit loathe to add a dependency on `ctypes` though - even though it's in the Python standard library I worry that it might not be available on some architectures. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Hitting `_csv.Error: field larger than field limit (131072)` 807817197 | |
778843362 | https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843362 | https://api.github.com/repos/simonw/sqlite-utils/issues/229 | MDEyOklzc3VlQ29tbWVudDc3ODg0MzM2Mg== | simonw 9599 | 2021-02-14T21:17:53Z | 2021-02-14T21:17:53Z | OWNER | Same issue as #227. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Hitting `_csv.Error: field larger than field limit (131072)` 807817197 | |
778843086 | https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778843086 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | MDEyOklzc3VlQ29tbWVudDc3ODg0MzA4Ng== | simonw 9599 | 2021-02-14T21:15:43Z | 2021-02-14T21:15:43Z | OWNER | I'm not convinced the `.has_header()` rules are useful for the kind of CSV files I work with: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L383 ```python def has_header(self, sample): # Creates a dictionary of types of data in each column. If any # column is of a single type (say, integers), *except* for the first # row, then the first row is presumed to be labels. If the type # can't be determined, it is assumed to be a string in which case # the length of the string is the determining factor: if all of the # rows except for the first are the same length, it's a header. # Finally, a 'vote' is taken at the end for each column, adding or # subtracting from the likelihood of the first row being a header. ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --no-headers option for CSV and TSV 807437089 | |
778842982 | https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778842982 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | MDEyOklzc3VlQ29tbWVudDc3ODg0Mjk4Mg== | simonw 9599 | 2021-02-14T21:15:11Z | 2021-02-14T21:15:11Z | OWNER | Implementation tip: I have code that reads the first row and uses it as headers here: https://github.com/simonw/sqlite-utils/blob/8f042ae1fd323995d966a94e8e6df85cc843b938/sqlite_utils/cli.py#L689-L691 So If I want to use `unknown1,unknown2...` I can do that by reading the first row, counting the number of columns, generating headers based on that range and then continuing to build that generator (maybe with `itertools.chain()` to replay the record we already read). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --no-headers option for CSV and TSV 807437089 | |
778841704 | https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778841704 | https://api.github.com/repos/simonw/sqlite-utils/issues/227 | MDEyOklzc3VlQ29tbWVudDc3ODg0MTcwNA== | simonw 9599 | 2021-02-14T21:05:20Z | 2021-02-14T21:05:20Z | OWNER | This has also been reported in #229. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Error reading csv files with large column data 807174161 | |
778841547 | https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778841547 | https://api.github.com/repos/simonw/sqlite-utils/issues/225 | MDEyOklzc3VlQ29tbWVudDc3ODg0MTU0Nw== | simonw 9599 | 2021-02-14T21:04:13Z | 2021-02-14T21:04:13Z | OWNER | I added a test and fixed this in #234 - thanks for the fix. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | fix for problem in Table.insert_all on search for columns per chunk of rows 797159961 | |
778841278 | https://github.com/simonw/sqlite-utils/issues/234#issuecomment-778841278 | https://api.github.com/repos/simonw/sqlite-utils/issues/234 | MDEyOklzc3VlQ29tbWVudDc3ODg0MTI3OA== | simonw 9599 | 2021-02-14T21:02:11Z | 2021-02-14T21:02:11Z | OWNER | I managed to replicate this in a test: ```python def test_insert_all_with_extra_columns_in_later_chunks(fresh_db): chunk = [ {"record": "Record 1"}, {"record": "Record 2"}, {"record": "Record 3"}, {"record": "Record 4", "extra": 1}, ] fresh_db["t"].insert_all(chunk, batch_size=2, alter=True) assert list(fresh_db["t"].rows) == [ {"record": "Record 1", "extra": None}, {"record": "Record 2", "extra": None}, {"record": "Record 3", "extra": None}, {"record": "Record 4", "extra": 1}, ] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .insert_all() fails if subsequent chunks contain additional columns 808046597 | |
778834504 | https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778834504 | https://api.github.com/repos/simonw/sqlite-utils/issues/225 | MDEyOklzc3VlQ29tbWVudDc3ODgzNDUwNA== | simonw 9599 | 2021-02-14T20:09:30Z | 2021-02-14T20:09:30Z | OWNER | Thanks for this. I'm going to try and get the test suite to run in Windows on GitHub Actions. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | fix for problem in Table.insert_all on search for columns per chunk of rows 797159961 | |
778829456 | https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778829456 | https://api.github.com/repos/simonw/sqlite-utils/issues/231 | MDEyOklzc3VlQ29tbWVudDc3ODgyOTQ1Ng== | simonw 9599 | 2021-02-14T19:37:52Z | 2021-02-14T19:37:52Z | OWNER | I'm going to add `limit` and `offset` to the following methods: - `rows_where()` - `search_sql()` - `search()` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | limit=X, offset=Y parameters for more Python methods 808028757 | |
778828758 | https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778828758 | https://api.github.com/repos/simonw/sqlite-utils/issues/231 | MDEyOklzc3VlQ29tbWVudDc3ODgyODc1OA== | simonw 9599 | 2021-02-14T19:33:14Z | 2021-02-14T19:33:14Z | OWNER | The `limit=` parameter is currently only available on the `.search()` method - it would make sense to add this to other methods as well. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | limit=X, offset=Y parameters for more Python methods 808028757 | |
778828495 | https://github.com/simonw/sqlite-utils/pull/224#issuecomment-778828495 | https://api.github.com/repos/simonw/sqlite-utils/issues/224 | MDEyOklzc3VlQ29tbWVudDc3ODgyODQ5NQ== | simonw 9599 | 2021-02-14T19:31:06Z | 2021-02-14T19:31:06Z | OWNER | I'm going to add a `offset=` parameter to support this case. Thanks for the suggestion! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add fts offset docs. 792297010 | |
778827570 | https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778827570 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | MDEyOklzc3VlQ29tbWVudDc3ODgyNzU3MA== | simonw 9599 | 2021-02-14T19:24:20Z | 2021-02-14T19:24:20Z | OWNER | Here's the implementation in Python: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L204-L225 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --sniff option for sniffing delimiters 808008305 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
created_at (date) >30 ✖