issues
2,892 rows sorted by closed_at
This data as json, CSV (advanced)
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at ▼ | author_association | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
550293770 | MDU6SXNzdWU1NTAyOTM3NzA= | 658 | How do I use the app.css as style sheet? | null92 49656826 | open | 0 | 2 | 2020-01-15T16:27:57Z | 2020-02-07T00:29:50Z | NONE | Simon, I'm trying to use the app.css (in static folder) as style sheet but the datasette on Heroku simply ignore it! I read everything about customization here and on readthedocs but still can't. Is this possible? Thanks! | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/658/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
559964149 | MDU6SXNzdWU1NTk5NjQxNDk= | 665 | Introduce a SQL statement parser in Python | simonw 9599 | open | 0 | 1 | 2020-02-04T20:36:05Z | 2020-02-04T20:36:48Z | OWNER | #254 and #653 are both examples of problems that could be solved using a real SQL parser in Python. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/665/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
564833696 | MDU6SXNzdWU1NjQ4MzM2OTY= | 670 | Prototoype for Datasette on PostgreSQL | simonw 9599 | open | 0 | 14 | 2020-02-13T17:17:55Z | 2023-07-17T02:23:32Z | OWNER | I thought this would never happen, but now that I'm deep in the weeds of running SQLite in production for Datasette Cloud I'm starting to reconsider my policy of only supporting SQLite. Some of the factors making me think PostgreSQL support could be worth the effort: - Serverless. I'm getting increasingly excited about writable-database use-cases for Datasette. If it could talk to PostgreSQL then users could easily deploy it on Heroku or other serverless providers that can talk to a managed RDS-style PostgreSQL. - Existing databases. Plenty of organizations have PostgreSQL databases. They can export to SQLite using [db-to-sqlite](https://github.com/simonw/db-to-sqlite) but that's a pretty big barrier to getting started - being able to run `datasette postgresql://connection-string` and start trying it out would be a massively better experience. - Data size. I keep running into use-cases where I want to run Datasette against many GBs of data. SQLite can do this but PostgreSQL is much more optimized for large data, especially given the existence of tools like Citus. - Marketing. Convincing people to trust their data to SQLite is potentially a big barrier to adoption. Even if I've convinced myself it's trustworthy I still have to convince everyone else. - It might not be that hard? If this required a ground-up rewrite it wouldn't be worth the effort, but I have a hunch that it may not be too hard - most of the SQL in Datasette should work on both databases since it's almost all portable SELECT statements. If Datasette did DML this would be a lot harder, but it doesn't. - Plugins! This feels like a natural surface for a plugin - at which point people could add MySQL support and suchlike in the future. The above reasons feel strong enough to justify a prototype. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/670/reactions", "total_count": 15, "+1": 11, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 4, "rocket": 0, "eyes": 0} | ||||||||
565064079 | MDExOlB1bGxSZXF1ZXN0Mzc1MTgwODMy | 672 | --dirs option for scanning directories for SQLite databases | simonw 9599 | open | 0 | 15 | 2020-02-14T02:25:52Z | 2020-03-27T01:03:53Z | OWNER | simonw/datasette/pulls/672 | Refs #417. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/672/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||||
567902704 | MDU6SXNzdWU1Njc5MDI3MDQ= | 675 | --cp option for datasette publish and datasette package for shipping additional files and directories | aviflax 141844 | open | 0 | 12 | 2020-02-19T22:55:56Z | 2020-12-28T18:49:21Z | NONE | I’m working on integrating Datasette into a documentation-oriented publishing workflow internally in my company, and in order to deploy the Docker image created by `datasette package` I need to add an additional file to the image — in my case, it’s a sort of a deployment directive. I’ve worked out a way to do this after the image has been created, but it’s convoluted and brittle. So it’d be excellent if there was an additional option for this command, something like, like, `--copy`. I’d envision it looking something like: ```shell $ datasette package --copy /the/source/path:/the/target/path data.db ``` I’d be happy to help design, specify, implement, and test this feature, if you’d be interested. Thanks for the fantastic tools! | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/675/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
573578548 | MDU6SXNzdWU1NzM1Nzg1NDg= | 89 | Ability to customize columns used by extracts= feature | simonw 9599 | open | 0 | 3 | 2020-03-01T16:54:48Z | 2020-10-16T19:17:50Z | OWNER | @simonw any thoughts on allow extracts to specify the lookup column name? If I'm understanding the documentation right, `.lookup()` allows you to define the "value" column (the documentation uses name), but when you use `extracts` keyword as part of `.insert()`, `.upsert()` etc. the lookup must be done against a column named "value". I have an existing lookup table that I've populated with columns "id" and "name" as opposed to "id" and "value", and seems I can't use `extracts=`, unless I'm missing something... Initial thought on how to do this would be to allow the dictionary value to be a tuple of table name column pair... so: ``` table = db.table("trees", extracts={"species_id": ("Species", "name"}) ``` I haven't dug too much into the existing code yet, but does this make sense? Worth doing? _Originally posted by @chrishas35 in https://github.com/simonw/sqlite-utils/issues/46#issuecomment-592999503_ | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/89/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
574021194 | MDU6SXNzdWU1NzQwMjExOTQ= | 691 | --reload sould reload server if code in --plugins-dir changes | simonw 9599 | open | 0 | 1 | 2020-03-02T14:42:21Z | 2020-06-14T02:35:17Z | OWNER | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/691/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | |||||||||
574035432 | MDU6SXNzdWU1NzQwMzU0MzI= | 692 | is_hidden_table context variable on table.html page | simonw 9599 | open | 0 | 1 | 2020-03-02T15:03:25Z | 2020-03-02T15:03:48Z | OWNER | It's useful to know if a table is hidden when rendering that page. `datasette-configure-fts` for example may want to disallow enabling search on hidden tables. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/692/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
581795570 | MDU6SXNzdWU1ODE3OTU1NzA= | 93 | Support more string values for types in .add_column() | simonw 9599 | open | 0 | 0 | 2020-03-15T19:32:49Z | 2020-09-24T20:36:46Z | OWNER | https://sqlite-utils.readthedocs.io/en/2.4.2/python-api.html#adding-columns says: > SQLite types you can specify are "TEXT", "INTEGER", "FLOAT" or "BLOB". As discovered in #92 this isn't the right list of values. I should expand this to match https://www.sqlite.org/datatype3.html | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/93/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
593006814 | MDU6SXNzdWU1OTMwMDY4MTQ= | 715 | Refactor duplicate cell display logic | simonw 9599 | open | 0 | 0 | 2020-04-03T00:58:11Z | 2020-04-03T00:58:11Z | OWNER | The logic for rendering cells in table view and in database (or canned query) view is currently very similar: https://github.com/simonw/datasette/blob/7656fd64d8b6a32ebc34d89c1b8711cc5ea240f7/datasette/views/base.py#L514-L539 Compared with: https://github.com/simonw/datasette/blob/7656fd64d8b6a32ebc34d89c1b8711cc5ea240f7/datasette/views/table.py#L104-L195 I'll be changing this a bit in #698 but I should still try to clean this up more further in the future. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/715/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
594237015 | MDU6SXNzdWU1OTQyMzcwMTU= | 718 | Plugin idea: datasette-redirects | simonw 9599 | open | 0 | 0 | 2020-04-05T03:41:38Z | 2020-04-05T03:41:38Z | OWNER | I just had to write a one-off custom plugin to redirect niche-musems.com to www.niche-museums.com (https://github.com/simonw/museums/issues/21) - it would be great if this kind of thing could be handled by a configurable plugin. https://github.com/simonw/museums/blob/6b1faf00c463b2228860d4d62d104b11935e01b1/plugins/redirect_www.py | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/718/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
599776345 | MDU6SXNzdWU1OTk3NzYzNDU= | 24 | Feature idea: github-to-sqlite everything ... | simonw 9599 | open | 0 | 0 | 2020-04-14T18:34:00Z | 2020-04-14T18:34:00Z | MEMBER | At the moment if you want to pull all your repos, issues, issues comments etc you have to do it with a sequence of separate commands. Consider adding a `everything` or `all` command which fetches everything that the tool knows how to fetch, and is designed to be run on a cron in a way that fetches just new stuff each time. | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/24/reactions", "total_count": 7, "+1": 7, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
602533300 | MDU6SXNzdWU2MDI1MzMzMDA= | 1 | Import photo metadata from Apple Photos into SQLite | simonw 9599 | open | 0 | Apple Photos online and securely browsable 5324096 | 8 | 2020-04-18T19:23:26Z | 2020-05-04T02:41:40Z | MEMBER | Faces, albums, locations, that kind of thing. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | |||||||
602533481 | MDU6SXNzdWU2MDI1MzM0ODE= | 3 | Import EXIF data into SQLite - lens used, ISO, aperture etc | simonw 9599 | open | 0 | Apple Photos online and securely browsable 5324096 | 2 | 2020-04-18T19:24:31Z | 2021-10-05T12:38:24Z | MEMBER | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
602585497 | MDU6SXNzdWU2MDI1ODU0OTc= | 7 | Integrate image content hashing | simonw 9599 | open | 0 | 2 | 2020-04-19T00:36:58Z | 2021-08-26T02:01:01Z | MEMBER | To spot duplicate images (where the file content differs such that the sha256 is no longer a match) it would be useful to calculate and store perceptual hashes of some sort. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7/reactions", "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0} | ||||||||
602619330 | MDU6SXNzdWU2MDI2MTkzMzA= | 45 | Use raise_for_status() everywhere | simonw 9599 | open | 0 | 1 | 2020-04-19T04:38:28Z | 2020-04-19T04:39:22Z | MEMBER | I keep seeing errors which I think are caused by authentication or rate limit problems but which appear to be unexpected JSON responses - presumably because they are actually an error message. Recent example: https://github.com/simonw/jsk-fellows-on-twitter/runs/598892575 Using `response.raise_for_status()` everywhere will make these errors less confusing. | twitter-to-sqlite 206156866 | issue | {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
606033104 | MDU6SXNzdWU2MDYwMzMxMDQ= | 12 | If less than 500MB, show size in MB not GB | simonw 9599 | open | 0 | 1 | 2020-04-24T04:35:01Z | 2020-04-24T04:35:25Z | MEMBER | Just saw this: ``` Uploading 0.05 GB ``` | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
607223136 | MDU6SXNzdWU2MDcyMjMxMzY= | 741 | Replace "datasette publish --extra-options" with "--setting" | simonw 9599 | open | 0 | Datasette 1.0 3268330 | 9 | 2020-04-27T04:29:04Z | 2022-05-12T19:21:16Z | OWNER | See https://github.com/simonw/datasette-publish-now/issues/9#issuecomment-618155764 - the `--extra-options` mechanism is in practice just used to set `--config` options in data that you publish, but that means you end up with pretty messy looking commands: datasette publish my.db --extra-options="--config default_page_size:50 --config sql_time_limit_ms:3500" A neater design would be to support `--config` as an option for `datasette publish` directly: datasette publish my.db --config default_page_size:50 --config sql_time_limit_ms:3500 | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/741/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | |||||||
607888367 | MDU6SXNzdWU2MDc4ODgzNjc= | 13 | Also upload movie files | simonw 9599 | open | 0 | 2 | 2020-04-27T22:11:25Z | 2020-04-28T00:39:45Z | MEMBER | The `upload` command currently only handles static images: https://github.com/dogsheep/photos-to-sqlite/blob/d939455af00e07866686457ee2fcb9b2d1b7194e/photos_to_sqlite/utils.py#L26-L33 Need to cover movies taken by my phone and DSLR too. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
608512747 | MDU6SXNzdWU2MDg1MTI3NDc= | 14 | Annotate photos using the Google Cloud Vision API | simonw 9599 | open | 0 | 5 | 2020-04-28T18:09:03Z | 2020-04-28T18:19:06Z | MEMBER | It can detect faces, run OCR, do image labeling (it knows what a lemur is!) and do object localization where it identifies objects and returns bounding polygons for them. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14/reactions", "total_count": 3, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0} | ||||||||
612287234 | MDU6SXNzdWU2MTIyODcyMzQ= | 16 | Import machine-learning detected labels (dog, llama etc) from Apple Photos | simonw 9599 | open | 0 | 13 | 2020-05-05T02:45:43Z | 2020-05-05T05:38:16Z | MEMBER | Follow-on from #1. Apple Photos runs some very sophisticated machine learning on-device to figure out if photos are of dogs, llamas and so on. I really want to extract those labels out into my own database. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16/reactions", "total_count": 2, "+1": 0, "-1": 0, "laugh": 1, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
612382643 | MDU6SXNzdWU2MTIzODI2NDM= | 758 | Question: Access to immutable database-path | clausjuhl 2181410 | open | 0 | 6 | 2020-05-05T07:01:18Z | 2020-05-28T08:23:27Z | NONE | Hi Simon Is there anywhere in the app-context where one can access the hashed urlpath of the database? Currently it's included in the template-context (`databases[0]["path")` when rendering urls of the database (eg. `/db-44b06v9/cases`...), but where can I find the hashed url when rendering the index-page? I'm trying to avoid redirects. Thanks! | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/758/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
612860758 | MDU6SXNzdWU2MTI4NjA3NTg= | 18 | Switch CI solution to GitHub Actions with a macOS runner | simonw 9599 | open | 0 | 1 | 2020-05-05T20:03:50Z | 2020-05-05T23:49:18Z | MEMBER | Refs #17. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
613422636 | MDU6SXNzdWU2MTM0MjI2MzY= | 760 | Way of seeing full schema for a database | simonw 9599 | open | 0 | 3 | 2020-05-06T15:46:08Z | 2020-05-06T23:49:06Z | OWNER | I find myself wanting to quickly figure out all of the BLOB columns in a database. A `/-/schema` page showing the full schema (actually since it's per-database probably `/dbname/-/schema` or `/-/schema/dbname`) would be really handy. It would need to be carefully constructed from various queries against `sqlite_master` - just doing `select * from sqlite_master where type='table'` isn't quite enough because I also want to show indexes, triggers etc. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/760/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
613491342 | MDU6SXNzdWU2MTM0OTEzNDI= | 762 | Experiment with PRAGMA hard_heap_limit | simonw 9599 | open | 0 | 0 | 2020-05-06T17:33:23Z | 2020-05-07T03:08:44Z | OWNER | This was added in SQLite 2020-01-22 (3.31.0): https://www.sqlite.org/changes.html#version_3_31_0 > Add the [sqlite3_hard_heap_limit64()](https://www.sqlite.org/c3ref/hard_heap_limit64.html) interface and the corresponding [PRAGMA hard_heap_limit](https://www.sqlite.org/pragma.html#pragma_hard_heap_limit) command. This sounds like it could be a nice extra safety measure. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/762/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
615626118 | MDU6SXNzdWU2MTU2MjYxMTg= | 22 | Try out ExifReader | simonw 9599 | open | 0 | 4 | 2020-05-11T06:32:13Z | 2020-05-14T05:59:53Z | MEMBER | https://pypi.org/project/ExifReader/ New fork that should be able to handle EXIF in HEIC files. Forked here: https://github.com/ianare/exif-py/issues/102#issuecomment-626376522 Refs #3 | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
616087149 | MDU6SXNzdWU2MTYwODcxNDk= | 765 | publish heroku should default to currently tagged version | simonw 9599 | open | 0 | 1 | 2020-05-11T18:24:06Z | 2020-05-11T18:25:43Z | OWNER | Had a report that deploying to Heroku was using the previously installed version of Datasette, not the latest. Could be because of this: https://github.com/simonw/datasette/blob/af6c6c5d6f929f951c0e63bfd1c82e37a071b50f/datasette/publish/heroku.py#L172-L179 Heroku documentation recommends pinning to specific versions https://devcenter.heroku.com/articles/python-pip So... we could ensure we default to an install value of `["datasette>=current_tag"]`. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/765/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
617323873 | MDU6SXNzdWU2MTczMjM4NzM= | 766 | Enable wildcard-searches by default | clausjuhl 2181410 | open | 0 | 2 | 2020-05-13T10:14:48Z | 2021-03-05T16:35:21Z | NONE | Hi Simon. It seems that datasette currently has wildcard-searches disabled by default (along with the boolean search-options, NEAR-queries and more, and despite the docs). If I try out the search-url provided in the [docs](https://datasette.readthedocs.io/en/stable/full_text_search.html#the-table-page-and-table-view-api) (https://fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort), it does not handle wildcard-searches, and I'm unable to make it work on my datasette-instance. I would argue that wildcard-searches is such a standard query, that it should be enabled by default. Requiring "_searchmode=raw" when using prefix-searches seems unnecessary. Plus: What happens to non-ascii searches when using "_searchmode=raw"? Is the "escape_fts"-function from datasette.utils ignored? Thanks! /Claus | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/766/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
621323348 | MDU6SXNzdWU2MjEzMjMzNDg= | 24 | Configurable URL for images | simonw 9599 | open | 0 | 1 | 2020-05-19T22:25:56Z | 2020-05-20T06:00:29Z | MEMBER | This is hard-coded at the moment, which is bad: https://github.com/dogsheep/photos-to-sqlite/blob/d5d69b9019703c47bc251444838578dd752801e2/photos_to_sqlite/cli.py#L269-L272 | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/24/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
621486115 | MDU6SXNzdWU2MjE0ODYxMTU= | 27 | photos_with_apple_metadata view should include labels | simonw 9599 | open | 0 | 0 | 2020-05-20T06:06:17Z | 2020-05-20T06:06:17Z | MEMBER | https://dogsheep-photos.dogsheep.net/public/photos_with_apple_metadata?place_city=New+Orleans&_facet=place_city&_facet_array=albums&_facet_array=persons Here's one way to add that: ```sql select rowid, photo, ( select json_group_array( json_object( 'label', normalized_string, 'href', '/photos/labelled?_hide_sql=1&label=' || normalized_string ) ) from labels where labels.uuid = photos_with_apple_metadata.uuid ) as labels, date, ``` | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/27/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
624490929 | MDU6SXNzdWU2MjQ0OTA5Mjk= | 28 | Invalid SQL no such table: main.uploads | dmd 41439 | open | 0 | 1 | 2020-05-25T21:25:39Z | 2020-12-24T22:26:22Z | NONE | http://127.0.0.1:8001/photos/photos_with_apple_metadata gives "Invalid SQL no such table: main.uploads" | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/28/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
626211658 | MDU6SXNzdWU2MjYyMTE2NTg= | 778 | Ability to configure keyset pagination for views and queries | simonw 9599 | open | 0 | 1 | 2020-05-28T04:48:56Z | 2020-10-02T02:26:25Z | OWNER | Currently views offer pagination, but it uses offset/limit - e.g. https://latest.datasette.io/fixtures/paginated_view?_next=100 This means pagination will perform poorly on deeper pages. If a view is based on a table that has a primary key it should be possible to configure efficient keyset pagination that works the same way that table pagination works. This may be as simple as configuring a column that can be treated as a "primary key" for the purpose of pagination using `metadata.json` - or with a `?_view_pk=colname` querystring argument. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/778/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
626582657 | MDU6SXNzdWU2MjY1ODI2NTc= | 779 | Make human_description_en explicitly available to output renderers | simonw 9599 | open | 0 | 0 | 2020-05-28T14:59:54Z | 2020-05-28T14:59:54Z | OWNER | `datasette-atom` uses this: https://github.com/simonw/datasette-atom/blob/df98a6c43a443224b6cd232f84703ec297ef046b/datasette_atom/__init__.py#L36-L37 ```python if data.get("human_description_en"): title += ": " + data["human_description_en"] ``` It's a nice way to generate a useful title for a filtered table. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/779/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
626593402 | MDU6SXNzdWU2MjY1OTM0MDI= | 780 | Internals documentation for datasette.metadata() method | simonw 9599 | open | 0 | Datasette 1.0 3268330 | 2 | 2020-05-28T15:14:22Z | 2022-03-15T20:50:34Z | OWNER | https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/datasette/app.py#L297-L328 | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/780/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | |||||||
627794879 | MDU6SXNzdWU2Mjc3OTQ4Nzk= | 782 | Redesign default .json format | simonw 9599 | open | 0 | Datasette 1.0a3 8755003 | 54 | 2020-05-30T18:47:07Z | 2023-01-17T02:05:45Z | OWNER | The default JSON just isn't right. I find myself using `?_shape=array` for almost everything I build against the API. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/782/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | |||||||
628156527 | MDU6SXNzdWU2MjgxNTY1Mjc= | 789 | Mechanism for enabling pluggy tracing | simonw 9599 | open | 0 | 2 | 2020-06-01T05:10:14Z | 2020-06-01T05:11:03Z | OWNER | Could be useful for debugging plugins: https://pluggy.readthedocs.io/en/latest/#call-tracing I tried this out by adding these two lines in `plugins.py`: ```python pm = pluggy.PluginManager("datasette") pm.add_hookspecs(hookspecs) # Added these: pm.trace.root.setwriter(print) pm.enable_tracing() ``` Output looked something like this: ``` INFO: 127.0.0.1:52724 - "GET /-/-/static/app.css HTTP/1.1" 404 Not Found actor_from_request [hook] datasette: <datasette.app.Datasette object at 0x106277ad0> request: <datasette.utils.asgi.Request object at 0x106550a50> finish actor_from_request --> [] [hook] extra_body_script [hook] template: show_json.html database: None table: None view_name: json_data datasette: <datasette.app.Datasette object at 0x106277ad0> finish extra_body_script --> [] [hook] extra_template_vars [hook] template: show_json.html database: None table: None view_name: json_data request: <datasette.utils.asgi.Request object at 0x1065504d0> datasette: <datasette.app.Datasette object at 0x106277ad0> finish extra_template_vars --> [] [hook] extra_css_urls [hook] template: show_json.html database: None table: None datasette: <datasette.app.Datasette object at 0x106277ad0> finish extra_css_urls --> [] [hook] extra_js_urls [hook] template: show_json.html database: None table: None datasette: <datasette.app.Datasette object at 0x106277ad0> finish extra_js_urls --> [] [hook] INFO: 127.0.0.1:52724 - "GET /-/actor HTTP/1.1" 200 OK actor_from_request [hook] datasette: <datasette.app.Datasette object at 0x106277ad0> request: <datasette.utils.asgi.Request object at 0x1065500d0> finish actor_from_request --> [] [hook] ``` | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/789/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
628572716 | MDU6SXNzdWU2Mjg1NzI3MTY= | 791 | Tutorial: building a something-interesting with writable canned queries | simonw 9599 | open | 0 | 2 | 2020-06-01T16:32:05Z | 2020-10-10T23:34:42Z | OWNER | Initial idea: TODO list, as a tutorial for #698 writable canned queries. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/791/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
629473827 | MDU6SXNzdWU2Mjk0NzM4Mjc= | 5 | Set up a demo | harryvederci 26745575 | open | 0 | 1 | 2020-06-02T19:56:49Z | 2020-09-01T06:18:43Z | NONE | First off, thanks for open sourcing this application! This is a suggestion to increase the amount of people that would make use of it: an example in the readme file would help. Currently, users have to clone the app, install it, authorize through pocket, run a command, an then find out if this application does what they hope it does. Another possibility is to add a file `example-output.db`, containing one (mock) Pocket article. Keep up the good work! | pocket-to-sqlite 213286752 | issue | {"url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
632724154 | MDU6SXNzdWU2MzI3MjQxNTQ= | 805 | Writable canned queries live demo on Glitch | simonw 9599 | open | 0 | 11 | 2020-06-06T20:52:13Z | 2020-07-01T22:44:01Z | OWNER | Needs to run somewhere with a mutable disk drive, so not Cloud Run or Heroku or Vercel. I think I'll put it on Glitch. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/805/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
634663505 | MDU6SXNzdWU2MzQ2NjM1MDU= | 815 | Group permission checks by request on /-/permissions debug page | simonw 9599 | open | 0 | 8 | 2020-06-08T14:25:23Z | 2020-12-17T22:06:48Z | OWNER | Now that we're making a LOT more permission checks (on the DB index page we do a check for every listed table for example) the `/-/permissions` page gets filled up pretty quickly. Can make this more readable by grouping permission checks by request. Have most recent request at the top of the page but the permission requests within that page sorted chronologically by most recent last. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/815/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
636511683 | MDU6SXNzdWU2MzY1MTE2ODM= | 830 | Redesign register_facet_classes plugin hook | simonw 9599 | open | 0 | Datasette 1.0 3268330 | 3 | 2020-06-10T20:03:27Z | 2021-12-16T19:58:22Z | OWNER | Nothing uses this plugin hook yet, so the design is not yet proven. I'm going to build a real plugin against it and use that process to inform any design changes that may need to be made. I'll add a warning about this to the documentation. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/830/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | |||||||
638238548 | MDU6SXNzdWU2MzgyMzg1NDg= | 845 | Code coverage should ignore files in .coveragerc | simonw 9599 | open | 0 | 0 | 2020-06-13T21:45:42Z | 2020-06-13T21:46:03Z | OWNER | I'm not sure why this is, but the code coverage I have running in a GitHub Action doesn't take my `.coveragerc` file into account. It should: https://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/.github/workflows/test-coverage.yml#L31-L35 Here's the bit that's ignored: https://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/.coveragerc#L1-L2 As a result my coverage score is 84%, when it should be 92%: ``` 2020-06-13T21:41:18.4404252Z ----------- coverage: platform linux, python 3.8.3-final-0 ----------- 2020-06-13T21:41:18.4404570Z Name Stmts Miss Cover 2020-06-13T21:41:18.4404971Z -------------------------------------------------------- 2020-06-13T21:41:18.4405227Z datasette/__init__.py 3 0 100% 2020-06-13T21:41:18.4405441Z datasette/__main__.py 3 3 0% 2020-06-13T21:41:18.4405668Z datasette/_version.py 279 279 0% 2020-06-13T21:41:18.4405921Z datasette/actor_auth_cookie.py 20 0 100% 2020-06-13T21:41:18.4406135Z datasette/app.py 499 27 95% 2020-06-13T21:41:18.4406343Z datasette/cli.py 162 45 72% 2020-06-13T21:41:18.4406553Z datasette/database.py 236 17 93% 2020-06-13T21:41:18.4406761Z datasette/default_permissions.py 40 0 100% 2020-06-13T21:41:18.4406975Z datasette/facets.py 210 24 89% 2020-06-13T21:41:18.4407186Z datasette/filters.py 122 7 94% 2020-06-13T21:41:18.4407394Z datasette/hookspecs.py 34 0 100% 2020-06-13T21:41:18.4407600Z datasette/inspect.py 36 23 36% 2020-06-13T21:41:18.4407807Z datasette/plugins.py 34 6 82% 2020-06-13T21:41:18.4408014Z datasette/publish/__init__.py 0 0 100% 2020-06-13T21:41:18.4408240Z datasette/publish/cloudrun.py 57 … | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/845/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
639542974 | MDU6SXNzdWU2Mzk1NDI5NzQ= | 47 | Fall back to FTS4 if FTS5 is not available | hpk42 73579 | open | 0 | 3 | 2020-06-16T10:11:23Z | 2020-06-17T20:13:48Z | NONE | got this with version 0.21.1 from pypi. twitter-to-sqlite auth worked but then "twitter-to-sqlite user-timeline USER.db" produced a tracekback ending in "no such module: FTS5". | twitter-to-sqlite 206156866 | issue | {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
639993467 | MDU6SXNzdWU2Mzk5OTM0Njc= | 850 | Proof of concept for Datasette on AWS Lambda with EFS | simonw 9599 | open | 0 | 25 | 2020-06-16T21:48:31Z | 2020-06-16T23:52:16Z | OWNER | https://aws.amazon.com/about-aws/whats-new/2020/06/aws-lambda-support-for-amazon-elastic-file-system-now-generally-/ If Datasette can run on Lambda with access to EFS it could both read AND write large databases there. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/850/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
642296989 | MDU6SXNzdWU2NDIyOTY5ODk= | 856 | Consider pagination of canned queries | simonw 9599 | open | 0 | 3 | 2020-06-20T03:15:59Z | 2021-05-21T14:22:41Z | OWNER | The new `canned_queries()` plugin hook from #852 combined with plugins like https://github.com/simonw/datasette-saved-queries could mean that some installations end up with hundreds or even thousands of canned queries. I should consider pagination or some other way of ensuring that this doesn't cause performance problems for Datasette. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/856/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
642388564 | MDU6SXNzdWU2NDIzODg1NjQ= | 858 | publish heroku does not work on Windows 10 | simonlau 870912 | open | 0 | 7 | 2020-06-20T14:40:28Z | 2021-06-10T17:44:09Z | NONE | When executing "datasette publish heroku schools.db" on Windows 10, I get the following error ```shell File "c:\users\dell\.virtualenvs\sec-schools-jn-cwk8z\lib\site-packages\datasette\publish\heroku.py", line 54, in heroku line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines() File "c:\python38\lib\subprocess.py", line 411, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "c:\python38\lib\subprocess.py", line 489, in run with Popen(*popenargs, **kwargs) as process: File "c:\python38\lib\subprocess.py", line 854, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "c:\python38\lib\subprocess.py", line 1307, in _execute_child hp, ht, pid, tid = _winapi.CreateProcess(executable, args, FileNotFoundError: [WinError 2] The system cannot find the file specified ``` Changing https://github.com/simonw/datasette/blob/55a6ffb93c57680e71a070416baae1129a0243b8/datasette/publish/heroku.py#L54 to ```python line.split()[0] for line in check_output(["heroku", "plugins"], shell=True).splitlines() ``` as well as the other `check_output()` and `call()` within the same file leads me to another recursive error about temp files | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/858/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
642572841 | MDU6SXNzdWU2NDI1NzI4NDE= | 859 | Database page loads too slowly with many large tables (due to table counts) | abdusco 3243482 | open | 0 | 21 | 2020-06-21T14:23:17Z | 2021-08-25T21:59:55Z | CONTRIBUTOR | Hey, I have a database that I save in HTML from couple of web scrapers. There are around 200k+, 50+ rows in a couple of tables, with sqlite file weighing around 600MB. The app runs on a VPS with 2 core CPU, 4GB RAM and refreshing database page regularly takes more than 10 seconds. I was suspecting that counting tables was the culprit, but manually running `select count(*) from table_name` for the largest table finishes under a second. I've looked at the source code. There's a check for index page for mutable databases larger than 100MB https://github.com/simonw/datasette/blob/799c5d53570d773203527f19530cf772dc2eeb24/datasette/views/index.py#L15 but this check is not performed for database page. I've manually crippled `Database::table_counts` method ```py async def table_counts(self, limit=10): if not self.is_mutable and self.cached_table_counts is not None: return self.cached_table_counts # Try to get counts for each table, $limit timeout for each count counts = {} for table in await self.table_names(): try: # table_count = ( # await self.execute( # "select count(*) from [{}]".format(table), # custom_time_limit=limit, # ) # ).rows[0][0] counts[table] = 10 # table_count # In some cases I saw "SQL Logic Error" here in addition to # QueryInterrupted - so we catch that too: except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError): counts[table] = None if not self.is_mutable: self.cached_table_counts = counts return counts ``` now the page loads in <100ms. Is it possible to apply size check on database page too? <details> <summary> /-/versions output </summary> <pre> { "python": { "version": "3.8.0", "full": "3.8.0 (default, Oct 28 2019, 16:14:01) \n[GCC 8.3.0]" }, "datasette": { "version": "0.44" }, "asgi": "3.… | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/859/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
643510821 | MDU6SXNzdWU2NDM1MTA4MjE= | 862 | Set an upper limit on total facet suggestion time for a page | simonw 9599 | open | 0 | 1 | 2020-06-23T03:57:55Z | 2020-06-23T03:58:48Z | OWNER | If a table has 100 columns the facet suggestion code will currently run 100 times, taking a max of `facet_suggest_time_limit_ms` which defaults to 50ms per column: https://github.com/simonw/datasette/blob/000528192eaf891118932250141dabe7a1561ece/datasette/facets.py#L142-L162 So for 100 columns, that's 100 * 50ms = 5s total time that might be spent attempting to calculate facets on a large table! I should implement a hard upper limit on the total amount of time taken suggesting facets - probably of around 500ms. If it takes longer than that the remaining columns will not be considered. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/862/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
644161221 | MDU6SXNzdWU2NDQxNjEyMjE= | 117 | Support for compound (composite) foreign keys | simonw 9599 | open | 0 | 3 | 2020-06-23T21:33:42Z | 2020-06-23T21:40:31Z | OWNER | It turns out SQLite supports composite foreign keys: https://www.sqlite.org/foreignkeys.html#fk_composite Their example looks like this: ```sql CREATE TABLE album( albumartist TEXT, albumname TEXT, albumcover BINARY, PRIMARY KEY(albumartist, albumname) ); CREATE TABLE song( songid INTEGER, songartist TEXT, songalbum TEXT, songname TEXT, FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) ); ``` Here's what that looks like in sqlite-utils: ``` In [1]: import sqlite_utils In [2]: import sqlite3 In [3]: conn = sqlite3.connect(":memory:") In [4]: conn Out[4]: <sqlite3.Connection at 0x1087186c0> In [5]: conn.executescript(""" ...: CREATE TABLE album( ...: albumartist TEXT, ...: albumname TEXT, ...: albumcover BINARY, ...: PRIMARY KEY(albumartist, albumname) ...: ); ...: ...: CREATE TABLE song( ...: songid INTEGER, ...: songartist TEXT, ...: songalbum TEXT, ...: songname TEXT, ...: FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) ...: ); ...: """) Out[5]: <sqlite3.Cursor at 0x1088def10> In [6]: db = sqlite_utils.Database(conn) In [7]: db.tables … | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/117/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
646448486 | MDExOlB1bGxSZXF1ZXN0NDQwNzM1ODE0 | 868 | initial windows ci setup | joshmgrant 702729 | open | 0 | 3 | 2020-06-26T18:49:13Z | 2021-07-10T23:41:43Z | FIRST_TIME_CONTRIBUTOR | simonw/datasette/pulls/868 | Picking up the work done on #557 with a new PR. Seeing if I can get this working. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/868/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||||
646737558 | MDU6SXNzdWU2NDY3Mzc1NTg= | 870 | Refactor default views to use register_routes | simonw 9599 | open | 0 | 10 | 2020-06-27T18:53:12Z | 2022-03-15T20:07:18Z | OWNER | It would be much cleaner if Datasette's default views were all registered using the new `register_routes()` plugin hook. Could dramatically reduce the code in `datasette/app.py`. > The ideal fix here would be to rework my `BaseView` subclass mechanism to work with `register_routes()` so that those views don't have any special privileges above plugin-provided views. _Originally posted by @simonw in https://github.com/simonw/datasette/issues/864#issuecomment-648580556_ | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/870/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
647095487 | MDU6SXNzdWU2NDcwOTU0ODc= | 873 | "datasette -p 0 --root" gives the wrong URL | simonw 9599 | open | 0 | 14 | 2020-06-29T04:03:06Z | 2020-08-18T17:26:10Z | OWNER | ``` $ datasette -p 0 --root http://127.0.0.1:0/-/auth-token?token=2d498c... ``` The port is incorrect. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/873/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
648435885 | MDU6SXNzdWU2NDg0MzU4ODU= | 878 | New pattern for views that return either JSON or HTML, available for plugins | simonw 9599 | open | 0 | Datasette 1.0 3268330 | 26 | 2020-06-30T19:26:13Z | 2022-03-19T16:19:30Z | OWNER | Can be part of #870 - refactoring existing views to use `register_routes()`. > I'm going to put the new `check_permissions()` method on `BaseView` as well. If I want that method to be available to plugins I can do so by turning that `BaseView` class into a documented API that plugins are encouraged to use themselves. _Originally posted by @simonw in https://github.com/simonw/datasette/issues/832#issuecomment-651995453_ | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/878/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | |||||||
648659536 | MDU6SXNzdWU2NDg2NTk1MzY= | 881 | Figure out why restore_working_directory is needed in some places | simonw 9599 | open | 0 | 0 | 2020-07-01T04:19:25Z | 2020-07-01T04:19:25Z | OWNER | This is a frustrating workaround. I have a `restore_working_directory` fixture that I wrote to solve errors that look like this: ``` /Users/simon/Dropbox/Development/datasette/tests/test_publish_cloudrun.py:148: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:112: in __enter__ return next(self.gen) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <click.testing.CliRunner object at 0x1135ad110> @contextlib.contextmanager def isolated_filesystem(self): """A context manager that creates a temporary folder and changes the current working directory to it for isolated filesystem tests. """ > cwd = os.getcwd() E FileNotFoundError: [Errno 2] No such file or directory ``` Here's an example of it in use: removing the `restore_working_directory` argument from this function causes the failure. https://github.com/simonw/datasette/blob/549b1c2063db48c4622ee5c7b478a1e3cbc1ac07/tests/test_plugins.py#L689-L690 I'd like to not have to do this. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/881/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
648749062 | MDExOlB1bGxSZXF1ZXN0NDQyNTA1MDg4 | 883 | Skip counting hidden tables | abdusco 3243482 | open | 0 | 4 | 2020-07-01T07:38:08Z | 2020-07-02T00:25:44Z | CONTRIBUTOR | simonw/datasette/pulls/883 | Potential fix for https://github.com/simonw/datasette/issues/859. Disabling table counts for hidden tables speeds up database page quite a bit. In my setup it reduced load time by 2/3 (~300 -> ~90ms) | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/883/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||||
649429772 | MDU6SXNzdWU2NDk0Mjk3NzI= | 886 | Reconsider how _actor_X magic parameter deals with missing values | simonw 9599 | open | 0 | 2 | 2020-07-02T00:00:38Z | 2020-09-11T21:35:26Z | OWNER | I had to build a custom `_actorornull` prefix for [datasette-saved-queries](https://github.com/simonw/datasette-saved-queries/blob/37c00e56ac398e1f9aa342d30357de013a9b37b4/datasette_saved_queries/__init__.py): ```python def actorornull(key, request): if request.actor is None: return None return request.actor.get(key) @hookimpl def register_magic_parameters(): return [ ("actorornull", actorornull), ] ``` Maybe the `actor` magic in Datasette core should do that out of the box? https://github.com/simonw/datasette/blob/f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb/datasette/default_magic_parameters.py#L14-L17 | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/886/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
652961907 | MDU6SXNzdWU2NTI5NjE5MDc= | 121 | Improved (and better documented) support for transactions | simonw 9599 | open | 0 | 3 | 2020-07-08T04:56:51Z | 2020-09-24T20:36:46Z | OWNER | _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393_ We should put some thought into how this library supports and encourages smart use of transactions. | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/121/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
655974395 | MDExOlB1bGxSZXF1ZXN0NDQ4MzU1Njgw | 30 | Handle empty bucket on first upload. Allow specifying the endpoint_url for services other than S3 (like b2 and digitalocean spaces) | scanner 110038 | open | 0 | 0 | 2020-07-13T16:15:26Z | 2020-07-13T16:15:26Z | FIRST_TIME_CONTRIBUTOR | dogsheep/dogsheep-photos/pulls/30 | Finally got around to trying dogsheep-photos but I want to use backblaze's b2 service instead of AWS S3. Had to add a way to optionally specify the endpoint_url to connect to. Then with the bucket being empty the initial key retrieval would fail. Probably a better way to see that the bucket is empty than doing a test inside the paginator loop. Also probably a better way to specify the endpoint_url as we get and test for it twice using the same code in two different places but did not want to spend too much time worrying about it. | dogsheep-photos 256834907 | pull | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/30/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||||
657572753 | MDU6SXNzdWU2NTc1NzI3NTM= | 894 | ?sort=colname~numeric to sort by by column cast to real | simonw 9599 | open | 0 | 21 | 2020-07-15T18:47:48Z | 2021-08-20T02:07:53Z | OWNER | If a text column actually contains numbers, being able to "sort by column, treated as numeric" would be really useful. Probably depends on column actions enabled by #690 | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/894/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
659873662 | MDU6SXNzdWU2NTk4NzM2NjI= | 898 | datasette.utils.testing module | simonw 9599 | open | 0 | 2 | 2020-07-18T03:53:24Z | 2020-07-18T03:57:46Z | OWNER | The unit tests for plugins could benefit from reusing code from Datasette's own testing fixtures, e.g.: > I may need to borrow this function from Datasette for the tests: > https://github.com/simonw/datasette/blob/1f6a134369e6a7efaae9db469f15b1dd2b7f3709/tests/fixtures.py#L836-L851 > > It's not importable (it lives in `fixtures.py` and not in the `datasette` package that gets packaged for PyPI) - maybe I should fix that in Datasette by adding a `from datasette.utils.testing` module. _Originally posted by @simonw in https://github.com/simonw/datasette-update-api/issues/4#issuecomment-660419182_ | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/898/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
664485022 | MDU6SXNzdWU2NjQ0ODUwMjI= | 46 | Feature: pull request reviews and comments | bhrutledge 1326704 | open | 0 | 6 | 2020-07-23T13:43:45Z | 2022-12-20T14:40:15Z | NONE | Hi there! I saw your [presentation at Boston Python](https://www.meetup.com/bostonpython/events/271887195). I'm already a light user of Datasette (thank you!), but wasn't aware of this project. I've been working on a "pull request dashboard" to get a comprehensive view of the state of open PR's, esp. related to reviews (i.e., pending, approved, changes requested). Currently it's a CLI command, but I thought a Datasette UI might be fun. I see that PR's are available from the `issues` command, but I don't see reviews anywhere. From the [API docs](https://docs.github.com/en/rest/reference/pulls#reviews), it looks like there are separate endpoints for those (as well as pull requests in general). What do you think about adding that? Would you accept a PR? Any sense of the level of effort? | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
664793260 | MDU6SXNzdWU2NjQ3OTMyNjA= | 2 | Yak shave | ekg 145425 | open | 0 | 0 | 2020-07-23T22:04:18Z | 2020-07-23T22:04:18Z | NONE | Just a quick note... The 23andme data is not exactly your genome, but a SNP chip of your genome. It's "some of your genotypes." Or about 0.1% of your genome. Nice work in any case! It deserves to be liberated!!!!! | genome-to-sqlite 209590345 | issue | {"url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
668064026 | MDU6SXNzdWU2NjgwNjQwMjY= | 911 | Rethink the --name option to "datasette publish" | simonw 9599 | open | 0 | Datasette 1.0 3268330 | 0 | 2020-07-29T18:49:49Z | 2020-07-29T18:49:49Z | OWNER | `--name` works inconsistently across the different publish providers - on Cloud Run you should use `--service` instead for example. Need to review it across all of them and either remove it or clarify what it does. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/911/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | |||||||
670209331 | MDU6SXNzdWU2NzAyMDkzMzE= | 913 | Mechanism for passing additional options to `datasette my.db` that affect plugins | simonw 9599 | open | 0 | 5 | 2020-07-31T20:38:26Z | 2021-01-04T20:04:11Z | OWNER | > It's a shame there's no obvious mechanism for passing additional options to `datasette my.db` that affect how plugins work. > >The only way I can think of at the moment is via environment variables: > > DATASETTE_INSERT_UNSAFE=1 datasette my.db > >This will have to do for the moment - it's ugly enough that people will at least know they are doing something unsafe, which is the goal here. _Originally posted by @simonw in https://github.com/simonw/datasette-insert/issues/15#issuecomment-667346438_ | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/913/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
672421411 | MDU6SXNzdWU2NzI0MjE0MTE= | 916 | Support reverse pagination (previous page, has-previous-items) | simonw 9599 | open | 0 | 7 | 2020-08-04T00:32:06Z | 2021-04-03T23:43:11Z | OWNER | I need this for `datasette-graphql` for full compatibility with the way Relay likes to paginate - using cursors for paginating backwards as well as for paginating forwards. > This may be the kick I need to get Datasette pagination to work in reverse too. _Originally posted by @simonw in https://github.com/simonw/datasette-graphql/issues/2#issuecomment-668305853_ | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/916/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
673602857 | MDU6SXNzdWU2NzM2MDI4NTc= | 9 | Define a view that displays photos correctly | simonw 9599 | open | 0 | 0 | 2020-08-05T14:53:39Z | 2020-08-05T14:53:39Z | MEMBER | The `photos` table stores data like this: id | createdAt | source | prefix | suffix | width | height | visibility | created ▲ | user -- | -- | -- | -- | -- | -- | -- | -- | -- | -- 5e12c9708506bc000840262a | January 06, 2020 - 05:45:20 UTC | Swarm for iOS 1 | https://fastly.4sqi.net/img/general/ | /15889193_AXxGk4I1nbzUZuyYqObgbXdJNyEHiwj6AUDq0tPZWtw.jpg | 1920 | 1440 | public | 2020-01-06T05:45:20 | 15889193 The photo URL can be derived from those pieces - define a SQL view which does that (using `datasette-json-html` to display the pictures) | swarm-to-sqlite 205429375 | issue | {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
675594325 | MDU6SXNzdWU2NzU1OTQzMjU= | 917 | Idea: "datasette publish" option for "only if the data has changed | simonw 9599 | open | 0 | 0 | 2020-08-08T21:58:27Z | 2020-08-08T21:58:27Z | OWNER | This is a pattern I often find myself needing. I usually implement this in GitHub Actions like this: https://github.com/simonw/covid-19-datasette/blob/efa01c39abc832b8641fc2a92840cc3acae2fb08/.github/workflows/scheduled.yml#L52-L63 ```yaml - name: Set variables to decide if we should deploy id: decide_variables run: |- echo "##[set-output name=latest;]$(datasette inspect covid.db | jq '.covid.hash' -r)" echo "##[set-output name=deployed;]$(curl -s https://covid-19.datasettes.com/-/databases.json | jq '.[0].hash' -r)" - name: Set up Cloud Run if: github.event_name == 'workflow_dispatch' || steps.decide_variables.outputs.latest != steps.decide_variables.outputs.deployed uses: GoogleCloudPlatform/github-actions/setup-gcloud@master ``` This is pretty fiddly. It might be good for `datasette publish` to grow a helper option that does effectively this - hashes the databases (and the `metadata.json`) and compares them to the deployed version. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/917/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
675753042 | MDU6SXNzdWU2NzU3NTMwNDI= | 131 | sqlite-utils insert: options for column types | simonw 9599 | open | 0 | 5 | 2020-08-09T18:59:11Z | 2022-03-15T13:21:42Z | OWNER | The `insert` command currently results in string types for every column - at least when used against CSV or TSV inputs. It would be useful if you could do the following: - automatically detects the column types based on eg the first 1000 records - explicitly state the rule for specific columns `--detect-types` could work for the former - or it could do that by default and allow opt-out using `--no-detect-types` For specific columns maybe this: sqlite-utils insert db.db images images.tsv \ --tsv \ -c id int \ -c score float | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/131/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
678760988 | MDU6SXNzdWU2Nzg3NjA5ODg= | 932 | End-user documentation | simonw 9599 | open | 0 | Datasette 1.0 3268330 | 6 | 2020-08-13T22:04:39Z | 2022-03-08T15:20:48Z | OWNER | Datasette's documentation is aimed at people who install and configure it. What about end users of preconfigured and deployed Datasette instances? Something that can be linked to from the Datasette UI would be really useful. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/932/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | |||||||
685806511 | MDU6SXNzdWU2ODU4MDY1MTE= | 950 | Private/secret databases: database files that are only visible to plugins | simonw 9599 | open | 0 | 5 | 2020-08-25T20:46:17Z | 2020-08-26T00:50:48Z | OWNER | In thinking about the best way to implement https://github.com/simonw/datasette-auth-passwords/issues/6 (SQL-backed user accounts for `datasette-auth-passwords`) I realized that there are a few different use-cases where a plugin might want to store data that isn't visible to regular Datasette users: - Storing password hashes - Storing API tokens - Storing secrets that are used for data import integrations (secrets for talking to the Twitter API for example) Idea: allow one or more private database files to be attached to Datasette, something like this: datasette github.db linkedin.db -s secrets.db -m metadata.yml The `secrets.db` file would not be visible using any of the Datasette's usual interface or API routes - but plugins would be able to run queries against it. So `datasette-auth-passwords` might then be configured like this: ```yaml plugins: datasette-auth-passwords: database: secrets sql: "select password_hash from passwords where username = :username" ``` The plugin could even refuse to operate against a database that hadn't been loaded as a secret database. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/950/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
687694947 | MDU6SXNzdWU2ODc2OTQ5NDc= | 954 | Remove old register_output_renderer dict mechanism in Datasette 1.0 | simonw 9599 | open | 0 | Datasette 1.0 3268330 | 1 | 2020-08-28T04:04:23Z | 2020-08-28T04:56:31Z | OWNER | > Documentation says that the old dictionary mechanism will be deprecated by 1.0: > > https://github.com/simonw/datasette/blob/799ecae94824640bdff21f86997f69844048d5c3/docs/plugin_hooks.rst#L460 _Originally posted by @simonw in https://github.com/simonw/datasette/issues/953#issuecomment-682312494_ | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/954/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | |||||||
688351054 | MDU6SXNzdWU2ODgzNTEwNTQ= | 140 | Idea: insert-files mechanism for adding extra columns with fixed values | simonw 9599 | open | 0 | 1 | 2020-08-28T20:57:36Z | 2022-03-20T19:45:45Z | OWNER | Say for example you want to populate a `file_type` column with the value `gif`. That could work like this: ``` sqlite-utils insert-files gifs.db images *.gif \ -c path -c md5 -c last_modified:mtime \ -c file_type:text:gif --pk=path ``` So a column defined as a `text` column with a value that follows a second colon. | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/140/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
688352145 | MDU6SXNzdWU2ODgzNTIxNDU= | 141 | insert-files support for compressed values | simonw 9599 | open | 0 | 0 | 2020-08-28T20:59:46Z | 2020-09-24T20:36:08Z | OWNER | The `sqlar` format supports this, it would be useful if `insert-files` could support this too. https://www.sqlite.org/sqlar.html | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/141/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
688670158 | MDU6SXNzdWU2ODg2NzAxNTg= | 147 | SQLITE_MAX_VARS maybe hard-coded too low | simonwiles 96218 | open | 0 | 7 | 2020-08-30T07:26:45Z | 2021-02-15T21:27:55Z | CONTRIBUTOR | I came across this while about to open an issue and PR against the documentation for `batch_size`, which is a bit incomplete. As mentioned in #145, while: > [`SQLITE_MAX_VARIABLE_NUMBER`](https://www.sqlite.org/limits.html#max_variable_number) ... defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0. it is common that it is increased at compile time. Debian-based systems, for example, seem to ship with a version of sqlite compiled with SQLITE_MAX_VARIABLE_NUMBER set to 250,000, and I believe this is the case for homebrew installations too. In working to understand what `batch_size` was actually doing and why, I realized that by setting `SQLITE_MAX_VARS` in `db.py` to match the value my sqlite was compiled with (I'm on Debian), I was able to decrease the time to `insert_all()` my test data set (~128k records across 7 tables) from ~26.5s to ~3.5s. Given that this about .05% of my total dataset, this is time I am keen to save... Unfortunately, it seems that `sqlite3` in the python standard library doesn't expose the `get_limit()` C API (even though `pysqlite` used to), so it's hard to know what value sqlite has been compiled with (note that this could mean, I suppose, that it's less than 999, and even hardcoding `SQLITE_MAX_VARS` to the conservative default might not be adequate. It can also be lowered -- but not raised -- at runtime). The best I could come up with is `echo "" | sqlite3 -cmd ".limits variable_number"` (only available in `sqlite >= 2015-05-07 (3.8.10)`). Obviously this couldn't be relied upon in `sqlite_utils`, but I wonder what your opinion would be about exposing `SQLITE_MAX_VARS` as a user-configurable parameter (with suitable "here be dragons" warnings)? I'm going to go ahead and monkey-patch it for my purposes in any event, but it seems like it might be worth considering. | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/147/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
689848827 | MDU6SXNzdWU2ODk4NDg4Mjc= | 6 | ISO timestamps | simonw 9599 | open | 0 | 0 | 2020-09-01T06:16:42Z | 2020-09-01T06:16:42Z | MEMBER | The `time_added`, `time_updated` and `time_read` columns currently store data like this: September 19, 2019 - 00:30:30 UTC Should use ISO instead, e.g. `2020-07-26T01:05:24+00:00` | pocket-to-sqlite 213286752 | issue | {"url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
689850810 | MDU6SXNzdWU2ODk4NTA4MTA= | 6 | Set up a demo instance | simonw 9599 | open | 0 | 0 | 2020-09-01T06:20:24Z | 2020-09-01T06:20:24Z | MEMBER | Once I've got the Datasette plugin to a state where it's worth building a demo: #3 I can use data from my public https://github-to-sqlite.dogsheep.net/ demo plus the Pocket data subset I use for the demo in https://github.com/dogsheep/pocket-to-sqlite/issues/5 - I could pull in the https://dogsheep-photos.dogsheep.net/ photos data too. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
691537426 | MDU6SXNzdWU2OTE1Mzc0MjY= | 959 | Internals API idea: results.dicts in addition to results.rows | simonw 9599 | open | 0 | 0 | 2020-09-03T00:50:17Z | 2020-09-03T00:50:17Z | OWNER | I just wrote this code: ```python results = await database.execute(SEARCH_SQL, {"query": query}) return [dict(r) for r in results.rows] ``` How about having `results.dicts` as a utility property that does that? | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/959/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
692202408 | MDU6SXNzdWU2OTIyMDI0MDg= | 12 | Idea: maps and GeoJSON support | simonw 9599 | open | 0 | 0 | 2020-09-03T18:47:10Z | 2020-09-04T01:45:03Z | MEMBER | It would be cool if the `display_sql` could return a column populated with GeoJSON which would the automatically be displayed on a map in the results (or maybe default JS would look for a `class="geojson"` element output by the `display` template) - ala https://github.com/simonw/datasette-leaflet-geojson Then I could render workout routes on a map, or Swarm checkin points. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
694136490 | MDU6SXNzdWU2OTQxMzY0OTA= | 15 | Add a bunch of config examples | simonw 9599 | open | 0 | 1 | 2020-09-05T17:58:43Z | 2020-09-18T23:17:39Z | MEMBER | I can bring these over from my personal Dogsheep. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/15/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
694493566 | MDU6SXNzdWU2OTQ0OTM1NjY= | 16 | Timeline view | simonw 9599 | open | 0 | 3 | 2020-09-06T19:13:58Z | 2020-09-21T02:42:29Z | MEMBER | Ability to browse (and facet) by date. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
695441530 | MDU6SXNzdWU2OTU0NDE1MzA= | 154 | OperationalError: cannot change into wal mode from within a transaction | simonw 9599 | open | 0 | 2 | 2020-09-07T23:42:44Z | 2020-09-07T23:47:10Z | OWNER | I'm getting this error when running: sqlite-utils enable-wal beta.db `OperationalError: cannot change into wal mode from within a transaction` I'm worried that maybe that's because of this new code from #152: https://github.com/simonw/sqlite-utils/blob/deb2eb013ff85bbc828ebc244a9654f0d9c3139e/sqlite_utils/db.py#L128-L129 | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/154/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
695553522 | MDU6SXNzdWU2OTU1NTM1MjI= | 18 | Deleted records stay in the search index | simonw 9599 | open | 0 | 2 | 2020-09-08T05:14:23Z | 2020-09-08T05:15:51Z | MEMBER | Here's why: https://github.com/dogsheep/dogsheep-beta/blob/24f7898d41a39218058f174c75ba62f7c0fcfff6/dogsheep_beta/utils.py#L44-L53 That should probably do `DELETE FROM index1.search_index WHERE [table] = ?` first. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
695556681 | MDU6SXNzdWU2OTU1NTY2ODE= | 19 | Figure out incremental re-indexing | simonw 9599 | open | 0 | 2 | 2020-09-08T05:23:31Z | 2020-09-08T05:27:07Z | MEMBER | As tables get bigger reindexing everything on a schedule (essentially recreating the entire index from scratch) will start to become a performance bottleneck. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
696908389 | MDU6SXNzdWU2OTY5MDgzODk= | 961 | Verification checks for metadata.json on startup | simonw 9599 | open | 0 | 2 | 2020-09-09T15:21:53Z | 2020-09-09T15:24:31Z | OWNER | I lost a bunch of time yesterday trying to figure out why a Datasette instance wasn't starting up - it turned out it was because I had a `facets:` reference that mentioned a column that did not exist. Catching these on startup would be good. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/961/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
697162939 | MDU6SXNzdWU2OTcxNjI5Mzk= | 20 | Add more tags so people can find your project. | ran88dom99 7902810 | open | 0 | 0 | 2020-09-09T21:14:09Z | 2020-09-09T21:14:09Z | NONE | quantified-self habit-tracking google-fit time-tracking wearables quantifiedself for example | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/20/reactions", "total_count": 1, "+1": 0, "-1": 1, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
698791218 | MDU6SXNzdWU2OTg3OTEyMTg= | 50 | favorites --stop_after=N stops after min(N, 200) | mikepqr 370930 | open | 0 | 2 | 2020-09-11T03:38:14Z | 2020-09-13T05:11:14Z | CONTRIBUTOR | For any number greater than 200, `favorites --stop_after` stops after getting 200 tweets, e.g. ``` $ twitter-to-sqlite favorites tweets.db --stop_after=300 Importing favorites [####################################] 199 $ ``` I don't _think_ this is a limitation of the API (if you omit `--stop_after` you get some very large number, possibly all of them), so I _think_ this is a bug. | twitter-to-sqlite 206156866 | issue | {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
702386948 | MDU6SXNzdWU3MDIzODY5NDg= | 159 | .delete_where() does not auto-commit (unlike .insert() or .upsert()) | spdkils 11712349 | open | 0 | 9 | 2020-09-16T01:55:52Z | 2023-04-01T17:21:05Z | NONE | When you use the delete_where() function on a table, it never commits.... Is that intentional? | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/159/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
703216044 | MDU6SXNzdWU3MDMyMTYwNDQ= | 49 | Feature: gists and starred gists | simonw 9599 | open | 0 | 0 | 2020-09-17T02:30:52Z | 2020-09-17T02:30:52Z | MEMBER | https://developer.github.com/v3/gists/#list-starred-gists | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/49/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
703218448 | MDU6SXNzdWU3MDMyMTg0NDg= | 51 | Documentation for twitter-to-sqlite fetch | simonw 9599 | open | 0 | 0 | 2020-09-17T02:38:10Z | 2020-09-17T02:38:10Z | MEMBER | It's mentioned in passing in the README but it deserves its own section: ``` $ twitter-to-sqlite fetch \ "https://api.twitter.com/1.1/account/verify_credentials.json" \ | grep '"id"' | head -n 1 ``` | twitter-to-sqlite 206156866 | issue | {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/51/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
703218756 | MDU6SXNzdWU3MDMyMTg3NTY= | 50 | Commands for making authenticated API calls | simonw 9599 | open | 0 | 7 | 2020-09-17T02:39:07Z | 2020-10-19T05:01:29Z | MEMBER | Similar to `twitter-to-sqlite fetch`, see https://github.com/dogsheep/twitter-to-sqlite/issues/51 | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
703246031 | MDU6SXNzdWU3MDMyNDYwMzE= | 51 | github-to-sqlite should handle rate limits better | simonw 9599 | open | 0 | 4 | 2020-09-17T04:01:50Z | 2022-10-14T16:34:07Z | MEMBER | From #50 - right now it will crash with an error of it hits the rate limit. Since the rate limit information (including reset time) is available in the headers it could automatically sleep and try again instead. | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
705215230 | MDU6SXNzdWU3MDUyMTUyMzA= | 26 | Pagination | simonw 9599 | open | 0 | 7 | 2020-09-21T00:14:37Z | 2020-09-21T02:55:54Z | MEMBER | Useful for #16 (timeline view) since you can now filter to just the items on a specific day - but if there are more than 50 items you can't see them all. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
705840673 | MDU6SXNzdWU3MDU4NDA2NzM= | 972 | Support faceting against arbitrary SQL queries | simonw 9599 | open | 0 | 1 | 2020-09-21T19:00:43Z | 2021-12-15T18:02:20Z | OWNER | > ... support for running facets against arbitrary custom SQL queries is half-done in that facets now execute against wrapped subqueries as-of ea66c45df96479ef66a89caa71fff1a97a862646 > > https://github.com/simonw/datasette/blob/ea66c45df96479ef66a89caa71fff1a97a862646/datasette/facets.py#L192-L200 _Originally posted by @simonw in https://github.com/simonw/datasette/issues/971#issuecomment-696307922_ | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/972/reactions", "total_count": 3, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 3, "rocket": 0, "eyes": 0} | ||||||||
706001517 | MDU6SXNzdWU3MDYwMDE1MTc= | 163 | Idea: conversions= could take Python functions | simonw 9599 | open | 0 | 4 | 2020-09-22T00:37:12Z | 2021-12-20T00:56:52Z | OWNER | Right now you use `conversions=` like this: ```python db["example"].insert({ "name": "The Bigfoot Discovery Museum" }, conversions={"name": "upper(?)"}) ``` How about if you could optionally provide a Python function (or a lambda) like this? ```python db["example"].insert({ "name": "The Bigfoot Discovery Museum" }, conversions={"name": lambda s: s.upper()}) ``` This would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end. | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/163/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
707849175 | MDU6SXNzdWU3MDc4NDkxNzU= | 974 | static assets and favicon aren't cached by the browser | obra 45416 | open | 0 | 1 | 2020-09-24T04:44:55Z | 2022-01-13T22:21:28Z | NONE | Using datasette to solve some frustrating problems with our fulfillment provider today, I was surprised to see repeated requests for assets under /-/static and the favicon. While it won't likely be a huge performance bottleneck, I bet datasette would feel a bit zippier if you had Uvicorn serving up some caching-related headers telling the browser it was safe to cache static assets. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/974/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
709789634 | MDU6SXNzdWU3MDk3ODk2MzQ= | 27 | Sort order is not persisted by facet filter links | simonw 9599 | open | 0 | 0 | 2020-09-27T18:22:07Z | 2020-09-27T18:22:07Z | MEMBER | A link to `/-/beta?category=1×tamp__date=2018-08-01&q=swedish` should be to `/-/beta?category=1×tamp__date=2018-08-01&q=swedish&sort=newest` | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/27/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
712202333 | MDU6SXNzdWU3MTIyMDIzMzM= | 982 | SQL editor should allow execution of write queries, if you have permission | simonw 9599 | open | 0 | 2 | 2020-09-30T19:04:35Z | 2022-01-13T22:21:29Z | OWNER | The `datasette-write` plugin provides this at the moment https://github.com/simonw/datasette-write - but it feels like it should be a built-in capability, protected by a default permission. UI concept: if you have write permission then the existing SQL editor gets an "execute write" checkbox underneath it. JavaScript can spot if you appear to be trying to execute an UPDATE or INSERT or DELETE query and check that checkbox for you. If you link to a query page with a non-SELECT then that query will be displayed in the box ready for you to POST submit it. The page will also then get "cannot be embedded" headers to protect against clickjacking. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/982/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
712260429 | MDU6SXNzdWU3MTIyNjA0Mjk= | 983 | JavaScript plugin hooks mechanism similar to pluggy | simonw 9599 | open | 0 | 47 | 2020-09-30T20:32:43Z | 2021-01-25T04:43:58Z | OWNER | > It would be neat to provide a JavaScript plugin hook that plugins can use to add their own options to this menu. No idea what that would look like though. _Originally posted by @simonw in https://github.com/simonw/datasette/issues/981#issuecomment-701616922_ | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/983/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
712368432 | MDU6SXNzdWU3MTIzNjg0MzI= | 984 | Review accessibility of new column action menus | simonw 9599 | open | 0 | 1 | 2020-09-30T23:56:44Z | 2020-10-01T00:01:36Z | OWNER | Feature added in #981 | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/984/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
712984738 | MDU6SXNzdWU3MTI5ODQ3Mzg= | 987 | Documented HTML hooks for JavaScript plugin authors | simonw 9599 | open | 0 | 7 | 2020-10-01T16:10:14Z | 2021-01-25T04:00:03Z | OWNER | In #981 I added `data-column=` attributes to the `<th>` on the table page. These should become part of Datasette's documented API so JavaScript plugin authors can use them to derive things about the tables shown on a page (`datasette-cluster-map uses them as-of https://github.com/simonw/datasette-cluster-map/issues/18). | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/987/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issues] ( [id] INTEGER PRIMARY KEY, [node_id] TEXT, [number] INTEGER, [title] TEXT, [user] INTEGER REFERENCES [users]([id]), [state] TEXT, [locked] INTEGER, [assignee] INTEGER REFERENCES [users]([id]), [milestone] INTEGER REFERENCES [milestones]([id]), [comments] INTEGER, [created_at] TEXT, [updated_at] TEXT, [closed_at] TEXT, [author_association] TEXT, [pull_request] TEXT, [body] TEXT, [repo] INTEGER REFERENCES [repos]([id]), [type] TEXT , [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT); CREATE INDEX [idx_issues_repo] ON [issues] ([repo]); CREATE INDEX [idx_issues_milestone] ON [issues] ([milestone]); CREATE INDEX [idx_issues_assignee] ON [issues] ([assignee]); CREATE INDEX [idx_issues_user] ON [issues] ([user]);