issues
1,991 rows where user = 9599 sorted by reactions
This data as json, CSV (advanced)
Suggested facets: state, author_association, repo, type, draft, state_reason, created_at (date), updated_at (date), closed_at (date)
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions ▼ | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1616347574 | I_kwDOJHON9s5gV4G2 | 1 | Initial proof of concept with ChatGPT | simonw 9599 | closed | 0 | 3 | 2023-03-09T03:44:39Z | 2023-03-09T03:51:55Z | 2023-03-09T03:51:55Z | MEMBER | I'm using ChatGPT to figure out enough AppleScript to get at my notes data. | apple-notes-to-sqlite 611552758 | issue | {"url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
1617962395 | I_kwDOJHON9s5gcCWb | 10 | Include schema in README | simonw 9599 | closed | 0 | 0 | 2023-03-09T20:38:59Z | 2023-03-09T20:48:18Z | 2023-03-09T20:48:18Z | MEMBER | As seen in other tools like https://github.com/simonw/git-history | apple-notes-to-sqlite 611552758 | issue | {"url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
1618130434 | I_kwDOJHON9s5gcrYC | 11 | Implement a SQL view to make it easier to query files in a nested folder | simonw 9599 | open | 0 | 3 | 2023-03-09T23:19:28Z | 2023-03-09T23:24:01Z | MEMBER | Working with nested data in SQL is tricky, can I make it easier with a view or canned query? | apple-notes-to-sqlite 611552758 | issue | {"url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
1616354999 | I_kwDOJHON9s5gV563 | 2 | First working version | simonw 9599 | closed | 0 | 7 | 2023-03-09T03:53:00Z | 2023-03-09T05:10:22Z | 2023-03-09T05:10:22Z | MEMBER | It's going to shell out to `osascript` as seen in: - #1 I'm going with that option because https://appscript.sourceforge.io/status.html warns against the other potential methods: > Apple eliminated its Mac Automation department in 2016. The future of AppleScript and its related technologies is unclear. Caveat emptor. But `osascript` looks pretty stable to me. | apple-notes-to-sqlite 611552758 | issue | {"url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
1616422013 | I_kwDOJHON9s5gWKR9 | 3 | `apple-notes-to-sqlite --dump` option | simonw 9599 | closed | 0 | 0 | 2023-03-09T05:05:49Z | 2023-03-09T05:06:14Z | 2023-03-09T05:06:14Z | MEMBER | Option that doesn't write to the database at all, it just outputs all the notes to stdout as newline-delimited JSON. | apple-notes-to-sqlite 611552758 | issue | {"url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
1616429236 | I_kwDOJHON9s5gWMC0 | 4 | Support incremental updates | simonw 9599 | open | 0 | 2 | 2023-03-09T05:14:00Z | 2023-03-09T18:20:56Z | MEMBER | Running this script can take several hours against a large notes database. Would be neat if it could run against just the notes that have been modified since it last ran. Could pull the max `updated` date and then keep on looping until it finds one modified before then. Problem is I don't actually know what order it iterates over the notes in. | apple-notes-to-sqlite 611552758 | issue | {"url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
1616440856 | I_kwDOJHON9s5gWO4Y | 5 | Configure full text search | simonw 9599 | open | 0 | 0 | 2023-03-09T05:20:46Z | 2023-03-09T05:20:46Z | MEMBER | FTS would be useful. Maybe even extract the plain text from the notes to make that index easier to create, rather than creating it against the HTML. Can use the `plaintext` property for that. | apple-notes-to-sqlite 611552758 | issue | {"url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
1617602868 | I_kwDOJHON9s5gaqk0 | 6 | Character encoding problem | simonw 9599 | open | 0 | 2 | 2023-03-09T16:44:34Z | 2023-04-14T15:22:09Z | MEMBER | I ran against a recent note with this in it: > Or just "Actions ⚙️ " And got back: <img width="367" alt="image" src="https://user-images.githubusercontent.com/9599/224092586-e2475d8f-868c-489d-b495-2bfeb1a7db99.png"> > `Actions ‚öôÔ∏è` Pasting that into https://ftfy.vercel.app/?s=Actions+%E2%80%9A%C3%B6%C3%B4%C3%94%E2%88%8F%C3%A8+ gives this: ```python s = 'Actions â\x80\x9aöôÃ\x94â\x88\x8fè' s = s.encode('latin-1') s = s.decode('utf-8') s = s.encode('macroman') s = s.decode('utf-8') print(s) ``` <img width="816" alt="image" src="https://user-images.githubusercontent.com/9599/224092864-2b796c40-7c68-40b1-ae38-d0c61721b742.png"> | apple-notes-to-sqlite 611552758 | issue | {"url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
1617769847 | I_kwDOJHON9s5gbTV3 | 7 | Folder support | simonw 9599 | closed | 0 | 6 | 2023-03-09T18:21:33Z | 2023-03-09T20:48:18Z | 2023-03-09T20:48:18Z | MEMBER | Notes can live in folders. These relationships should be exported too. | apple-notes-to-sqlite 611552758 | issue | {"url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
1617938730 | I_kwDOJHON9s5gb8kq | 9 | Default to just storing plaintext, store HTML if `--html` is passed | simonw 9599 | open | 0 | 0 | 2023-03-09T20:19:06Z | 2023-03-09T20:19:06Z | MEMBER | The full `body` version of the notes can get HUGE, due to embedded images. It turns out for my own purposes I'm usually happy with just the `plaintext` version. I'm tempted to say you don't get HTML unless you pass a `--html` option. | apple-notes-to-sqlite 611552758 | issue | {"url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
689800307 | MDU6SXNzdWU2ODk4MDAzMDc= | 1 | Add an index on the timestamp column | simonw 9599 | closed | 0 | 0 | 2020-09-01T04:33:37Z | 2020-09-01T04:49:23Z | 2020-09-01T04:49:23Z | MEMBER | Since default view will likely be ordered by timestamp descending. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
691557547 | MDU6SXNzdWU2OTE1NTc1NDc= | 10 | Category 3: received | simonw 9599 | closed | 0 | 1 | 2020-09-03T01:40:36Z | 2020-09-03T17:38:51Z | 2020-09-03T17:38:51Z | MEMBER | A category for things that were sent to me: DMs, emails etc. Follows #7. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
692125110 | MDU6SXNzdWU2OTIxMjUxMTA= | 11 | Public / Private mechanism | simonw 9599 | closed | 0 | 1 | 2020-09-03T16:47:03Z | 2020-09-03T17:33:52Z | 2020-09-03T17:33:52Z | MEMBER | Some of the data in Dogsheep is stuff that was written publicly - tweets, blog posts, GitHub commits to public repos. Some of it is private data - emails, photos, direct messages, Swarm checkins, commits to private repos. Being able to filter for just one or the other (or both) would be useful. Especially when giving demos! | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/11/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
692202408 | MDU6SXNzdWU2OTIyMDI0MDg= | 12 | Idea: maps and GeoJSON support | simonw 9599 | open | 0 | 0 | 2020-09-03T18:47:10Z | 2020-09-04T01:45:03Z | MEMBER | It would be cool if the `display_sql` could return a column populated with GeoJSON which would the automatically be displayed on a map in the results (or maybe default JS would look for a `class="geojson"` element output by the `display` template) - ala https://github.com/simonw/datasette-leaflet-geojson Then I could render workout routes on a map, or Swarm checkin points. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
692386625 | MDU6SXNzdWU2OTIzODY2MjU= | 13 | Support advanced FTS queries | simonw 9599 | closed | 0 | 1 | 2020-09-03T21:29:56Z | 2020-09-03T21:40:51Z | 2020-09-03T21:40:51Z | MEMBER | `simon willison NOT screenshot` for example. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
693318095 | MDU6SXNzdWU2OTMzMTgwOTU= | 14 | On FTS exception rerun the query with quoting | simonw 9599 | closed | 0 | 0 | 2020-09-04T15:44:18Z | 2020-09-05T16:23:01Z | 2020-09-05T16:23:01Z | MEMBER | Searching for eg `#dogfest` currently throws an FTS exception - but I want to support advanced FTS query tricks as seen in #13. https://dogsheep.simonwillison.net/-/beta?q=%23dogfest > fts5: syntax error near "#" Idea: catch that error and re-run the query with FTS escaping applied! | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/14/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
694136490 | MDU6SXNzdWU2OTQxMzY0OTA= | 15 | Add a bunch of config examples | simonw 9599 | open | 0 | 1 | 2020-09-05T17:58:43Z | 2020-09-18T23:17:39Z | MEMBER | I can bring these over from my personal Dogsheep. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/15/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
694493566 | MDU6SXNzdWU2OTQ0OTM1NjY= | 16 | Timeline view | simonw 9599 | open | 0 | 3 | 2020-09-06T19:13:58Z | 2020-09-21T02:42:29Z | MEMBER | Ability to browse (and facet) by date. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
694500679 | MDU6SXNzdWU2OTQ1MDA2Nzk= | 17 | Rename "table" to "type" | simonw 9599 | closed | 0 | 2 | 2020-09-06T19:34:41Z | 2020-09-09T03:03:22Z | 2020-09-09T03:03:22Z | MEMBER | I think "table" is the wrong name for the concept I'm using it for here. Two reasons: firstly, `table` is a reserved word in SQLite. More importantly, it turns out there's not a direct mapping from tables to types of search result. In particular, for GitHub I ended up having two different "tables" of repositories - one for repos created by me, another for repos that I have starred. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
695553522 | MDU6SXNzdWU2OTU1NTM1MjI= | 18 | Deleted records stay in the search index | simonw 9599 | open | 0 | 2 | 2020-09-08T05:14:23Z | 2020-09-08T05:15:51Z | MEMBER | Here's why: https://github.com/dogsheep/dogsheep-beta/blob/24f7898d41a39218058f174c75ba62f7c0fcfff6/dogsheep_beta/utils.py#L44-L53 That should probably do `DELETE FROM index1.search_index WHERE [table] = ?` first. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
695556681 | MDU6SXNzdWU2OTU1NTY2ODE= | 19 | Figure out incremental re-indexing | simonw 9599 | open | 0 | 2 | 2020-09-08T05:23:31Z | 2020-09-08T05:27:07Z | MEMBER | As tables get bigger reindexing everything on a schedule (essentially recreating the entire index from scratch) will start to become a performance bottleneck. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
689809225 | MDU6SXNzdWU2ODk4MDkyMjU= | 2 | Apply porter stemming | simonw 9599 | closed | 0 | 2 | 2020-09-01T04:57:55Z | 2020-09-01T20:42:00Z | 2020-09-01T20:40:24Z | MEMBER | This can be on by default. You can turn it off for a table in the config file using `stemming: none` - or maybe `tokenize: none` to match the terminology used by SQLite and `sqlite-utils`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#enabling-full-text-search | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
703951918 | MDU6SXNzdWU3MDM5NTE5MTg= | 21 | Option to sort search results by date | simonw 9599 | closed | 0 | 0 | 2020-09-17T22:32:39Z | 2020-09-17T22:55:35Z | 2020-09-17T22:55:35Z | MEMBER | Sometimes I want to sort by date, not by relevance. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/21/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
703962917 | MDU6SXNzdWU3MDM5NjI5MTc= | 22 | Bug: UI says sorted by relevance in timeline view | simonw 9599 | closed | 0 | 0 | 2020-09-17T23:02:07Z | 2020-09-17T23:13:14Z | 2020-09-17T23:13:14Z | MEMBER | In regular timeline view sort defaults to newest, not relevance - so this UI is incorrect: <img width="837" alt="Dogsheep_Beta_and_The_latest_news_from_the_Datasette_ecosystem_of_tools_-_Datasette_Newsletter_and_Ability_to_invite_people_to_a_team_who_don_t_have_acounts_yet_·_Issue__307_·_simonw_datasettecloud" src="https://user-images.githubusercontent.com/9599/93536956-1facf900-f8ff-11ea-889b-bc8356e366df.png"> | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/22/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
703970713 | MDU6SXNzdWU3MDM5NzA3MTM= | 23 | Sort option should persist between multiple searches | simonw 9599 | closed | 0 | 0 | 2020-09-17T23:21:26Z | 2020-09-18T22:39:12Z | 2020-09-18T22:39:12Z | MEMBER | Following #21 | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/23/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
703970814 | MDU6SXNzdWU3MDM5NzA4MTQ= | 24 | the JSON object must be str, bytes or bytearray, not 'Undefined' | simonw 9599 | closed | 0 | 8 | 2020-09-17T23:21:41Z | 2020-09-18T22:33:32Z | 2020-09-18T22:33:32Z | MEMBER | Got this on a search results page. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
704685890 | MDU6SXNzdWU3MDQ2ODU4OTA= | 25 | template_debug mechanism | simonw 9599 | closed | 0 | 2 | 2020-09-18T22:11:09Z | 2020-09-18T22:12:21Z | 2020-09-18T22:12:03Z | MEMBER | > I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible. _Originally posted by @simonw in https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584_ | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
705215230 | MDU6SXNzdWU3MDUyMTUyMzA= | 26 | Pagination | simonw 9599 | open | 0 | 7 | 2020-09-21T00:14:37Z | 2020-09-21T02:55:54Z | MEMBER | Useful for #16 (timeline view) since you can now filter to just the items on a specific day - but if there are more than 50 items you can't see them all. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
709789634 | MDU6SXNzdWU3MDk3ODk2MzQ= | 27 | Sort order is not persisted by facet filter links | simonw 9599 | open | 0 | 0 | 2020-09-27T18:22:07Z | 2020-09-27T18:22:07Z | MEMBER | A link to `/-/beta?category=1×tamp__date=2018-08-01&q=swedish` should be to `/-/beta?category=1×tamp__date=2018-08-01&q=swedish&sort=newest` | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/27/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
723861683 | MDU6SXNzdWU3MjM4NjE2ODM= | 28 | Switch to using datasette.client | simonw 9599 | closed | 0 | 1 | 2020-10-17T22:42:26Z | 2020-10-17T23:00:47Z | 2020-10-17T23:00:47Z | MEMBER | `datasette.client` is designed for this kind of thing, to replace this code: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L223-L232 | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/28/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
724759588 | MDU6SXNzdWU3MjQ3NTk1ODg= | 29 | Add search highlighting snippets | simonw 9599 | open | 0 | 5 | 2020-10-19T16:00:48Z | 2021-08-26T20:23:11Z | MEMBER | Like on https://til.simonwillison.net/til/search?q=Snippet | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29/reactions", "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0} | ||||||||
689810340 | MDU6SXNzdWU2ODk4MTAzNDA= | 3 | Datasette plugin to provide custom page for running faceted, ranked searches | simonw 9599 | closed | 0 | 3 | 2020-09-01T05:00:22Z | 2020-09-03T21:01:41Z | 2020-09-03T21:01:41Z | MEMBER | This will be a page at `/-/beta` which renders using a custom template. It will offer a default timeline view plus search and facet by type/date. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
769282206 | MDU6SXNzdWU3NjkyODIyMDY= | 30 | Upgrade to sqlite-utils 3.0 (tests are failing) | simonw 9599 | closed | 0 | 0 | 2020-12-16T21:25:15Z | 2020-12-16T21:27:11Z | 2020-12-16T21:27:10Z | MEMBER | ``` results = beta_db["search_index"].search("run") if use_porter: assert results == [ ( "dogs.db/dogs", "1", "Cleo", "2020-08-22 04:41:33", 1, 0, "running", None, None, ) ] else: > assert results == [] E assert <generator ob...x7f3c080a3e08> == [] E +<generator object Table.search at 0x7f3c080a3e08> E -[] E Full diff: E - [] E + <generator object Table.search at 0x7f3c080a3e08> ``` This was caused by a backwards incompatible change in sqlite-utils 3.0: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v3-0 | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/30/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
771316301 | MDU6SXNzdWU3NzEzMTYzMDE= | 31 | Searching for "github-to-sqlite" throws an error | simonw 9599 | closed | 0 | 4 | 2020-12-19T06:07:20Z | 2020-12-19T06:18:07Z | 2020-12-19T06:18:07Z | MEMBER | https://datasette.io/-/beta?q=github-to-sqlite&sort=relevance&type=blog.db%2Fentries - "no such column: to" | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
836923194 | MDU6SXNzdWU4MzY5MjMxOTQ= | 32 | JSON API for search results | simonw 9599 | open | 0 | 0 | 2021-03-20T22:21:36Z | 2021-03-20T22:21:36Z | MEMBER | Refs https://github.com/simonw/datasette/issues/878 | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/32/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
919733213 | MDU6SXNzdWU5MTk3MzMyMTM= | 33 | Searching for whitespace throws an error | simonw 9599 | closed | 0 | 0 | 2021-06-13T06:57:57Z | 2021-06-13T14:36:39Z | 2021-06-13T14:36:39Z | MEMBER | https://datasette.io/-/beta?q=+ returns a 500 > fts5: syntax error near "" | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/33/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
689839399 | MDU6SXNzdWU2ODk4MzkzOTk= | 4 | Optimize the FTS table | simonw 9599 | closed | 0 | 1 | 2020-09-01T05:58:17Z | 2020-09-01T06:10:08Z | 2020-09-01T06:10:08Z | MEMBER | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | |||||||
689847361 | MDU6SXNzdWU2ODk4NDczNjE= | 5 | Add a context column that's not searchable | simonw 9599 | closed | 0 | 1 | 2020-09-01T06:13:42Z | 2020-09-03T18:43:50Z | 2020-09-03T18:43:50Z | MEMBER | I sometimes like to configure titles that are things like "Comment on issue X" or "Photo in Golden Gate Park" - these shouldn't be included in the search index but should be stored so they can be displayed to provide context. Add a column for this - probably called `context` - and make it so it can be populated. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
689850810 | MDU6SXNzdWU2ODk4NTA4MTA= | 6 | Set up a demo instance | simonw 9599 | open | 0 | 0 | 2020-09-01T06:20:24Z | 2020-09-01T06:20:24Z | MEMBER | Once I've got the Datasette plugin to a state where it's worth building a demo: #3 I can use data from my public https://github-to-sqlite.dogsheep.net/ demo plus the Pocket data subset I use for the demo in https://github.com/dogsheep/pocket-to-sqlite/issues/5 - I could pull in the https://dogsheep-photos.dogsheep.net/ photos data too. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
691265198 | MDU6SXNzdWU2OTEyNjUxOTg= | 7 | Mechanism for differentiating between "by me" and "liked by me" | simonw 9599 | closed | 0 | 6 | 2020-09-02T17:44:37Z | 2020-09-02T21:06:28Z | 2020-09-02T21:06:28Z | MEMBER | Some of the content I'm indexing is by me - photos I've taken, tweets I wrote, commits, comments I posted. Some of it is stuff that I've "liked" or "bookmarked" in some way - favourited tweets, Pocket articles, starred GitHub repos. It woud be useful to be able to differentiate between the two. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
691369691 | MDU6SXNzdWU2OTEzNjk2OTE= | 8 | Create a view for running faceted searches | simonw 9599 | closed | 0 | 1 | 2020-09-02T19:44:07Z | 2020-09-02T19:50:47Z | 2020-09-02T19:50:47Z | MEMBER | ```sql select search_index_fts.rank, search_index.rowid, search_index.[table], search_index.key, search_index.title, search_index.timestamp, search_index.search_1 from search_index join search_index_fts on search_index.rowid = search_index_fts.rowid order by search_index_fts.rank, search_index.timestamp desc ``` | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
691521965 | MDU6SXNzdWU2OTE1MjE5NjU= | 9 | Mechanism for defining custom display of results | simonw 9599 | closed | 0 | 8 | 2020-09-03T00:14:07Z | 2020-09-03T21:12:14Z | 2020-09-03T21:09:55Z | MEMBER | Part of #3 - in particular I want to make sure my photos are displayed with a thumbnail. | dogsheep-beta 197431109 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
602533300 | MDU6SXNzdWU2MDI1MzMzMDA= | 1 | Import photo metadata from Apple Photos into SQLite | simonw 9599 | open | 0 | Apple Photos online and securely browsable 5324096 | 8 | 2020-04-18T19:23:26Z | 2020-05-04T02:41:40Z | MEMBER | Faces, albums, locations, that kind of thing. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | |||||||
606028272 | MDU6SXNzdWU2MDYwMjgyNzI= | 10 | Speed up hashing step using threads | simonw 9599 | closed | 0 | 0 | 2020-04-24T04:20:08Z | 2020-04-24T04:32:35Z | 2020-04-24T04:32:35Z | MEMBER | This TODO from the code: https://github.com/dogsheep/photos-to-sqlite/blob/2e7f2c67cc18b02c75bb64992a05b0196e507252/photos_to_sqlite/cli.py#L82-L90 | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
606032950 | MDU6SXNzdWU2MDYwMzI5NTA= | 11 | Try running S3 uploads in a thread pool | simonw 9599 | closed | 0 | 0 | 2020-04-24T04:34:31Z | 2020-04-24T16:45:41Z | 2020-04-24T16:45:41Z | MEMBER | Since #10 provided such a speedup, can the same thing be done for the actual uploads? http://ls.pwd.io/2013/06/parallel-s3-uploads-using-boto-and-threads-in-python/ suggests it can really help performance. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/11/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
606033104 | MDU6SXNzdWU2MDYwMzMxMDQ= | 12 | If less than 500MB, show size in MB not GB | simonw 9599 | open | 0 | 1 | 2020-04-24T04:35:01Z | 2020-04-24T04:35:25Z | MEMBER | Just saw this: ``` Uploading 0.05 GB ``` | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
607888367 | MDU6SXNzdWU2MDc4ODgzNjc= | 13 | Also upload movie files | simonw 9599 | open | 0 | 2 | 2020-04-27T22:11:25Z | 2020-04-28T00:39:45Z | MEMBER | The `upload` command currently only handles static images: https://github.com/dogsheep/photos-to-sqlite/blob/d939455af00e07866686457ee2fcb9b2d1b7194e/photos_to_sqlite/utils.py#L26-L33 Need to cover movies taken by my phone and DSLR too. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
608512747 | MDU6SXNzdWU2MDg1MTI3NDc= | 14 | Annotate photos using the Google Cloud Vision API | simonw 9599 | open | 0 | 5 | 2020-04-28T18:09:03Z | 2020-04-28T18:19:06Z | MEMBER | It can detect faces, run OCR, do image labeling (it knows what a lemur is!) and do object localization where it identifies objects and returns bounding polygons for them. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14/reactions", "total_count": 3, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0} | ||||||||
612151767 | MDU6SXNzdWU2MTIxNTE3Njc= | 15 | Expose scores from ZCOMPUTEDASSETATTRIBUTES | simonw 9599 | closed | 0 | 7 | 2020-05-04T20:36:07Z | 2020-12-20T04:44:22Z | 2020-05-05T00:11:45Z | MEMBER | The Apple Photos database has a `ZCOMPUTEDASSETATTRIBUTES` that looks absurdly interesting... it has calculated scores for every photo: <img width="273" alt="Photos__ZCOMPUTEDASSETATTRIBUTES" src="https://user-images.githubusercontent.com/9599/81011044-1f0e3d00-8e0c-11ea-84b6-f302b09e7bc9.png"> | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
612287234 | MDU6SXNzdWU2MTIyODcyMzQ= | 16 | Import machine-learning detected labels (dog, llama etc) from Apple Photos | simonw 9599 | open | 0 | 13 | 2020-05-05T02:45:43Z | 2020-05-05T05:38:16Z | MEMBER | Follow-on from #1. Apple Photos runs some very sophisticated machine learning on-device to figure out if photos are of dogs, llamas and so on. I really want to extract those labels out into my own database. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16/reactions", "total_count": 2, "+1": 0, "-1": 0, "laugh": 1, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
612860531 | MDU6SXNzdWU2MTI4NjA1MzE= | 17 | Only install osxphotos if running on macOS | simonw 9599 | closed | 0 | 3 | 2020-05-05T20:03:26Z | 2020-05-05T20:20:05Z | 2020-05-05T20:11:23Z | MEMBER | The build is broken right now because you can't `pip install osxphotos` on Ubuntu. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
612860758 | MDU6SXNzdWU2MTI4NjA3NTg= | 18 | Switch CI solution to GitHub Actions with a macOS runner | simonw 9599 | open | 0 | 1 | 2020-05-05T20:03:50Z | 2020-05-05T23:49:18Z | MEMBER | Refs #17. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
613002220 | MDU6SXNzdWU2MTMwMDIyMjA= | 19 | apple-photos command should work even if upload has not run | simonw 9599 | closed | 0 | 1 | 2020-05-06T02:02:25Z | 2020-05-19T20:59:59Z | 2020-05-19T20:59:59Z | MEMBER | I want people to be able to query their Apple Photos metadata without having to first run `upload` to upload all of their files to their own S3 bucket. To do this I can have `apple-photos` calculate SHA256 hashes of each photo if the `uploads` table does not yet exist (or does not contain that photo). | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/19/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
602533352 | MDU6SXNzdWU2MDI1MzMzNTI= | 2 | Ability to convert HEIC images to JPEG | simonw 9599 | closed | 0 | Apple Photos online and securely browsable 5324096 | 1 | 2020-04-18T19:23:43Z | 2020-04-28T16:47:21Z | 2020-04-28T16:47:21Z | MEMBER | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
613006393 | MDU6SXNzdWU2MTMwMDYzOTM= | 20 | Ability to serve thumbnailed Apple Photo from its place on disk | simonw 9599 | closed | 0 | 10 | 2020-05-06T02:17:50Z | 2020-05-25T20:14:22Z | 2020-05-25T20:09:41Z | MEMBER | A custom Datasette plugin that can be run locally on a Mac laptop which knows how to serve photos such that they can be seen in the browser. _Originally posted by @simonw in https://github.com/dogsheep/photos-to-sqlite/issues/19#issuecomment-624406285_ | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
615474990 | MDU6SXNzdWU2MTU0NzQ5OTA= | 21 | bpylist.archiver.CircularReference: archive has a cycle with uid(13) | simonw 9599 | closed | 0 | 11 | 2020-05-10T20:58:06Z | 2020-12-19T07:44:49Z | 2020-05-10T21:57:13Z | MEMBER | ``` % python -i $(which photos-to-sqlite) apple-photos photos.db Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py", line 611, in place return self._place # pylint: disable=access-member-before-definition AttributeError: 'PhotoInfo' object has no attribute '_place' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/bin/photos-to-sqlite", line 11, in <module> load_entry_point('photos-to-sqlite', 'console_scripts', 'photos-to-sqlite')() File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/cli.py", line 249, in apple_photos photo_row = osxphoto_to_row(sha256, photo) File "/Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/utils.py", line 91, in osxphoto_to_row place = photo.place File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py", line … | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
615626118 | MDU6SXNzdWU2MTU2MjYxMTg= | 22 | Try out ExifReader | simonw 9599 | open | 0 | 4 | 2020-05-11T06:32:13Z | 2020-05-14T05:59:53Z | MEMBER | https://pypi.org/project/ExifReader/ New fork that should be able to handle EXIF in HEIC files. Forked here: https://github.com/ianare/exif-py/issues/102#issuecomment-626376522 Refs #3 | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
621280529 | MDU6SXNzdWU2MjEyODA1Mjk= | 23 | create-subset command for creating a publishable subset of a photos database | simonw 9599 | closed | 0 | 1 | 2020-05-19T20:58:20Z | 2020-05-19T22:32:48Z | 2020-05-19T22:32:37Z | MEMBER | I want to share a subset of my photos, without sharing everything. Idea: $ photos-to-sqlite create-subset photos.db public.db "select sha256 from ... where ..." So the command takes a SQL query that returns sha256 hashes, then creates a new file called `public.db` containing just the data corresponding to those photos. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/23/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
621323348 | MDU6SXNzdWU2MjEzMjMzNDg= | 24 | Configurable URL for images | simonw 9599 | open | 0 | 1 | 2020-05-19T22:25:56Z | 2020-05-20T06:00:29Z | MEMBER | This is hard-coded at the moment, which is bad: https://github.com/dogsheep/photos-to-sqlite/blob/d5d69b9019703c47bc251444838578dd752801e2/photos_to_sqlite/cli.py#L269-L272 | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/24/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
621332242 | MDU6SXNzdWU2MjEzMzIyNDI= | 25 | Create a public demo | simonw 9599 | closed | 0 | 5 | 2020-05-19T22:47:20Z | 2020-05-21T22:26:16Z | 2020-05-20T05:54:18Z | MEMBER | So I can show people what this does, using some of my photos. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
621444763 | MDU6SXNzdWU2MjE0NDQ3NjM= | 26 | Rename project to dogsheep-photos | simonw 9599 | closed | 0 | 8 | 2020-05-20T04:12:34Z | 2020-05-20T04:31:02Z | 2020-05-20T04:30:40Z | MEMBER | `photos-to-sqlite` doesn't really capture the full scope of this project anymore. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
621486115 | MDU6SXNzdWU2MjE0ODYxMTU= | 27 | photos_with_apple_metadata view should include labels | simonw 9599 | open | 0 | 0 | 2020-05-20T06:06:17Z | 2020-05-20T06:06:17Z | MEMBER | https://dogsheep-photos.dogsheep.net/public/photos_with_apple_metadata?place_city=New+Orleans&_facet=place_city&_facet_array=albums&_facet_array=persons Here's one way to add that: ```sql select rowid, photo, ( select json_group_array( json_object( 'label', normalized_string, 'href', '/photos/labelled?_hide_sql=1&label=' || normalized_string ) ) from labels where labels.uuid = photos_with_apple_metadata.uuid ) as labels, date, ``` | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/27/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
602533481 | MDU6SXNzdWU2MDI1MzM0ODE= | 3 | Import EXIF data into SQLite - lens used, ISO, aperture etc | simonw 9599 | open | 0 | Apple Photos online and securely browsable 5324096 | 2 | 2020-04-18T19:24:31Z | 2021-10-05T12:38:24Z | MEMBER | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
602533539 | MDU6SXNzdWU2MDI1MzM1Mzk= | 4 | Upload all my photos to a secure S3 bucket | simonw 9599 | closed | 0 | Apple Photos online and securely browsable 5324096 | 14 | 2020-04-18T19:24:50Z | 2020-04-18T21:58:11Z | 2020-04-18T21:57:13Z | MEMBER | - [x] Create a bucket with bucket credentials - [x] Programmatically upload some recent photos to it (from a notebook) - [x] Turn this into a script | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | |||||
602551638 | MDU6SXNzdWU2MDI1NTE2Mzg= | 5 | photos-to-sqlite s3-auth command | simonw 9599 | closed | 0 | 1 | 2020-04-18T21:05:25Z | 2020-04-18T21:08:44Z | 2020-04-18T21:08:44Z | MEMBER | Modeled on `github-to-sqlite auth` - prompts the user for their S3 credentials and saves them to `auth.json`. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
602575575 | MDU6SXNzdWU2MDI1NzU1NzU= | 6 | Add progress bar to upload command | simonw 9599 | closed | 0 | 2 | 2020-04-18T23:32:41Z | 2020-04-19T00:15:24Z | 2020-04-19T00:15:24Z | MEMBER | Upload was added in #4 | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
602585497 | MDU6SXNzdWU2MDI1ODU0OTc= | 7 | Integrate image content hashing | simonw 9599 | open | 0 | 2 | 2020-04-19T00:36:58Z | 2021-08-26T02:01:01Z | MEMBER | To spot duplicate images (where the file content differs such that the sha256 is no longer a match) it would be useful to calculate and store perceptual hashes of some sort. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7/reactions", "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0} | ||||||||
605147638 | MDU6SXNzdWU2MDUxNDc2Mzg= | 8 | Should I have used MD5 instead of SHA256? | simonw 9599 | closed | 0 | 2 | 2020-04-23T00:02:08Z | 2020-04-23T00:03:35Z | 2020-04-23T00:03:35Z | MEMBER | https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html > Objects created by the PUT Object, POST Object, or Copy operation, or through the AWS Management Console, and are encrypted by SSE-S3 or plaintext, have ETags that are an MD5 digest of their object data. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
605938063 | MDU6SXNzdWU2MDU5MzgwNjM= | 9 | upload command should be resumable, should only upload photos not already uploaded | simonw 9599 | closed | 0 | 2 | 2020-04-23T23:31:08Z | 2020-04-23T23:39:14Z | 2020-04-23T23:39:14Z | MEMBER | Follow on from #4. | dogsheep-photos 256834907 | issue | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
718934942 | MDU6SXNzdWU3MTg5MzQ5NDI= | 1 | Documentation on how to use this with Datasette | simonw 9599 | open | 0 | 1 | 2020-10-11T21:56:27Z | 2020-10-11T22:14:00Z | MEMBER | In particular how to use `datasette-render-images` to see the images. | evernote-to-sqlite 303218369 | issue | {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
978743426 | MDU6SXNzdWU5Nzg3NDM0MjY= | 13 | xml.etree.ElementTree.ParseError: not well-formed (invalid token) | simonw 9599 | closed | 0 | 4 | 2021-08-25T05:48:21Z | 2021-08-26T18:45:13Z | 2021-08-26T18:45:13Z | MEMBER | Got this error today: ``` (evernote-to-sqlite) /tmp % evernote-to-sqlite enex evernote.db simonwillison\'s\ notebook.enex Importing from ENEX [######------------------------------] 17% Traceback (most recent call last): File "/Users/simon/.local/bin/evernote-to-sqlite", line 8, in <module> sys.exit(cli()) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 763, in invoke return __callback(*args, **kwargs) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/evernote_to_sqlite/cli.py", line 31, in enex save_note(db, note) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/evernote_to_sqlite/utils.py", line 36, in save_note content = ET.tostring(ET.fromstring(content_xml)).decode("utf-8") File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py", line 1347, in XML parser.feed(text) xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 2, column 132 ``` | evernote-to-sqlite 303218369 | issue | {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
718938046 | MDU6SXNzdWU3MTg5MzgwNDY= | 2 | Convert dates to a better format | simonw 9599 | closed | 0 | 0 | 2020-10-11T22:12:33Z | 2020-10-11T23:15:03Z | 2020-10-11T23:15:03Z | MEMBER | They currently look like this: https://github.com/dogsheep/evernote-to-sqlite/blob/9d8efd17580f6ddf76745c145d1e69dd24e52b64/tests/test_evernote_to_sqlite.py#L35-L36 | evernote-to-sqlite 303218369 | issue | {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
718938321 | MDU6SXNzdWU3MTg5MzgzMjE= | 3 | Use a content hash for the note IDs | simonw 9599 | closed | 0 | 0 | 2020-10-11T22:13:46Z | 2020-10-11T23:15:04Z | 2020-10-11T23:15:04Z | MEMBER | Without a GUID note IDs are pretty ineffective, but using a hash of the contents will at least avoid creating identical duplicates in the future. https://sqlite-utils.readthedocs.io/en/stable/python-api.html#setting-an-id-based-on-the-hash-of-the-row-contents | evernote-to-sqlite 303218369 | issue | {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
718938508 | MDU6SXNzdWU3MTg5Mzg1MDg= | 4 | Configure FTS + add an index on the date columns | simonw 9599 | closed | 0 | 2 | 2020-10-11T22:14:40Z | 2020-10-11T23:41:29Z | 2020-10-11T23:41:29Z | MEMBER | Sort by date descending is likely the most common way of sorting, so that column should be indexed. Also add FTS configuration for both notes and the OCR column on resources. | evernote-to-sqlite 303218369 | issue | {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
718938889 | MDU6SXNzdWU3MTg5Mzg4ODk= | 5 | Figure out how to display images from <en-media> tags inline in Datasette | simonw 9599 | open | 0 | 6 | 2020-10-11T22:17:03Z | 2020-10-16T20:16:28Z | MEMBER | Relates to #1. Evernote XML looks like this: ```xml <?xml version="1.0"?> <en-note> <div>This note includes two images.</div> <div> <b>The Python logo</b> </div> <div> <en-media hash="61098c2c541de7f0a907c301dd6542da" type="image/svg+xml" width="125"/> </div> <div> <b>The Evernote logo</b> </div> <div> <en-media hash="91bd26175acac0b2ffdb6efac199f8ca" type="image/svg+xml" width="125"/> </div> </en-note> ``` That hash is the md5 we use to store resources. It should be possible to turn these into embedded image tags, especially if done in conjunction with the https://github.com/simonw/datasette-media plugin. | evernote-to-sqlite 303218369 | issue | {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
718949182 | MDU6SXNzdWU3MTg5NDkxODI= | 6 | Better handling of OCR data | simonw 9599 | closed | 0 | 2 | 2020-10-11T23:20:52Z | 2020-10-12T00:04:10Z | 2020-10-12T00:04:10Z | MEMBER | > I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought. _Originally posted by @simonw in https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706784028_ | evernote-to-sqlite 303218369 | issue | {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
496415321 | MDU6SXNzdWU0OTY0MTUzMjE= | 1 | Figure out some interesting example SQL queries | simonw 9599 | open | 0 | 9 | 2019-09-20T15:28:07Z | 2021-05-03T03:46:23Z | MEMBER | My knowledge of genetics has left me short here. I'd love to be able to provide some interesting example SELECT queries - maybe one that spots if you are [likely to have red hair?](https://www.snpedia.com/index.php/Rs1805007) | genome-to-sqlite 209590345 | issue | {"url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
493599818 | MDU6SXNzdWU0OTM1OTk4MTg= | 1 | Command for fetching starred repos | simonw 9599 | closed | 0 | 0 | 2019-09-14T08:36:29Z | 2019-09-14T21:30:48Z | 2019-09-14T21:30:48Z | MEMBER | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | |||||||
516967682 | MDU6SXNzdWU1MTY5Njc2ODI= | 10 | Add this repos_starred view | simonw 9599 | closed | 0 | 3 | 2019-11-04T05:44:38Z | 2020-05-02T16:37:36Z | 2020-05-02T16:37:36Z | MEMBER | ```sql create view repos_starred as select stars.starred_at, users.login, repos.* from repos join stars on repos.id = stars.repo join users on repos.owner = users.id order by starred_at desc; ``` | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
520521843 | MDU6SXNzdWU1MjA1MjE4NDM= | 11 | Command to fetch releases | simonw 9599 | closed | 0 | 0 | 2019-11-09T22:23:30Z | 2019-11-09T22:57:00Z | 2019-11-09T22:57:00Z | MEMBER | https://developer.github.com/v3/repos/releases/#list-releases-for-a-repository `GET /repos/:owner/:repo/releases` | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/11/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
520756546 | MDU6SXNzdWU1MjA3NTY1NDY= | 12 | Add this view for seeing new releases | simonw 9599 | closed | 0 | 5 | 2019-11-11T06:00:12Z | 2020-05-02T18:58:18Z | 2020-05-02T18:58:17Z | MEMBER | ```sql CREATE VIEW recent_releases AS select json_object("label", repos.full_name, "href", repos.html_url) as repo, json_object( "href", releases.html_url, "label", releases.name ) as release, substr(releases.published_at, 0, 11) as date, releases.body as body_markdown, releases.published_at from releases join repos on repos.id = releases.repo order by releases.published_at desc ``` | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
521275281 | MDU6SXNzdWU1MjEyNzUyODE= | 13 | Set up a live demo Datasette instance | simonw 9599 | closed | 0 | 1.0 5225818 | 9 | 2019-11-12T01:27:02Z | 2020-03-24T00:03:26Z | 2020-03-24T00:03:25Z | MEMBER | I deployed https://github-to-sqlite-releases-j7hipcg4aq-uc.a.run.app/ by running this: ``` #!/bin/bash # Fetch repos for simonw and dogsheep github-to-sqlite repos github.db simonw dogsheep -a auth.json # Fetch releases for the repos tagged 'datasette-io' sqlite-utils github.db " select full_name from repos where rowid in ( select repos.rowid from repos, json_each(repos.topics) j where j.value = 'datasette-io' )" --csv --no-headers | while read repo; do github-to-sqlite releases \ github.db $(echo $repo | tr -d '\r') \ -a auth.json; sleep 2; done; ``` And then deploying using this: ``` $ datasette publish cloudrun github.db \ --title "github-to-sqlite releases demo" \ --about_url="https://github.com/simonw/github-to-sqlite" \ --about='github-to-sqlite' \ --install=datasette-render-markdown \ --install=datasette-json-html \ --service=github-to-sqlite-releases ``` This should happen automatically for every release. I can run it once a day in Circle CI to keep the demo database up-to-date. | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | |||||
530491074 | MDU6SXNzdWU1MzA0OTEwNzQ= | 14 | Command for importing events | simonw 9599 | open | 0 | 3 | 2019-11-29T21:28:58Z | 2020-04-14T19:38:34Z | MEMBER | Eg from https://api.github.com/users/simonw/events Docs here: https://developer.github.com/v3/activity/events/#list-events-performed-by-a-user | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
578883725 | MDU6SXNzdWU1Nzg4ODM3MjU= | 17 | Command for importing commits | simonw 9599 | closed | 0 | 2 | 2020-03-10T21:55:12Z | 2020-03-11T02:47:37Z | 2020-03-11T02:47:37Z | MEMBER | Using this API: https://api.github.com/repos/dogsheep/github-to-sqlite/commits | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
585411547 | MDU6SXNzdWU1ODU0MTE1NDc= | 18 | Commits in GitHub API can have null author | simonw 9599 | closed | 0 | 1.0 5225818 | 8 | 2020-03-21T02:20:56Z | 2020-03-23T20:44:49Z | 2020-03-23T20:44:26Z | MEMBER | ``` Traceback (most recent call last): File "/home/ubuntu/datasette-venv/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/cli.py", line 235, in commits utils.save_commits(db, commits, repo_full["id"]) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 290, in save_commits commit_to_insert["author"] = save_user(db, commit["author"]) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 54, in save_user for key, value in user.items() AttributeError: 'NoneType' object has no attribute 'items' ``` Got this running the `commits` command from cron. | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | |||||
585850715 | MDU6SXNzdWU1ODU4NTA3MTU= | 19 | Enable full-text search for more stuff (like commits, issues and issue_comments) | simonw 9599 | closed | 0 | 1.0 5225818 | 2 | 2020-03-23T00:19:56Z | 2020-03-23T19:06:39Z | 2020-03-23T19:06:39Z | MEMBER | Currently FTS is only enabled for repos and releases. | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | |||||
493668862 | MDU6SXNzdWU0OTM2Njg4NjI= | 2 | Extract licenses from repos into a separate table | simonw 9599 | closed | 0 | 0 | 2019-09-14T21:33:41Z | 2019-09-14T21:46:58Z | 2019-09-14T21:46:58Z | MEMBER | <img width="588" alt="starred3__repos__800_rows" src="https://user-images.githubusercontent.com/9599/64913845-a2074980-d6fc-11e9-815a-f103ad0c6f56.png"> | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
586454513 | MDU6SXNzdWU1ODY0NTQ1MTM= | 20 | Upgrade to sqlite-utils 2.x | simonw 9599 | closed | 0 | 1.0 5225818 | 0 | 2020-03-23T19:17:58Z | 2020-03-23T19:22:52Z | 2020-03-23T19:22:52Z | MEMBER | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/20/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
586561727 | MDU6SXNzdWU1ODY1NjE3Mjc= | 21 | Turn GitHub API errors into exceptions | simonw 9599 | closed | 0 | 1.0 5225818 | 2 | 2020-03-23T22:37:24Z | 2020-03-23T23:48:23Z | 2020-03-23T23:48:22Z | MEMBER | This would have really helped in debugging the mess in #13. Running with this `auth.json` is a useful demo: ```json {"github_personal_token": ""} ``` | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | |||||
586567379 | MDU6SXNzdWU1ODY1NjczNzk= | 22 | Handle empty git repositories | simonw 9599 | closed | 0 | 0 | 2020-03-23T22:49:48Z | 2020-03-23T23:13:11Z | 2020-03-23T23:13:11Z | MEMBER | Got this error: ``` github_to_sqlite.utils.GitHubError: {'message': 'Git Repository is empty.', 'documentation_url': 'https://developer.github.com/v3/repos/commits/#list-commits-on-a-repository'} ``` From https://api.github.com/repos/dogsheep/beta/commits | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/22/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
586595839 | MDU6SXNzdWU1ODY1OTU4Mzk= | 23 | Release 1.0 | simonw 9599 | closed | 0 | 1.0 5225818 | 1 | 2020-03-24T00:03:55Z | 2020-03-24T00:15:50Z | 2020-03-24T00:15:50Z | MEMBER | Need to compile release notes. | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | |||||
599776345 | MDU6SXNzdWU1OTk3NzYzNDU= | 24 | Feature idea: github-to-sqlite everything ... | simonw 9599 | open | 0 | 0 | 2020-04-14T18:34:00Z | 2020-04-14T18:34:00Z | MEMBER | At the moment if you want to pull all your repos, issues, issues comments etc you have to do it with a sequence of separate commands. Consider adding a `everything` or `all` command which fetches everything that the tool knows how to fetch, and is designed to be run on a cron in a way that fetches just new stuff each time. | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/24/reactions", "total_count": 7, "+1": 7, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
601265023 | MDU6SXNzdWU2MDEyNjUwMjM= | 25 | Improvements to demo instance | simonw 9599 | closed | 0 | 1 | 2020-04-16T17:26:55Z | 2020-04-16T18:07:12Z | 2020-04-16T18:07:12Z | MEMBER | - [x] Demo should pull issue-comments as well | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
601271612 | MDU6SXNzdWU2MDEyNzE2MTI= | 26 | Topics are missing from repositories | simonw 9599 | closed | 0 | 2 | 2020-04-16T17:36:32Z | 2020-04-16T17:41:11Z | 2020-04-16T17:41:11Z | MEMBER | I'm sure this used to work, but right now repositories are fetched without their topics. https://developer.github.com/v3/repos/ says you need to send a custom `Accept` header of `application/vnd.github.mercy-preview+json` to get topics. | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
601330277 | MDU6SXNzdWU2MDEzMzAyNzc= | 27 | Repos have a big blob of JSON in the organization column | simonw 9599 | closed | 0 | 5 | 2020-04-16T18:43:14Z | 2020-04-18T00:19:16Z | 2020-04-18T00:18:52Z | MEMBER | e.g. https://github-to-sqlite.dogsheep.net/github/repos ![github__repos__11_rows_where_sorted_by_updated_at_descending](https://user-images.githubusercontent.com/9599/79494124-5640b980-7fd7-11ea-99a2-17ffbd82f9ce.png) This appears to be obsolete because the `owner` column already links to that record, albeit in the `users` table with `type` set to `Organization`: https://github-to-sqlite.dogsheep.net/github/users/53015001 | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
601333634 | MDU6SXNzdWU2MDEzMzM2MzQ= | 28 | Pull repository contributors | simonw 9599 | closed | 0 | 3 | 2020-04-16T18:46:40Z | 2020-04-18T15:05:10Z | 2020-04-18T15:05:10Z | MEMBER | https://developer.github.com/v3/repos/#list-contributors `GET /repos/:owner/:repo/contributors` Not sure if this should be a separate command or should be part of the existing `repos` command. I'm leaning towards a new `contributors` command. | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
603617013 | MDU6SXNzdWU2MDM2MTcwMTM= | 29 | Milestones should have foreign key to creator and repo | simonw 9599 | closed | 0 | 1 | 2020-04-21T00:20:44Z | 2020-04-21T00:43:58Z | 2020-04-21T00:43:58Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/milestones Creator is an integer but not a foreign key to users Repo is missing entirely! | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
493670426 | MDU6SXNzdWU0OTM2NzA0MjY= | 3 | Command to fetch all repos belonging to a user or organization | simonw 9599 | closed | 0 | 2 | 2019-09-14T21:54:21Z | 2019-09-17T00:17:53Z | 2019-09-17T00:17:53Z | MEMBER | How about this: $ github-to-sqlite repos simonw | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
603618244 | MDU6SXNzdWU2MDM2MTgyNDQ= | 30 | Issues milestone column is the wrong type | simonw 9599 | closed | 0 | 2 | 2020-04-21T00:24:34Z | 2020-04-21T00:45:23Z | 2020-04-21T00:36:22Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/issues?milestone=2857392 ![2A4C1185-2434-4F29-9EA0-3246E2F03F77](https://user-images.githubusercontent.com/9599/79811760-b7e08b00-832b-11ea-9ad7-684a6ae097a6.jpeg) It is TEXT when it should be an INTEGER - which is why the foreign key label is not correctly displayed. | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
603624862 | MDU6SXNzdWU2MDM2MjQ4NjI= | 31 | Issue and milestone should have foreign key to repo | simonw 9599 | closed | 0 | 3 | 2020-04-21T00:46:24Z | 2020-04-22T01:20:19Z | 2020-04-22T01:20:19Z | MEMBER | Currently the `repo` column on those tables is a string `simonw/datasette` rather than an ID referencing a row in `repos`. _Originally posted by @simonw in https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275_ | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issues] ( [id] INTEGER PRIMARY KEY, [node_id] TEXT, [number] INTEGER, [title] TEXT, [user] INTEGER REFERENCES [users]([id]), [state] TEXT, [locked] INTEGER, [assignee] INTEGER REFERENCES [users]([id]), [milestone] INTEGER REFERENCES [milestones]([id]), [comments] INTEGER, [created_at] TEXT, [updated_at] TEXT, [closed_at] TEXT, [author_association] TEXT, [pull_request] TEXT, [body] TEXT, [repo] INTEGER REFERENCES [repos]([id]), [type] TEXT , [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT); CREATE INDEX [idx_issues_repo] ON [issues] ([repo]); CREATE INDEX [idx_issues_milestone] ON [issues] ([milestone]); CREATE INDEX [idx_issues_assignee] ON [issues] ([assignee]); CREATE INDEX [idx_issues_user] ON [issues] ([user]);