issue_comments
525 rows where author_association = "MEMBER" sorted by id descending
This data as json, CSV (advanced)
id ▲ | html_url | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
660554162 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554162 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1NDE2Mg== | simonw 9599 | 2020-07-18T22:56:58Z | 2020-07-18T22:56:58Z | MEMBER | Manually fixing the database: ``` $ wget 'https://github-to-sqlite.dogsheep.net/github.db' --2020-07-18 15:52:33-- https://github-to-sqlite.dogsheep.net/github.db Resolving github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)... 172.217.5.115 Connecting to github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)|172.217.5.115|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 14626816 (14M) [application/octet-stream] Saving to: ‘github.db’ github.db 100%[============================================================================>] 13.95M 1.22MB/s in 18s 2020-07-18 15:52:53 (773 KB/s) - ‘github.db’ saved [14626816/14626816] $ sqlite3 github.db SQLite version 3.28.0 2019-04-15 14:49:49 Enter ".help" for usage hints. sqlite> drop table tags; sqlite> ^D $ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils dogsheep/healthkit-to-sqlite dogsheep/swarm-to-sqlite dogsheep/twitter-to-sqlite dogsheep/inaturalist-to-sqlite dogsheep/google-takeout-to-sqlite dogsheep/github-to-sqlite dogsheep/genome-to-sqlite dogsheep/pocket-to-sqlite dogsheep/hacker-news-to-sqlite dogsheep/dogsheep-photos $ sqlite-utils tables github.db --counts [{"table": "users", "count": 4048}, {"table": "repos", "count": 210}, ... {"table": "stars", "count": 4140}, {"table": "tags", "count": 188}] $ sqlite-utils rows github.db tags [{"repo": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"}, {"repo": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"}, ... ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660553711 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553711 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1MzcxMQ== | simonw 9599 | 2020-07-18T22:52:16Z | 2020-07-18T22:52:16Z | MEMBER | I think the best fix is to download the `github.db` database, manually fix it and then manually deploy it to Cloud Run from my laptop. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660553646 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553646 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1MzY0Ng== | simonw 9599 | 2020-07-18T22:51:41Z | 2020-07-18T22:51:41Z | MEMBER | I could fix this by putting `REFRESH_DB` in a commit message: https://github.com/dogsheep/github-to-sqlite/blob/4ae4aa6f172344b19ff3513707195ee6d2654bd4/.github/workflows/deploy-demo.yml#L41-L46 But... doing so would lose the data I've collected in https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc concerning the first time each dependent repo was spotted. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660551397 | https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660551397 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDY2MDU1MTM5Nw== | simonw 9599 | 2020-07-18T22:27:32Z | 2020-07-18T23:05:45Z | MEMBER | ```sql with most_recent_releases as ( with ranked as ( select repo, tag_name, published_at, row_number() OVER ( partition BY repo ORDER BY published_at DESC ) rank FROM releases ) select * from ranked where rank = 1 ) select repos.full_name as repo, most_recent_releases.tag_name as release, commits.committer_date as release_commit_date, ( select count(*) from commits c2 where c2.repo = repos.id and c2.committer_date > commits.committer_date ) as commits_since_release, 'https://github.com/' || repos.full_name || '/compare/' || most_recent_releases.tag_name || '...' || repos.default_branch as view_commits from most_recent_releases join repos on most_recent_releases.repo = repos.id join tags on tags.repo = repos.id and tags.name = most_recent_releases.tag_name join commits on tags.sha = commits.sha order by commits_since_release desc ``` repo | release | release_commit_date | commits_since_release | view_commits -- | -- | -- | -- | -- simonw/datasette | 0.45 | 2020-07-01T21:43:07Z | 9 | https://github.com/simonw/datasette/compare/0.45...master dogsheep/twitter-to-sqlite | 0.21.1 | 2020-04-30T18:20:43Z | 2 | https://github.com/dogsheep/twitter-to-sqlite/compare/0.21.1...master dogsheep/github-to-sqlite | 2.3 | 2020-07-09T23:26:34Z | 2 | https://github.com/dogsheep/github-to-sqlite/compare/2.3...master dogsheep/dogsheep-photos | 0.4.1 | 2020-05-25T20:11:20Z | 2 | https://github.com/dogsheep/dogsheep-photos/compare/0.4.1...master dogsheep/swarm-to-sqlite | 0.3.1 | 2020-03-28T02:29:41Z | 1 | https://github.com/dogsheep/swarm-to-sqlite/compare/0.3.1...master dogsheep/hacker-news-to-sqlite | 0.3.1 | 2020-03-21T22:39:34Z | 1 | https://github.com/dogsheep/hacker-news-to-sqlite/compare/0.3.1...master simonw/sqlite-utils | 2.11 | 2020-07-08T17:36:07Z | 0 | https://github.com/simonw/sqlite-utils/compare/2.11...master dog… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite tags command for fetching tags 660355904 | |
660548780 | https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660548780 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDY2MDU0ODc4MA== | simonw 9599 | 2020-07-18T22:02:37Z | 2020-07-18T23:05:56Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/tags?_facet=repo | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite tags command for fetching tags 660355904 | |
660547502 | https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660547502 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDY2MDU0NzUwMg== | simonw 9599 | 2020-07-18T21:50:37Z | 2020-07-18T21:50:37Z | MEMBER | ``` $ github-to-sqlite tags tags.db simonw/datasette dogsheep/github-to-sqlite $ sqlite-utils tables tags.db --counts [{"table": "users", "count": 2}, {"table": "licenses", "count": 1}, {"table": "repos", "count": 2}, {"table": "tags", "count": 76}, {"table": "licenses_fts", "count": 1}, {"table": "licenses_fts_data", "count": 3}, {"table": "licenses_fts_idx", "count": 1}, {"table": "licenses_fts_docsize", "count": 1}, {"table": "licenses_fts_config", "count": 1}, {"table": "repos_fts", "count": 2}, {"table": "repos_fts_data", "count": 3}, {"table": "repos_fts_idx", "count": 1}, {"table": "repos_fts_docsize", "count": 2}, {"table": "repos_fts_config", "count": 1}, {"table": "users_fts", "count": 2}, {"table": "users_fts_data", "count": 3}, {"table": "users_fts_idx", "count": 1}, {"table": "users_fts_docsize", "count": 2}, {"table": "users_fts_config", "count": 1}] $ sqlite-utils rows tags.db tags [{"repo_id": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"}, {"repo_id": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"}, {"repo_id": 107914493, "name": "0.45a4", "sha": "265483173bc8341dc02c8b782b9b59d2ce8bbedb"}, {"repo_id": 107914493, "name": "0.45a3", "sha": "1f55a4a2b68fa65e56a28baeb7f44122fdeca7e7"}, {"repo_id": 107914493, "name": "0.45a2", "sha": "1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f"}, {"repo_id": 107914493, "name": "0.45a1", "sha": "b59b92b1b0517cf18fa748ff9d0a0bf86298dd43"}, {"repo_id": 107914493, "name": "0.45a0", "sha": "dda932d818b34ccab11730a76554f0a3748d8348"}, {"repo_id": 107914493, "name": "0.44", "sha": "b906030235efbdff536405d66078f4868ce0d3bd"}, {"repo_id": 107914493, "name": "0.43", "sha": "d56f402822df102f9cf1a9a056449d01a15e3aae"}, {"repo_id": 107914493, "name": "0.42", "sha": "af6c6c5d6f929f951c0e63bfd1c82e37a071b50f"}, {"repo_id": 107914493, "name": "0.41", "sha": "182e5c8745c94576718315f7596ccc81e5e2417b"}, {"repo_id": 107914493, "name": "0.40", "sha": "8da108193b08ab… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite tags command for fetching tags 660355904 | |
660536265 | https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660536265 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDY2MDUzNjI2NQ== | simonw 9599 | 2020-07-18T20:15:12Z | 2020-07-18T20:15:12Z | MEMBER | I want to create a SQL query which shows me all of my repositories that have commits that are NOT in the most recent release. The releases table doesn't have enough information for this because it doesn't tell you the commit hash associated with each release, just the tag: https://github-to-sqlite.dogsheep.net/github/releases | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite tags command for fetching tags 660355904 | |
653966670 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653966670 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2NjY3MA== | simonw 9599 | 2020-07-06T01:07:02Z | 2020-07-06T01:07:02Z | MEMBER | OK that fix worked.https://github.com/dogsheep/github-to-sqlite/runs/839764768?check_suite_focus=true | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653962708 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962708 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjcwOA== | simonw 9599 | 2020-07-06T00:43:10Z | 2020-07-06T00:43:10Z | MEMBER | I bet it's datasette-search-all. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653962669 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962669 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjY2OQ== | simonw 9599 | 2020-07-06T00:42:57Z | 2020-07-06T00:42:57Z | MEMBER | https://github-to-sqlite.dogsheep.net/-/plugins ```json [ { "name": "datasette-json-html", "static": false, "templates": false, "version": "0.6", "hooks": [ "prepare_connection", "render_cell" ] }, { "name": "datasette-render-markdown", "static": false, "templates": false, "version": "1.1.2", "hooks": [ "extra_template_vars", "render_cell" ] }, { "name": "datasette-pretty-json", "static": false, "templates": false, "version": "0.2", "hooks": [ "render_cell" ] }, { "name": "datasette-search-all", "static": false, "templates": true, "version": "0.2.1", "hooks": [ "asgi_wrapper", "extra_template_vars" ] }, { "name": "datasette-vega", "static": true, "templates": false, "version": "0.6.2", "hooks": [ "extra_css_urls", "extra_js_urls" ] } ] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653962530 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962530 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjUzMA== | simonw 9599 | 2020-07-06T00:42:13Z | 2020-07-06T00:42:13Z | MEMBER | So it looks like it's the ASGI lifespan change I made in https://github.com/simonw/datasette/commit/16f592247a2a0e140ada487e9972645406dcae69 - It must be incompatible with one of the plugins. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653962418 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962418 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjQxOA== | simonw 9599 | 2020-07-06T00:41:38Z | 2020-07-06T00:41:38Z | MEMBER | https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/logs?project=datasette-222320 has some clues. <img width="818" alt="github-to-sqlite_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/86545887-92086280-bee6-11ea-8787-b3b85a7bb4b9.png"> <img width="1451" alt="github-to-sqlite_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/86545917-c11ed400-bee6-11ea-9cf8-957911ca9f6a.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653960989 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653960989 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MDk4OQ== | simonw 9599 | 2020-07-06T00:32:34Z | 2020-07-06T00:32:34Z | MEMBER | Same error. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653947916 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653947916 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk0NzkxNg== | simonw 9599 | 2020-07-05T22:40:47Z | 2020-07-05T22:40:47Z | MEMBER | Might be that it's not got enough RAM. I'll try deploying to a larger instance. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
645599881 | https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645599881 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47 | MDEyOklzc3VlQ29tbWVudDY0NTU5OTg4MQ== | simonw 9599 | 2020-06-17T20:13:48Z | 2020-06-17T20:13:48Z | MEMBER | I've now figured out how to compile specific SQLite versions to help replicate this problem: https://github.com/simonw/til/blob/master/sqlite/ld-preload.md Next step: replicate the problem! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fall back to FTS4 if FTS5 is not available 639542974 | |
645512127 | https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47 | MDEyOklzc3VlQ29tbWVudDY0NTUxMjEyNw== | simonw 9599 | 2020-06-17T17:24:22Z | 2020-06-17T17:24:22Z | MEMBER | That means your version of SQLite is old enough that it doesn't support the FTS5 extension. Could you share what operating system you're running, and what the output is that you get from running this? python -c 'import sqlite3; print(sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0])' I can teach this tool to fall back on FTS4 if FTS5 isn't available. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fall back to FTS4 if FTS5 is not available 639542974 | |
643414646 | https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643414646 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDY0MzQxNDY0Ng== | simonw 9599 | 2020-06-12T18:06:48Z | 2020-06-12T18:06:48Z | MEMBER | That fixed it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo deploy is broken 637899539 | |
643393506 | https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643393506 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDY0MzM5MzUwNg== | simonw 9599 | 2020-06-12T17:21:14Z | 2020-06-12T17:21:14Z | MEMBER | I only install SQLite for this: https://github.com/dogsheep/github-to-sqlite/blob/c0d54e0260468be38152293df5abd775c068495d/.github/workflows/deploy-demo.yml#L77-L78 I'm going to remove the need to install sqlite3 by making this possible with sqlite-utils: https://github.com/simonw/sqlite-utils/issues/115 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo deploy is broken 637899539 | |
633704127 | https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633704127 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 | MDEyOklzc3VlQ29tbWVudDYzMzcwNDEyNw== | simonw 9599 | 2020-05-25T20:14:22Z | 2020-05-25T20:14:22Z | MEMBER | https://github.com/dogsheep/dogsheep-photos/blob/0.4.1/README.md#serving-photos-locally-with-datasette-media | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to serve thumbnailed Apple Photo from its place on disk 613006393 | |
633644225 | https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633644225 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 | MDEyOklzc3VlQ29tbWVudDYzMzY0NDIyNQ== | simonw 9599 | 2020-05-25T16:30:44Z | 2020-05-25T16:30:44Z | MEMBER | I'll add docs on using `datasette-json-html` too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to serve thumbnailed Apple Photo from its place on disk 613006393 | |
633643921 | https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633643921 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 | MDEyOklzc3VlQ29tbWVudDYzMzY0MzkyMQ== | simonw 9599 | 2020-05-25T16:29:44Z | 2020-05-25T16:29:44Z | MEMBER | https://github.com/dogsheep/dogsheep-photos/blob/dc43fa8653cb9c7238a36f52239b91d1ec916d5c/README.md#serving-photos-locally-with-datasette-media | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to serve thumbnailed Apple Photo from its place on disk 613006393 | |
633629944 | https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633629944 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 | MDEyOklzc3VlQ29tbWVudDYzMzYyOTk0NA== | simonw 9599 | 2020-05-25T15:47:42Z | 2020-05-25T15:47:42Z | MEMBER | I'll add a proper section to the README, but for the moment here's how I do this. First, install `datasette` and the `datasette-media` plugin. Create a `metadata.yaml` file with the following content: ```yaml plugins: datasette-media: photo: sql: |- select path as filepath, 200 as resize_height from apple_photos where uuid = :key photo-big: sql: |- select path as filepath, 1024 as resize_height from apple_photos where uuid = :key ``` Now run `datasette -m metadata.yaml photos.db` - thumbnails will be served at http://127.0.0.1:8001/-/media/photo/F4469918-13F3-43D8-9EC1-734C0E6B60AD and larger sizes of the image at http://127.0.0.1:8001/-/media/photo-big/A8B02C7D-365E-448B-9510-69F80C26304D I also made myself two custom pages, one showing recent images and one showing random images. To do this, install the `datasette-template-sql` plugin and then create a `templates/pages` directory and add these files: `recent-photos.html` ```html <h1>Recent photos</h1> <div> {% for photo in sql("select * from apple_photos order by date desc limit 100") %} <img src="/-/media/photo/{{ photo['uuid'] }}"> {% endfor %} </div> ``` `random-photos.html` ```html <h1>Random photos</h1> <div> {% for photo in sql("with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 100") %} <img src="/-/media/photo/{{ photo['uuid'] }}"> {% endfor %} </div> ``` Now run `datasette -m metadata.yaml photos.db --template-dir=templates/` Visit http://127.0.0.1:8001/random-photos to see some random photos or http://127.0.0.1:8002/recent-photos for recent photos. This is using this mechanism: https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to serve thumbnailed Apple Photo from its place on disk 613006393 | |
633626741 | https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633626741 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 | MDEyOklzc3VlQ29tbWVudDYzMzYyNjc0MQ== | simonw 9599 | 2020-05-25T15:38:55Z | 2020-05-25T15:38:55Z | MEMBER | Sure, I should absolutely document this! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to serve thumbnailed Apple Photo from its place on disk 613006393 | |
631255206 | https://github.com/dogsheep/dogsheep-photos/issues/24#issuecomment-631255206 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/24 | MDEyOklzc3VlQ29tbWVudDYzMTI1NTIwNg== | simonw 9599 | 2020-05-20T06:00:25Z | 2020-05-20T06:00:25Z | MEMBER | This needs documentation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Configurable URL for images 621323348 | |
631253852 | https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253852 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25 | MDEyOklzc3VlQ29tbWVudDYzMTI1Mzg1Mg== | simonw 9599 | 2020-05-20T05:56:17Z | 2020-05-21T22:26:16Z | MEMBER | I have a `deploy-demo.sh` script now: ```bash #!/bin/bash if [ -f public.db ]; then rm public.db fi pipenv run dogsheep-photos create-subset photos.db public.db \ "select sha256 from apple_photos where albums like '%Public%'" pipenv run sqlite-utils create-view public.db photos_on_a_map \ "select date, latitude, longitude, apple_photos.sha256, uploads.ext, json_object( 'title', 'Taken on ' || date, 'image', 'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=400', 'link', 'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=1200' ) as popup from apple_photos join uploads on apple_photos.sha256 = uploads.sha256 where latitude is not null order by date desc" \ --replace pipenv run datasette publish now public.db --project dogsheep-photos \ --about=dogsheep/dogsheep-photos \ --about_url="https://github.com/dogsheep/dogsheep-photos" \ --install=datasette-json-html \ --install=datasette-pretty-json \ --install=datasette-cluster-map>=0.10 \ --title "Dogsheep Photos demo" ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create a public demo 621332242 | |
631253248 | https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253248 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25 | MDEyOklzc3VlQ29tbWVudDYzMTI1MzI0OA== | simonw 9599 | 2020-05-20T05:54:18Z | 2020-05-20T05:54:18Z | MEMBER | https://dogsheep-photos.dogsheep.net/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create a public demo 621332242 | |
631253136 | https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253136 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25 | MDEyOklzc3VlQ29tbWVudDYzMTI1MzEzNg== | simonw 9599 | 2020-05-20T05:53:58Z | 2020-05-20T05:53:58Z | MEMBER | Updated deploy command: ``` datasette publish now public.db --project dogsheep-photos \ --about=dogsheep/dogsheep-photos \ --about_url="https://github.com/dogsheep/dogsheep-photos" \ --install=datasette-json-html \ --install=datasette-cluster-map \ --title "Dogsheep Photos demo" ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create a public demo 621332242 | |
631251707 | https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631251707 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25 | MDEyOklzc3VlQ29tbWVudDYzMTI1MTcwNw== | simonw 9599 | 2020-05-20T05:49:27Z | 2020-05-21T15:58:42Z | MEMBER | Renaming this demo to `dogsheep-photos.dogsheep.net` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create a public demo 621332242 | |
631229485 | https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229485 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 | MDEyOklzc3VlQ29tbWVudDYzMTIyOTQ4NQ== | simonw 9599 | 2020-05-20T04:31:02Z | 2020-05-20T04:31:02Z | MEMBER | https://pypi.org/project/dogsheep-photos/ is live. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rename project to dogsheep-photos 621444763 | |
631229409 | https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229409 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 | MDEyOklzc3VlQ29tbWVudDYzMTIyOTQwOQ== | simonw 9599 | 2020-05-20T04:30:40Z | 2020-05-20T04:30:40Z | MEMBER | https://pypi.org/project/photos-to-sqlite/ now links to dogsheep-photos. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rename project to dogsheep-photos 621444763 | |
631227245 | https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227245 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 | MDEyOklzc3VlQ29tbWVudDYzMTIyNzI0NQ== | simonw 9599 | 2020-05-20T04:21:38Z | 2020-05-20T04:21:38Z | MEMBER | I'm going to release 0.4 now. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rename project to dogsheep-photos 621444763 | |
631227105 | https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227105 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 | MDEyOklzc3VlQ29tbWVudDYzMTIyNzEwNQ== | simonw 9599 | 2020-05-20T04:21:06Z | 2020-05-20T04:21:06Z | MEMBER | Then I just need to push a final photos-to-sqlite release that updates the README to tell people about the name change. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rename project to dogsheep-photos 621444763 | |
631227020 | https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227020 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 | MDEyOklzc3VlQ29tbWVudDYzMTIyNzAyMA== | simonw 9599 | 2020-05-20T04:20:48Z | 2020-05-20T04:21:16Z | MEMBER | Next time I push a release it will create `dogsheep-photos` on PyPI. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rename project to dogsheep-photos 621444763 | |
631226953 | https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226953 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 | MDEyOklzc3VlQ29tbWVudDYzMTIyNjk1Mw== | simonw 9599 | 2020-05-20T04:20:34Z | 2020-05-20T04:20:34Z | MEMBER | Huh, it looks like Circle CI picked up the name change automatically. https://app.circleci.com/pipelines/github/dogsheep/dogsheep-photos | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rename project to dogsheep-photos 621444763 | |
631226572 | https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226572 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 | MDEyOklzc3VlQ29tbWVudDYzMTIyNjU3Mg== | simonw 9599 | 2020-05-20T04:18:52Z | 2020-05-20T04:18:52Z | MEMBER | Need to reconfigure Circle CI. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rename project to dogsheep-photos 621444763 | |
631226481 | https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226481 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 | MDEyOklzc3VlQ29tbWVudDYzMTIyNjQ4MQ== | simonw 9599 | 2020-05-20T04:18:29Z | 2020-05-20T04:18:29Z | MEMBER | I just renamed the repository. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rename project to dogsheep-photos 621444763 | |
631127454 | https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631127454 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25 | MDEyOklzc3VlQ29tbWVudDYzMTEyNzQ1NA== | simonw 9599 | 2020-05-19T22:48:00Z | 2020-05-21T15:58:32Z | MEMBER | I built #23 to help with this. $ dogsheep-photos create-subset photos.db public.db \ "select sha256 from apple_photos where albums like '%Public%'" And publish with Vercel: $ datasette publish now public.db --project dogsheep-photos \ --about=dogsheep/dogsheep-photos \ --about_url="https://github.com/dogsheep/dogsheep-photos" \ --install=datasette-json-html \ --install=datasette-cluster-map | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create a public demo 621332242 | |
631120771 | https://github.com/dogsheep/dogsheep-photos/issues/23#issuecomment-631120771 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/23 | MDEyOklzc3VlQ29tbWVudDYzMTEyMDc3MQ== | simonw 9599 | 2020-05-19T22:32:48Z | 2020-05-19T22:32:48Z | MEMBER | Documentation: https://github.com/dogsheep/photos-to-sqlite/blob/e2fab012551eed05278040b5d57e7373a1b9a0bf/README.md#creating-a-subset-database | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | create-subset command for creating a publishable subset of a photos database 621280529 | |
626941278 | https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-626941278 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22 | MDEyOklzc3VlQ29tbWVudDYyNjk0MTI3OA== | simonw 9599 | 2020-05-11T20:25:58Z | 2020-05-11T20:25:58Z | MEMBER | Interesting - do you know if there's anything the `exiftool` process handles that `ExifReader` doesn't? I'm actually just going to extract a subset of the EXIF data at first - since the original photo files will always be available I don't feel the need to get everything out for the first step. My plan is to use EXIF to help support photo collections that aren't in Apple Photos - I'm going to build a database table keyed by the `sha256` of each photo that extracts the camera make, lens, a few settings (ISO, aperture etc) and the GPS lat/lon. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Try out ExifReader 615626118 | |
626395781 | https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395781 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21 | MDEyOklzc3VlQ29tbWVudDYyNjM5NTc4MQ== | simonw 9599 | 2020-05-10T21:57:09Z | 2020-05-10T21:57:09Z | MEMBER | Yes, I just recreated my virtual environment from scratch and the error went away. The problem occurred when I ran `pip install datasette-bplist` in the same virtual environment - https://github.com/simonw/datasette-bplist/blob/master/setup.py depends on `bpylist` which is incompatible with `bpylist2`. | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | bpylist.archiver.CircularReference: archive has a cycle with uid(13) 615474990 | |
626395209 | https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395209 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21 | MDEyOklzc3VlQ29tbWVudDYyNjM5NTIwOQ== | simonw 9599 | 2020-05-10T21:52:42Z | 2020-05-10T21:52:42Z | MEMBER | Aha! It looks like I accidentally installed the old bplist into the same environment: ``` $ pip freeze | grep bpylist bpylist==0.1.4 bpylist2==3.0.0 ``` | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | bpylist.archiver.CircularReference: archive has a cycle with uid(13) 615474990 | |
626395103 | https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395103 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21 | MDEyOklzc3VlQ29tbWVudDYyNjM5NTEwMw== | simonw 9599 | 2020-05-10T21:51:36Z | 2020-05-10T21:51:36Z | MEMBER | @RhetTbull I tried that workaround and it turns out I'm getting this error on ALL of my photos now! It's weird: a few day ago this wasn't happening. Now it's happening to everything. I'm not sure what I might have changed. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | bpylist.archiver.CircularReference: archive has a cycle with uid(13) 615474990 | |
626394989 | https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626394989 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21 | MDEyOklzc3VlQ29tbWVudDYyNjM5NDk4OQ== | simonw 9599 | 2020-05-10T21:50:36Z | 2020-05-10T21:50:36Z | MEMBER | https://github.com/Marketcircle/bpylist/pull/2 looks relevant here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | bpylist.archiver.CircularReference: archive has a cycle with uid(13) 615474990 | |
626388837 | https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388837 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21 | MDEyOklzc3VlQ29tbWVudDYyNjM4ODgzNw== | simonw 9599 | 2020-05-10T20:59:32Z | 2020-05-10T20:59:32Z | MEMBER | So it appears it's possible for `photo.place` to raise that exception. A workaround could be to catch that and treat those photos as not having a place. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | bpylist.archiver.CircularReference: archive has a cycle with uid(13) 615474990 | |
626388764 | https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388764 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21 | MDEyOklzc3VlQ29tbWVudDYyNjM4ODc2NA== | simonw 9599 | 2020-05-10T20:58:52Z | 2020-05-10T20:58:52Z | MEMBER | More from the debugger: ``` > /Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py(614)place() -> self._place = PlaceInfo5(self._info["reverse_geolocation"]) ``` And: ``` > /Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/utils.py(91)osxphoto_to_row() -> place = photo.place ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | bpylist.archiver.CircularReference: archive has a cycle with uid(13) 615474990 | |
625947133 | https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-625947133 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 | MDEyOklzc3VlQ29tbWVudDYyNTk0NzEzMw== | simonw 9599 | 2020-05-08T18:13:06Z | 2020-05-08T18:13:06Z | MEMBER | `datasette-media` will be able to handle this once I implement https://github.com/simonw/datasette-media/issues/3 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to serve thumbnailed Apple Photo from its place on disk 613006393 | |
624408738 | https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408738 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 | MDEyOklzc3VlQ29tbWVudDYyNDQwODczOA== | simonw 9599 | 2020-05-06T02:21:05Z | 2020-05-06T02:21:32Z | MEMBER | Here's rendering code from my hacked-together not-yet-released S3 image proxy: ```python from starlette.responses import Response from PIL import Image, ExifTags import pyheif for ORIENTATION_TAG in ExifTags.TAGS.keys(): if ExifTags.TAGS[ORIENTATION_TAG] == "Orientation": break ... # Load it into Pillow if ext == "heic": heic = pyheif.read_heif(image_response.content) image = Image.frombytes(mode=heic.mode, size=heic.size, data=heic.data) else: image = Image.open(io.BytesIO(image_response.content)) # Does EXIF tell us to rotate it? try: exif = dict(image._getexif().items()) if exif[ORIENTATION_TAG] == 3: image = image.rotate(180, expand=True) elif exif[ORIENTATION_TAG] == 6: image = image.rotate(270, expand=True) elif exif[ORIENTATION_TAG] == 8: image = image.rotate(90, expand=True) except (AttributeError, KeyError, IndexError): pass # Resize based on ?w= and ?h=, if set width, height = image.size w = request.query_params.get("w") h = request.query_params.get("h") if w is not None or h is not None: if h is None: # Set h based on w w = int(w) h = int((float(height) / width) * w) elif w is None: h = int(h) # Set w based on h w = int((float(width) / height) * h) w = int(w) h = int(h) image.thumbnail((w, h)) # ?bw= converts to black and white if request.query_params.get("bw"): image = image.convert("L") # ?q= sets the quality - defaults to 75 quality = 75 q = request.query_params.get("q") if q and q.isdigit() and 1 <= int(q) <= 100: quality = int(q) # Output as JPEG or PNG output_image = io.BytesIO() image_type = "JPEG" kwargs = {"quality": quality} if image.format == "PNG": image_type = "PNG" kwargs = {} … | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to serve thumbnailed Apple Photo from its place on disk 613006393 | |
624408370 | https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408370 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 | MDEyOklzc3VlQ29tbWVudDYyNDQwODM3MA== | simonw 9599 | 2020-05-06T02:19:27Z | 2020-05-06T02:19:27Z | MEMBER | The plugin can be generalized: it can be configured to know how to take the URL path, look it up in ANY table (via a custom SQL query) to get a path on disk and then serve that. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to serve thumbnailed Apple Photo from its place on disk 613006393 | |
624408220 | https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408220 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 | MDEyOklzc3VlQ29tbWVudDYyNDQwODIyMA== | simonw 9599 | 2020-05-06T02:18:47Z | 2020-05-06T02:18:47Z | MEMBER | The `apple_photos` table has an indexed `uuid` column and a `path` column which stores the full path to that photo file on disk. I can write a custom Datasette plugin which takes the `uuid` from the URL, looks up the path, then serves up a thumbnail of the jpeg or heic image file. I'll prototype this is a one-off plugin first, then package it on PyPI for other people to install. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to serve thumbnailed Apple Photo from its place on disk 613006393 | |
624406285 | https://github.com/dogsheep/dogsheep-photos/issues/19#issuecomment-624406285 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/19 | MDEyOklzc3VlQ29tbWVudDYyNDQwNjI4NQ== | simonw 9599 | 2020-05-06T02:10:03Z | 2020-05-06T02:10:03Z | MEMBER | Most annoying part of this is the difficulty of actually showing a photo. Maybe I need to run a local proxy that I can link to? A custom Datasette plugin perhaps? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | apple-photos command should work even if upload has not run 613002220 | |
624364557 | https://github.com/dogsheep/dogsheep-photos/issues/18#issuecomment-624364557 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18 | MDEyOklzc3VlQ29tbWVudDYyNDM2NDU1Nw== | simonw 9599 | 2020-05-05T23:49:18Z | 2020-05-05T23:49:18Z | MEMBER | Label is `macos-latest` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Switch CI solution to GitHub Actions with a macOS runner 612860758 | |
624278714 | https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624278714 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17 | MDEyOklzc3VlQ29tbWVudDYyNDI3ODcxNA== | simonw 9599 | 2020-05-05T20:07:19Z | 2020-05-05T20:07:19Z | MEMBER | From https://hynek.me/articles/conditional-python-dependencies/ I think this will look like: ```python setup( # ... install_requires=[ # ... "osxphotos>=0.28.13 ; sys_platform=='darwin'", ] ) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only install osxphotos if running on macOS 612860531 | |
624278090 | https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624278090 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17 | MDEyOklzc3VlQ29tbWVudDYyNDI3ODA5MA== | simonw 9599 | 2020-05-05T20:06:01Z | 2020-05-05T20:06:01Z | MEMBER | https://www.python.org/dev/peps/pep-0508/#environment-markers I think I want `sys_platform` of `darwin`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only install osxphotos if running on macOS 612860531 | |
623865250 | https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623865250 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 | MDEyOklzc3VlQ29tbWVudDYyMzg2NTI1MA== | simonw 9599 | 2020-05-05T05:38:16Z | 2020-05-05T05:38:16Z | MEMBER | It looks like `groups.content_string` often has a null byte in it. I should clean this up as part of the import. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234 | |
623863902 | https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623863902 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 | MDEyOklzc3VlQ29tbWVudDYyMzg2MzkwMg== | simonw 9599 | 2020-05-05T05:31:53Z | 2020-05-05T05:31:53Z | MEMBER | Yes! Turning those `rowid` values into `id` with this script did the job: ```python import sqlite3 import sqlite_utils conn = sqlite3.connect( "/Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite" ) def all_rows(table): result = conn.execute("select rowid as id, * from {}".format(table)) cols = [c[0] for c in result.description] for row in result.fetchall(): yield dict(zip(cols, row)) if __name__ == "__main__": db = sqlite_utils.Database("psi_copy.db") for table in ("assets", "collections", "ga", "gc", "groups"): db[table].upsert_all(all_rows(table), pk="id", alter=True) ``` Then I ran this query: ```sql select json_object('img_src', 'https://photos.simonwillison.net/i/' || photos.sha256 || '.' || photos.ext || '?w=400') as photo, group_concat(strip_null_chars(groups.content_string), ' ') as words, assets.uuid_0, assets.uuid_1, to_uuid(assets.uuid_0, assets.uuid_1) as uuid from assets join ga on assets.id = ga.assetid join groups on ga.groupid = groups.id join photos on photos.uuid = to_uuid(assets.uuid_0, assets.uuid_1) where groups.category = 2024 group by assets.id order by random() limit 10 ``` And got these results! <img width="1054" alt="psi_copy__select_json_object__img_src____https___photos_simonwillison_net_i______photos_sha256___________photos_ext______w_400___as_photo__group_concat_strip_null_chars_groups_content_string________as_words__assets_uuid_0__assets_uuid_1__to" src="https://user-images.githubusercontent.com/9599/81037264-f1021a80-8e56-11ea-9924-6f9f55a0fb4b.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234 | |
623857417 | https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623857417 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 | MDEyOklzc3VlQ29tbWVudDYyMzg1NzQxNw== | simonw 9599 | 2020-05-05T05:01:47Z | 2020-05-05T05:01:47Z | MEMBER | Even that didn't work - it didn't copy across the rowid values. I'm pretty sure that's what's wrong here: ``` sqlite3 /Users/simon/Pictures/Photos\ Library.photoslibrary/database/search/psi.sqlite 'select rowid, uuid_0, uuid_1 from assets limit 10' 1619605|-9205353363298198838|4814875488794983828 1641378|-9205348195631362269|390804289838822030 1634974|-9205331524553603243|-3834026796261633148 1619083|-9205326176986145401|7563404215614709654 22131|-9205315724827218763|8370531509591906734 1645633|-9205247376092758131|-1311540150497601346 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234 | |
623855885 | https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855885 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 | MDEyOklzc3VlQ29tbWVudDYyMzg1NTg4NQ== | simonw 9599 | 2020-05-05T04:54:39Z | 2020-05-05T04:54:53Z | MEMBER | Trying this import mechanism instead: `sqlite3 /Users/simon/Pictures/Photos\ Library.photoslibrary/database/search/psi.sqlite .dump | grep -v 'CREATE INDEX' | grep -v 'CREATE TRIGGER' | grep -v 'CREATE VIRTUAL TABLE' | sqlite3 search.db` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234 | |
623855841 | https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855841 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 | MDEyOklzc3VlQ29tbWVudDYyMzg1NTg0MQ== | simonw 9599 | 2020-05-05T04:54:28Z | 2020-05-05T04:54:28Z | MEMBER | Things were not matching up for me correctly: <img width="1143" alt="search__select_json_object__img_src____https___photos_simonwillison_net_i______photos_sha256___________photos_ext______w_400___as_photo__groups_content_string__assets_uuid_0__assets_uuid_1__to_uuid_assets_uuid_0__assets_uuid_1__as_uuid__pho" src="https://user-images.githubusercontent.com/9599/81035923-ca8db080-8e51-11ea-95a7-6ee60bae7502.png"> I think that's because my import script didn't correctly import the existing `rowid` values. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234 | |
623846880 | https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623846880 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 | MDEyOklzc3VlQ29tbWVudDYyMzg0Njg4MA== | simonw 9599 | 2020-05-05T04:06:08Z | 2020-05-05T04:06:08Z | MEMBER | This function seems to convert them into UUIDs that match my photos: ```python def to_uuid(uuid_0, uuid_1): b = uuid_0.to_bytes(8, 'little', signed=True) + uuid_1.to_bytes(8, 'little', signed=True) return str(uuid.UUID(bytes=b)).upper() ``` | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234 | |
623811131 | https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623811131 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 | MDEyOklzc3VlQ29tbWVudDYyMzgxMTEzMQ== | simonw 9599 | 2020-05-05T03:16:18Z | 2020-05-05T03:16:18Z | MEMBER | Here's how to convert two integers unto a UUID using Java. Not sure if it's the solution I need though (or how to do the same thing in Python): https://repl.it/repls/EuphoricSomberClasslibrary <img width="1494" alt="Repl_it_-_EuphoricSomberClasslibrary" src="https://user-images.githubusercontent.com/9599/81032267-0d488c00-8e44-11ea-9be7-680eaccd1611.png"> ```java import java.util.UUID; class Main { public static void main(String[] args) { java.util.UUID uuid = new java.util.UUID( 2544182952487526660L, -3640314103732024685L ); System.out.println( uuid ); } } ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234 | |
623807568 | https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623807568 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 | MDEyOklzc3VlQ29tbWVudDYyMzgwNzU2OA== | simonw 9599 | 2020-05-05T02:56:06Z | 2020-05-05T02:56:06Z | MEMBER | I'm pretty sure this is what I'm after. The `groups` table has what looks like identified labels in the rows with category = 2025: <img width="1122" alt="words__groups__2_528_rows_where_where_category___2025" src="https://user-images.githubusercontent.com/9599/81031361-e0df4080-8e40-11ea-9060-6d850aa52140.png"> Then there's a `ga` table that maps groups to assets: <img width="304" alt="words__ga__633_653_rows" src="https://user-images.githubusercontent.com/9599/81031387-f48aa700-8e40-11ea-9a3d-da23903be928.png"> And an `assets` table which looks like it has one row for every one of my photos: <img width="645" alt="words__assets__40_419_rows" src="https://user-images.githubusercontent.com/9599/81031402-04a28680-8e41-11ea-8047-e9199d068563.png"> One major challenge: these UUIDs are split into two integer numbers, `uuid_0` and `uuid_1` - but the main photos database uses regular UUIDs like this: ![image](https://user-images.githubusercontent.com/9599/81031481-39164280-8e41-11ea-983b-005ced641a18.png) I need to figure out how to match up these two different UUID representations. I asked on Twitter if anyone has any ideas: https://twitter.com/simonw/status/1257500689019703296 | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234 | |
623806687 | https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806687 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 | MDEyOklzc3VlQ29tbWVudDYyMzgwNjY4Nw== | simonw 9599 | 2020-05-05T02:51:16Z | 2020-05-05T02:51:16Z | MEMBER | Running datasette against it directly doesn't work: ``` simon@Simons-MacBook-Pro search % datasette psi.sqlite Serve! files=('psi.sqlite',) (immutables=()) on port 8001 Usage: datasette serve [OPTIONS] [FILES]... Error: Connection to psi.sqlite failed check: no such tokenizer: PSITokenizer ``` Instead, I created a new SQLite database with a copy of some of the key tables, like this: ``` sqlite-utils rows psi.sqlite groups | sqlite-utils insert /tmp/search.db groups - sqlite-utils rows psi.sqlite assets | sqlite-utils insert /tmp/search.db assets - sqlite-utils rows psi.sqlite ga | sqlite-utils insert /tmp/search.db ga - sqlite-utils rows psi.sqlite collections | sqlite-utils insert /tmp/search.db collections - sqlite-utils rows psi.sqlite gc | sqlite-utils insert /tmp/search.db gc - sqlite-utils rows psi.sqlite lookup | sqlite-utils insert /tmp/search.db lookup - ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234 | |
623806533 | https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806533 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 | MDEyOklzc3VlQ29tbWVudDYyMzgwNjUzMw== | simonw 9599 | 2020-05-05T02:50:16Z | 2020-05-05T02:50:16Z | MEMBER | I figured there must be a separate database that Photos uses to store the text of the identified labels. I used "Open Files and Ports" in Activity Monitor against the Photos app to try and spot candidates... and found `/Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite` - a 53MB SQLite database file. <img width="1365" alt="Item-0_and_Item-0_and_Item-0_and_Item-0" src="https://user-images.githubusercontent.com/9599/81031213-61ea0800-8e40-11ea-8237-cfce4a5128e0.png"> Here's the schema of that file: ``` $ sqlite3 psi.sqlite .schema CREATE TABLE word_embedding(word TEXT, extended_word TEXT, score DOUBLE); CREATE INDEX word_embedding_index ON word_embedding(word); CREATE VIRTUAL TABLE word_embedding_prefix USING fts5(extended_word) /* word_embedding_prefix(extended_word) */; CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_data'(id INTEGER PRIMARY KEY, block BLOB); CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID; CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_content'(id INTEGER PRIMARY KEY, c0); CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB); CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID; CREATE TABLE groups(category INT2, owning_groupid INT, content_string TEXT, normalized_string TEXT, lookup_identifier TEXT, token_ranges_0 INT8, token_ranges_1 INT8, UNIQUE(category, owning_groupid, content_string, lookup_identifier, token_ranges_0, token_ranges_1)); CREATE TABLE assets(uuid_0 INT, uuid_1 INT, creationDate INT, UNIQUE(uuid_0, uuid_1)); CREATE TABLE ga(groupid INT, assetid INT, PRIMARY KEY(groupid, assetid)); CREATE TABLE collections(uuid_0 INT, uuid_1 INT, startDate INT, endDate INT, title TEXT, subtitle TEXT, keyAssetUUID_0 INT, keyAssetUUID_1 INT, typeAndNumberOfAssets INT32, sortDate DOUBLE, UNIQUE(uuid_0, uuid_1)); CREATE TABLE gc(groupid INT, collectionid INT, PRIMARY KEY(groupid, collectionid)); CREATE… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234 | |
623806085 | https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806085 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 | MDEyOklzc3VlQ29tbWVudDYyMzgwNjA4NQ== | simonw 9599 | 2020-05-05T02:47:18Z | 2020-05-05T02:47:18Z | MEMBER | In https://github.com/RhetTbull/osxphotos/issues/121#issuecomment-623249263 Rhet Turnbull spotted a table called `ZSCENEIDENTIFIER` which looked like it might have the right data, but the columns in it aren't particularly helpful: ``` Z_PK,Z_ENT,Z_OPT,ZSCENEIDENTIFIER,ZASSETATTRIBUTES,ZCONFIDENCE 8,49,1,731,5,0.11834716796875 9,49,1,684,6,0.0233648251742125 10,49,1,1702,1,0.026153564453125 ``` I love the look of those confidence scores, but what do the numbers mean? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234 | |
623805823 | https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623805823 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 | MDEyOklzc3VlQ29tbWVudDYyMzgwNTgyMw== | simonw 9599 | 2020-05-05T02:45:56Z | 2020-05-05T02:45:56Z | MEMBER | I filed an issue with `osxphotos` about this here: https://github.com/RhetTbull/osxphotos/issues/121 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234 | |
623739934 | https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623739934 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15 | MDEyOklzc3VlQ29tbWVudDYyMzczOTkzNA== | simonw 9599 | 2020-05-04T22:24:26Z | 2020-05-04T22:24:26Z | MEMBER | Twitter thread with some examples of photos that are coming up from queries against these scores: https://twitter.com/simonw/status/1257434670750408705 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Expose scores from ZCOMPUTEDASSETATTRIBUTES 612151767 | |
623730934 | https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623730934 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15 | MDEyOklzc3VlQ29tbWVudDYyMzczMDkzNA== | simonw 9599 | 2020-05-04T22:00:38Z | 2020-05-04T22:00:48Z | MEMBER | Here's the query to create the new table: ```sql create table apple_photos_scores as select ZGENERICASSET.ZUUID, ZGENERICASSET.ZOVERALLAESTHETICSCORE, ZGENERICASSET.ZCURATIONSCORE, ZGENERICASSET.ZPROMOTIONSCORE, ZGENERICASSET.ZHIGHLIGHTVISIBILITYSCORE, ZCOMPUTEDASSETATTRIBUTES.ZBEHAVIORALSCORE, ZCOMPUTEDASSETATTRIBUTES.ZFAILURESCORE, ZCOMPUTEDASSETATTRIBUTES.ZHARMONIOUSCOLORSCORE, ZCOMPUTEDASSETATTRIBUTES.ZIMMERSIVENESSSCORE, ZCOMPUTEDASSETATTRIBUTES.ZINTERACTIONSCORE, ZCOMPUTEDASSETATTRIBUTES.ZINTERESTINGSUBJECTSCORE, ZCOMPUTEDASSETATTRIBUTES.ZINTRUSIVEOBJECTPRESENCESCORE, ZCOMPUTEDASSETATTRIBUTES.ZLIVELYCOLORSCORE, ZCOMPUTEDASSETATTRIBUTES.ZLOWLIGHT, ZCOMPUTEDASSETATTRIBUTES.ZNOISESCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCAMERATILTSCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCOMPOSITIONSCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTLIGHTINGSCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPATTERNSCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPERSPECTIVESCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPOSTPROCESSINGSCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTREFLECTIONSSCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTSYMMETRYSCORE, ZCOMPUTEDASSETATTRIBUTES.ZSHARPLYFOCUSEDSUBJECTSCORE, ZCOMPUTEDASSETATTRIBUTES.ZTASTEFULLYBLURREDSCORE, ZCOMPUTEDASSETATTRIBUTES.ZWELLCHOSENSUBJECTSCORE, ZCOMPUTEDASSETATTRIBUTES.ZWELLFRAMEDSUBJECTSCORE, ZCOMPUTEDASSETATTRIBUTES.ZWELLTIMEDSHOTSCORE from attached.ZGENERICASSET join attached.ZCOMPUTEDASSETATTRIBUTES on attached.ZGENERICASSET.Z_PK = attached.ZCOMPUTEDASSETATTRIBUTES.Z_PK; ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Expose scores from ZCOMPUTEDASSETATTRIBUTES 612151767 | |
623723687 | https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723687 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15 | MDEyOklzc3VlQ29tbWVudDYyMzcyMzY4Nw== | simonw 9599 | 2020-05-04T21:43:06Z | 2020-05-04T21:43:06Z | MEMBER | It looks like I can map the photos I'm importing to these tables using the `ZUUID` column on `ZGENERICASSET` to get a `Z_PK` which then maps to the rows in `ZGENERICASSET`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Expose scores from ZCOMPUTEDASSETATTRIBUTES 612151767 | |
623723026 | https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723026 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15 | MDEyOklzc3VlQ29tbWVudDYyMzcyMzAyNg== | simonw 9599 | 2020-05-04T21:41:30Z | 2020-05-04T21:41:30Z | MEMBER | I'm going to put these in a table called `apple_photos_scores` - I'll also pull in the following columns from the `ZGENERICASSET` table: * `ZOVERALLAESTHETICSCORE` * `ZCURATIONSCORE` * `ZHIGHLIGHTVISIBILITYSCORE` * `ZPROMOTIONSCORE` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Expose scores from ZCOMPUTEDASSETATTRIBUTES 612151767 | |
623232984 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623232984 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzIzMjk4NA== | simonw 9599 | 2020-05-04T02:41:32Z | 2020-05-04T02:41:32Z | MEMBER | Needs documentation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623199750 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199750 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5OTc1MA== | simonw 9599 | 2020-05-03T23:17:58Z | 2020-05-03T23:17:58Z | MEMBER | Reading this source code is really useful for figuring out how to store a photo in a DB table: https://github.com/RhetTbull/osxphotos/blob/7444b6d173918a3ad2a07aefce5ecf054786c787/osxphotos/photoinfo.py | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623199701 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199701 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5OTcwMQ== | simonw 9599 | 2020-05-03T23:17:38Z | 2020-05-03T23:17:38Z | MEMBER | Record burst_uuid as a column: ``` (Pdb) with_bursts[0]._info["burstUUID"] '703FAA23-57BF-40B4-8A33-D9CEB143391B' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623199214 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199214 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5OTIxNA== | simonw 9599 | 2020-05-03T23:14:08Z | 2020-05-03T23:14:08Z | MEMBER | Albums have UUIDs: ``` (Pdb) photo.album_info[0].__dict__ {'_uuid': '17816791-ABF3-447B-942C-9FA8065EEBBA', '_db': osxphotos.PhotosDB(dbfile='/Users/simon/Pictures/Photos Library.photoslibrary/database/photos.db'), '_title': 'Geotaggable Photos geotagged'} ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623198986 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198986 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5ODk4Ng== | simonw 9599 | 2020-05-03T23:12:31Z | 2020-05-03T23:12:46Z | MEMBER | To get the taken date in UTC: ``` from datetime import timezone (Pdb) photo.date.astimezone(timezone.utc).isoformat() '2018-02-13T20:21:31.620000+00:00' (Pdb) photo.date.astimezone(timezone.utc).isoformat().split(".") ['2018-02-13T20:21:31', '620000+00:00'] (Pdb) photo.date.astimezone(timezone.utc).isoformat().split(".")[0] '2018-02-13T20:21:31' (Pdb) photo.date.astimezone(timezone.utc).isoformat().split(".")[0] + "+00:00" '2018-02-13T20:21:31+00:00' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623198653 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198653 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5ODY1Mw== | simonw 9599 | 2020-05-03T23:09:57Z | 2020-05-03T23:09:57Z | MEMBER | For locations: I'll add `place_x` columns for all of these: ``` (Pdb) photo.place.address._asdict() {'street': None, 'sub_locality': None, 'city': 'Loreto', 'sub_administrative_area': 'Loreto', 'state_province': 'BCS', 'postal_code': None, 'country': 'Mexico', 'iso_country_code': 'MX'} ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623195197 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623195197 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5NTE5Nw== | simonw 9599 | 2020-05-03T22:44:33Z | 2020-05-03T22:44:33Z | MEMBER | Command will be this: $ photos-to-sqlite apple-photos photos.db This will populate a `apple_photos` table with the data imported by the `osxphotos` library, plus the calculated sha256. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623193947 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623193947 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5Mzk0Nw== | simonw 9599 | 2020-05-03T22:36:17Z | 2020-05-03T22:36:17Z | MEMBER | I'm going to use [osxphotos](https://github.com/RhetTbull/osxphotos) for this. Since I've already got code to upload photos and insert them into a table based on their `sha256` hash, my first go at this will be to import data using the tool and foreign-key it to the `sha256` hash in the existing table. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623038378 | https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038378 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38 | MDEyOklzc3VlQ29tbWVudDYyMzAzODM3OA== | simonw 9599 | 2020-05-03T01:21:13Z | 2020-05-03T01:21:13Z | MEMBER | No this is really useful feedback! I'm so close to this project that I miss what's not obvious to people dropping in for the first time. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | [Feature Request] Support Repo Name in Search 🥺 611284481 | |
623027889 | https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623027889 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38 | MDEyOklzc3VlQ29tbWVudDYyMzAyNzg4OQ== | simonw 9599 | 2020-05-02T23:15:11Z | 2020-05-02T23:15:11Z | MEMBER | This is one of the use-cases for the `repos_starred` view: it allows you to easily run this kid of query without having to construct the SQL by hand. Here's a demo: https://github-to-sqlite.dogsheep.net/github/repos_starred?name__contains=twitter My philosophy here is to keep the raw tables (like `stars`) as normalized as possible, then use SQL views which expose the data in a form that's easier to query. | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | [Feature Request] Support Repo Name in Search 🥺 611284481 | |
623010272 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623010272 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDYyMzAxMDI3Mg== | simonw 9599 | 2020-05-02T20:39:14Z | 2020-05-02T20:39:14Z | MEMBER | Graph of cumulative stars for Datasette over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29#g.mark=line&g.x_column=yyyymmdd&g.x_type=temporal&g.y_column=cumulative_count&g.y_type=quantitative <img width="291" alt="github__select_yyyymmdd__sum_n__over___order_by_yyyymmdd_rows_unbounded_preceding___as_cumulative_count_from___select_substr_starred_at__0__11__as_yyyymmdd__count____as_n_from_stars_where_repo___107914493_group_by_yyyymmdd__" src="https://user-images.githubusercontent.com/9599/80891642-31587180-8c7a-11ea-9169-60eb3cb6d3e0.png"> Stars per day (as a label bar chart, so very wide): https://github-to-sqlite.dogsheep.net/github?sql=%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++#g.mark=bar&g.x_column=yyyymmdd&g.x_type=ordinal&g.y_column=n&g.y_type=quantitative <img width="1424" alt="github__select_substr_starred_at__0__11__as_yyyymmdd__count____as_n_from_stars_where_repo___107914493_group_by_yyyymmdd" src="https://user-images.githubusercontent.com/9599/80891656-4e8d4000-8c7a-11ea-9ff2-02c9cf4cb5d5.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
623007441 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623007441 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDYyMzAwNzQ0MQ== | simonw 9599 | 2020-05-02T20:13:37Z | 2020-05-02T20:13:37Z | MEMBER | Datasette cumulative stars over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
623006154 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623006154 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDYyMzAwNjE1NA== | simonw 9599 | 2020-05-02T20:01:39Z | 2020-05-02T20:01:54Z | MEMBER | Needs tests and documentation. I shipped it early to check that the live demo works. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
623006004 | https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-623006004 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDYyMzAwNjAwNA== | simonw 9599 | 2020-05-02T20:00:26Z | 2020-05-02T20:00:26Z | MEMBER | I'm abandoning this in favour of a new implementation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | stargazers command, refs #4 516763727 | |
623004836 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623004836 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDYyMzAwNDgzNg== | simonw 9599 | 2020-05-02T19:49:51Z | 2020-05-02T19:49:51Z | MEMBER | Alternative pattern: ``` sqlite-utils releases.db 'select full_name from repos' --csv --no-headers \ | tr -d '\r' \ | xargs github-to-sqlite stargazers stars.db ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
623000814 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623000814 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDYyMzAwMDgxNA== | simonw 9599 | 2020-05-02T19:15:23Z | 2020-05-02T19:15:23Z | MEMBER | I'm not going to do the `--sql` bit just yet. I have patterns for working around this for other commands which are working fine: https://github.com/dogsheep/github-to-sqlite/blob/d00a53061556dc403c166b443d141c4e1adbd64a/.github/workflows/deploy-demo.yml#L53-L70 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
622998813 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw== | simonw 9599 | 2020-05-02T18:58:17Z | 2020-05-02T18:58:17Z | MEMBER | Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
622997410 | https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622997410 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33 | MDEyOklzc3VlQ29tbWVudDYyMjk5NzQxMA== | simonw 9599 | 2020-05-02T18:46:10Z | 2020-05-02T18:46:10Z | MEMBER | Documented here: https://github.com/dogsheep/github-to-sqlite/blob/10fb34de41aaa35681f08b5991540d65bfcf2e2e/README.md#authentication | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fall back to authentication via ENV 609950090 | |
622990947 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-622990947 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDYyMjk5MDk0Nw== | simonw 9599 | 2020-05-02T17:54:16Z | 2020-05-02T17:54:16Z | MEMBER | I could add that window function query as a view, but only if the detected version of SQLite supports window functions. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
622989874 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA== | simonw 9599 | 2020-05-02T17:46:14Z | 2020-05-02T17:46:14Z | MEMBER | Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
622982667 | https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622982667 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYyMjk4MjY2Nw== | simonw 9599 | 2020-05-02T16:52:53Z | 2020-05-02T16:52:53Z | MEMBER | Easiest option: use `db.index_foreign_keys()`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#adding-indexes-for-all-foreign-keys | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create index on issue_comments(user) and other foreign keys 610511450 | |
622982346 | https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622982346 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36 | MDEyOklzc3VlQ29tbWVudDYyMjk4MjM0Ng== | simonw 9599 | 2020-05-02T16:50:31Z | 2020-05-02T16:50:31Z | MEMBER | Demo: https://github-to-sqlite.dogsheep.net/github/dependent_repos | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add view for better display of dependent repos 610842926 | |
622980203 | https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622980203 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDYyMjk4MDIwMw== | simonw 9599 | 2020-05-02T16:34:29Z | 2020-05-02T16:34:29Z | MEMBER | Fixed definition: ```sql select stars.starred_at, starring_user.login as starred_by, repos.* from repos join stars on repos.id = stars.repo join users as starring_user on stars.user = starring_user.id join users on repos.owner = users.id order by starred_at desc; ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this repos_starred view 516967682 | |
622978173 | https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622978173 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37 | MDEyOklzc3VlQ29tbWVudDYyMjk3ODE3Mw== | simonw 9599 | 2020-05-02T16:19:31Z | 2020-05-02T16:19:47Z | MEMBER | I can use the new `.create_view(..., replace=True)` parameter in `sqlite-utils` 2.7.2 for this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for creating views if they don't yet exist 610843136 | |
622461948 | https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461948 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTk0OA== | simonw 9599 | 2020-05-01T16:36:42Z | 2020-05-01T16:36:42Z | MEMBER | It should only create views if the underlying tables exist. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for creating views if they don't yet exist 610843136 | |
622461537 | https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461537 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTUzNw== | simonw 9599 | 2020-05-01T16:35:40Z | 2020-05-01T16:35:40Z | MEMBER | This will check if the view exists and has the exact same matching definition as the one we want. If it doesn't, we will drop it (if it exists) and recreate it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for creating views if they don't yet exist 610843136 | |
622461223 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw== | simonw 9599 | 2020-05-01T16:34:52Z | 2020-05-01T16:34:52Z | MEMBER | Blocked on #37 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
622461122 | https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622461122 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTEyMg== | simonw 9599 | 2020-05-01T16:34:39Z | 2020-05-01T16:34:39Z | MEMBER | Blocked on #37 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this repos_starred view 516967682 | |
622461025 | https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622461025 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTAyNQ== | simonw 9599 | 2020-05-01T16:34:24Z | 2020-05-01T16:34:24Z | MEMBER | Blocked on #37 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add view for better display of dependent repos 610842926 | |
622214262 | https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622214262 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYyMjIxNDI2Mg== | simonw 9599 | 2020-05-01T02:10:32Z | 2020-05-01T02:11:19Z | MEMBER | This sped that query up even more - down to 4ms. ```sql create index issue_comments_issue on issue_comments(issue); ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create index on issue_comments(user) and other foreign keys 610511450 | |
622213950 | https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622213950 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYyMjIxMzk1MA== | simonw 9599 | 2020-05-01T02:09:04Z | 2020-05-01T02:09:04Z | MEMBER | It sped up this query a lot - 2.5s down to 300ms: ```sql select repos.full_name, json_object( 'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number, 'label', '#' || issues.number ) as issue, issues.title, users.login, users.id, issues.state, issues.locked, issues.assignee, issues.milestone, issues.comments, issues.created_at, issues.updated_at, issues.closed_at, issues.author_association, issues.pull_request, issues.repo, issues.type from issues join repos on repos.id = issues.repo join users on issues.user = users.id where issues.state = 'open' and issues.user not in (9599, 27856297) and not exists ( select id from issue_comments where issue_comments.user = 9599 and issues.id = issue_comments.issue ) order by issues.updated_at desc; ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create index on issue_comments(user) and other foreign keys 610511450 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
updated_at (date) >30 ✖