html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617348174,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,617348174,MDEyOklzc3VlQ29tbWVudDYxNzM0ODE3NA==,9599,2020-04-21T18:50:29Z,2020-04-21T18:50:29Z,MEMBER,"Since this represents a breaking schema change for anyone running SQL queries against these tables, I'm going to do a major version bump to 2.0 when I release this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862, https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617364956,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617364956,MDEyOklzc3VlQ29tbWVudDYxNzM2NDk1Ng==,9599,2020-04-21T19:24:45Z,2020-04-21T19:24:45Z,MEMBER,"That's because I just broke this code: https://github.com/dogsheep/github-to-sqlite/blob/2cf75a0a036719eb7e57fdc7c5c2ea0f4c26978a/github_to_sqlite/utils.py#L131-L139 It expects the `repo` column to be `simonw/datasette` but it's now an ID instead. I should add a test for this as part of the fix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295, https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617369247,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617369247,MDEyOklzc3VlQ29tbWVudDYxNzM2OTI0Nw==,9599,2020-04-21T19:33:03Z,2020-04-21T19:33:03Z,MEMBER,Caused by #31.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295, https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617490914,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617490914,MDEyOklzc3VlQ29tbWVudDYxNzQ5MDkxNA==,9599,2020-04-22T01:17:44Z,2020-04-22T01:17:44Z,MEMBER,"https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101 now shows issues. And https://github-to-sqlite.dogsheep.net/github/issue_comments links to them: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295, https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617491607,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,617491607,MDEyOklzc3VlQ29tbWVudDYxNzQ5MTYwNw==,9599,2020-04-22T01:20:19Z,2020-04-22T01:20:19Z,MEMBER,"https://github-to-sqlite.dogsheep.net/github/milestones now link to repo: And so do issues: https://github-to-sqlite.dogsheep.net/github/issues ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862, https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100434,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8,618100434,MDEyOklzc3VlQ29tbWVudDYxODEwMDQzNA==,9599,2020-04-23T00:02:53Z,2020-04-23T00:02:53Z,MEMBER,"I don't think it matters one way or the other - I'm storing the sha256 in the filename, so the fact that I could read the MD5 back from the list bucket operation doesn't give me any benefits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605147638, https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100658,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8,618100658,MDEyOklzc3VlQ29tbWVudDYxODEwMDY1OA==,9599,2020-04-23T00:03:35Z,2020-04-23T00:03:35Z,MEMBER,"Also MD5 isn't guaranteed for the ETag: > If an object is created by either the Multipart Upload or Part Copy operation, the ETag is not an MD5 digest, regardless of the method of encryption. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605147638, https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618724149,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9,618724149,MDEyOklzc3VlQ29tbWVudDYxODcyNDE0OQ==,9599,2020-04-23T23:35:29Z,2020-04-23T23:35:29Z,MEMBER,"``` % photos-to-sqlite upload photos.db ~/Pictures/Photos\ Library.photoslibrary/originals Fetching existing keys from S3... Got 22,446 existing keys Calculating hashes [####--------------------------------] 13% 00:04:14 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605938063, https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618725155,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9,618725155,MDEyOklzc3VlQ29tbWVudDYxODcyNTE1NQ==,9599,2020-04-23T23:39:14Z,2020-04-23T23:39:14Z,MEMBER,"A few minutes later... ``` Fetching existing keys from S3... Got 22,446 existing keys Calculating hashes [####################################] 100% 22,441 hashed files, 610 are not yet in S3 Uploading 0.99 GB Uploading 610 photos [------------------------------------] 1/610 03:10:35 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605938063, https://github.com/dogsheep/dogsheep-photos/issues/12#issuecomment-618796564,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/12,618796564,MDEyOklzc3VlQ29tbWVudDYxODc5NjU2NA==,9599,2020-04-24T04:35:25Z,2020-04-24T04:35:25Z,MEMBER,Code: https://github.com/dogsheep/photos-to-sqlite/blob/a388cf1f1b6b67752d669466cda8b171b6582171/photos_to_sqlite/cli.py#L109-L114,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",606033104, https://github.com/dogsheep/dogsheep-photos/issues/13#issuecomment-620273692,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13,620273692,MDEyOklzc3VlQ29tbWVudDYyMDI3MzY5Mg==,9599,2020-04-27T22:42:50Z,2020-04-27T22:42:50Z,MEMBER,"``` >>> def ext_counts(directory): ... counts = {} ... for path in pathlib.Path(directory).glob(""**/*""): ... ext = path.suffix ... counts[ext] = counts.get(ext, 0) + 1 ... return counts ... >>> >>> ext_counts(""/Users/simon/Pictures/Photos Library.photoslibrary/originals"") {'': 16, '.heic': 15478, '.jpeg': 21691, '.mov': 946, '.png': 2262, '.gif': 38, '.mp4': 116, '.aae': 2} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607888367, https://github.com/dogsheep/dogsheep-photos/issues/13#issuecomment-620309185,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13,620309185,MDEyOklzc3VlQ29tbWVudDYyMDMwOTE4NQ==,9599,2020-04-28T00:39:45Z,2020-04-28T00:39:45Z,MEMBER,I'm going to leave this until I have the mechanism for associating a live photo video with the photo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607888367, https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620769348,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620769348,MDEyOklzc3VlQ29tbWVudDYyMDc2OTM0OA==,9599,2020-04-28T18:09:21Z,2020-04-28T18:09:21Z,MEMBER,"Pricing is pretty good: free for first 1,000 calls per month, then $1.50 per thousand after that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747, https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620771067,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620771067,MDEyOklzc3VlQ29tbWVudDYyMDc3MTA2Nw==,9599,2020-04-28T18:12:34Z,2020-04-28T18:15:38Z,MEMBER,"Python library docs: https://googleapis.dev/python/vision/latest/index.html I'm creating a new project for this called simonwillison-photos: https://console.cloud.google.com/projectcreate https://console.cloud.google.com/home/dashboard?project=simonwillison-photos Then I enabled the Vision API. The direct link to https://console.cloud.google.com/flows/enableapi?apiid=vision-json.googleapis.com which they provided in the docs didn't work - it gave me a ""You don't have sufficient permissions to use the requested API"" error - but starting at the ""Enable APIs"" page and searching for it worked fine. I created a new service account as an ""owner"" of that project: https://console.cloud.google.com/apis/credentials/serviceaccountkey (and complained about it on Twitter and through their feedback form) `pip install google-cloud-vision` ```python from google.cloud import vision client = vision.ImageAnnotatorClient.from_service_account_file(""simonwillison-photos-18c570b301fe.json"") # Photo of a lemur response = client.annotate_image( { ""image"": { ""source"": { ""image_uri"": ""https://photos.simonwillison.net/i/1b3414ee9ade67ce04ade9042e6d4b433d1e523c9a16af17f490e2c0a619755b.jpeg"" } }, ""features"": [ {""type"": vision.enums.Feature.Type.IMAGE_PROPERTIES}, {""type"": vision.enums.Feature.Type.OBJECT_LOCALIZATION}, {""type"": vision.enums.Feature.Type.LABEL_DETECTION}, ], } ) response ``` Output is: ``` label_annotations { mid: ""/m/09686"" description: ""Vertebrate"" score: 0.9851104021072388 topicality: 0.9851104021072388 } label_annotations { mid: ""/m/04rky"" description: ""Mammal"" score: 0.975814163684845 topicality: 0.975814163684845 } label_annotations { mid: ""/m/01280g"" description: ""Wildlife"" score: 0.8973650336265564 topicality: 0.8973650336265564 } label_annotations { mid: ""/m/02f9pk"" description: ""Lemur"" score: 0.8270352482795715 topicality: 0.8270352482795715 } label_annotations { mid: ""/m/0fbf1m"" description: ""Terrestrial animal"" score: 0.7443860769271851 topicality: 0.7443860769271851 } label_annotations { mid: ""/m/06z_nw"" description: ""Tail"" score: 0.6934166550636292 topicality: 0.6934166550636292 } label_annotations { mid: ""/m/0b5gs"" description: ""Branch"" score: 0.6203985214233398 topicality: 0.6203985214233398 } label_annotations { mid: ""/m/05s2s"" description: ""Plant"" score: 0.585474967956543 topicality: 0.585474967956543 } label_annotations { mid: ""/m/089v3"" description: ""Zoo"" score: 0.5488107800483704 topicality: 0.5488107800483704 } label_annotations { mid: ""/m/02tcwp"" description: ""Trunk"" score: 0.5200017690658569 topicality: 0.5200017690658569 } image_properties_annotation { dominant_colors { colors { color { red: 172.0 green: 146.0 blue: 116.0 } score: 0.24523821473121643 pixel_fraction: 0.027533333748579025 } colors { color { red: 54.0 green: 50.0 blue: 42.0 } score: 0.10449723154306412 pixel_fraction: 0.12893334031105042 } colors { color { red: 141.0 green: 121.0 blue: 97.0 } score: 0.1391485631465912 pixel_fraction: 0.039133332669734955 } colors { color { red: 28.0 green: 25.0 blue: 20.0 } score: 0.08589499443769455 pixel_fraction: 0.11506666988134384 } colors { color { red: 87.0 green: 82.0 blue: 74.0 } score: 0.0845794677734375 pixel_fraction: 0.16113333404064178 } colors { color { red: 121.0 green: 117.0 blue: 108.0 } score: 0.05901569500565529 pixel_fraction: 0.13379999995231628 } colors { color { red: 94.0 green: 83.0 blue: 66.0 } score: 0.049011144787073135 pixel_fraction: 0.03946666792035103 } colors { color { red: 155.0 green: 117.0 blue: 90.0 } score: 0.04164913296699524 pixel_fraction: 0.0023333332501351833 } colors { color { red: 178.0 green: 143.0 blue: 102.0 } score: 0.02993861958384514 pixel_fraction: 0.0012666666880249977 } colors { color { red: 61.0 green: 51.0 blue: 35.0 } score: 0.027391711249947548 pixel_fraction: 0.01953333243727684 } } } crop_hints_annotation { crop_hints { bounding_poly { vertices { x: 2073 } vertices { x: 4008 } vertices { x: 4008 y: 3455 } vertices { x: 2073 y: 3455 } } confidence: 0.65625 importance_fraction: 0.746666669845581 } } localized_object_annotations { mid: ""/m/0jbk"" name: ""Animal"" score: 0.7008256912231445 bounding_poly { normalized_vertices { x: 0.0390297956764698 y: 0.26235100626945496 } normalized_vertices { x: 0.8466796875 y: 0.26235100626945496 } normalized_vertices { x: 0.8466796875 y: 0.9386426210403442 } normalized_vertices { x: 0.0390297956764698 y: 0.9386426210403442 } } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747, https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620771698,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620771698,MDEyOklzc3VlQ29tbWVudDYyMDc3MTY5OA==,9599,2020-04-28T18:13:48Z,2020-04-28T18:13:48Z,MEMBER,"For face detection: ``` {""type"": vision.enums.Feature.Type.Type.FACE_DETECTION} ``` For OCR: ``` {""type"": vision.enums.Feature.Type.DOCUMENT_TEXT_DETECTION} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747, https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620772190,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620772190,MDEyOklzc3VlQ29tbWVudDYyMDc3MjE5MA==,9599,2020-04-28T18:14:43Z,2020-04-28T18:14:43Z,MEMBER,"Database schema for this will require some thought. Just dumping the output into a JSON column isn't going to be flexible enough - I want to be able to FTS against labels and OCR text, and potentially query against other characteristics too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747, https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620774507,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620774507,MDEyOklzc3VlQ29tbWVudDYyMDc3NDUwNw==,9599,2020-04-28T18:19:06Z,2020-04-28T18:19:06Z,MEMBER,"The default timeout is a bit aggressive and sometimes failed for me if my resizing proxy took too long to fetch and resize the image. `client.annotate_image(..., timeout=3.0)` may be worth trying.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747, https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133298,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622133298,MDEyOklzc3VlQ29tbWVudDYyMjEzMzI5OA==,9599,2020-04-30T21:48:24Z,2020-04-30T21:48:24Z,MEMBER,"Unfortunately it's not available through any GitHub API - I managed to figure out how to get dependencies, but I need dependents. https://github.com/simonw/til/blob/master/github/dependencies-graphql-api.md","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908, https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133422,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622133422,MDEyOklzc3VlQ29tbWVudDYyMjEzMzQyMg==,9599,2020-04-30T21:48:39Z,2020-04-30T21:48:39Z,MEMBER,It looks like the only option is to scrape them. I'll do that and then replace with an API as soon as one becomes available.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908, https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133775,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622133775,MDEyOklzc3VlQ29tbWVudDYyMjEzMzc3NQ==,9599,2020-04-30T21:49:27Z,2020-04-30T21:49:27Z,MEMBER,"Proposed command: github-to-sqlite scrape-dependents github.db simonw/datasette I'll pull full details of the scraped repos from the regular API. I'll also record when they were ""first seen"" by the command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908, https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622135654,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622135654,MDEyOklzc3VlQ29tbWVudDYyMjEzNTY1NA==,9599,2020-04-30T21:53:44Z,2020-04-30T21:56:06Z,MEMBER,"I think this is the neatest scraping pattern: ```python [a[""href""].lstrip(""/"") for a in soup.select(""a[data-hovercard-type=repository]"")] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908, https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622136585,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622136585,MDEyOklzc3VlQ29tbWVudDYyMjEzNjU4NQ==,9599,2020-04-30T21:55:51Z,2020-04-30T21:55:51Z,MEMBER,"And to find the ""Next"" pagination link: ```python soup.select("".paginate-container"")[0].find(""a"", text=""Next"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908, https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622162835,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622162835,MDEyOklzc3VlQ29tbWVudDYyMjE2MjgzNQ==,9599,2020-04-30T22:59:26Z,2020-04-30T22:59:26Z,MEMBER,Documentation: https://github.com/dogsheep/github-to-sqlite/blob/c9f48404481882e8b3af06f35e4801a80ac79ed6/README.md#scraping-dependents-for-a-repository,"{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 2, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908, https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622169728,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622169728,MDEyOklzc3VlQ29tbWVudDYyMjE2OTcyOA==,9599,2020-04-30T23:18:51Z,2020-04-30T23:18:51Z,MEMBER,"Sure, that sounds fine to me.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090, https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622171097,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622171097,MDEyOklzc3VlQ29tbWVudDYyMjE3MTA5Nw==,9599,2020-04-30T23:22:45Z,2020-04-30T23:23:57Z,MEMBER,"The `auth.json` mechanism this uses is standard across all of the other Dogsheep tools - it's actually designed so you can have one `auth.json` with a bunch of different credentials for different tools: ```json { ""goodreads_personal_token"": ""..."", ""goodreads_user_id"": ""..."", ""github_personal_token"": ""..."", ""pocket_consumer_key"": ""..."", ""pocket_username"": ""..."", ""pocket_access_token"": ""..."" } ``` But... `github-to-sqlite` does feel like it deserves a special case here, since it's such a good fit for running inside of GitHub Actions - which even provide a `GITHUB_TOKEN` for you to use! So I don't think it will harm the family of tools too much if this has an environment variable alternative to the `-a` file.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090, https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622213950,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622213950,MDEyOklzc3VlQ29tbWVudDYyMjIxMzk1MA==,9599,2020-05-01T02:09:04Z,2020-05-01T02:09:04Z,MEMBER,"It sped up this query a lot - 2.5s down to 300ms: ```sql select repos.full_name, json_object( 'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number, 'label', '#' || issues.number ) as issue, issues.title, users.login, users.id, issues.state, issues.locked, issues.assignee, issues.milestone, issues.comments, issues.created_at, issues.updated_at, issues.closed_at, issues.author_association, issues.pull_request, issues.repo, issues.type from issues join repos on repos.id = issues.repo join users on issues.user = users.id where issues.state = 'open' and issues.user not in (9599, 27856297) and not exists ( select id from issue_comments where issue_comments.user = 9599 and issues.id = issue_comments.issue ) order by issues.updated_at desc; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450, https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622214262,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622214262,MDEyOklzc3VlQ29tbWVudDYyMjIxNDI2Mg==,9599,2020-05-01T02:10:32Z,2020-05-01T02:11:19Z,MEMBER,"This sped that query up even more - down to 4ms. ```sql create index issue_comments_issue on issue_comments(issue); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450, https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622461025,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36,622461025,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTAyNQ==,9599,2020-05-01T16:34:24Z,2020-05-01T16:34:24Z,MEMBER,Blocked on #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610842926, https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622461122,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10,622461122,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTEyMg==,9599,2020-05-01T16:34:39Z,2020-05-01T16:34:39Z,MEMBER,Blocked on #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516967682, https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,622461223,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw==,9599,2020-05-01T16:34:52Z,2020-05-01T16:34:52Z,MEMBER,Blocked on #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546, https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461537,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622461537,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTUzNw==,9599,2020-05-01T16:35:40Z,2020-05-01T16:35:40Z,MEMBER,"This will check if the view exists and has the exact same matching definition as the one we want. If it doesn't, we will drop it (if it exists) and recreate it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136, https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461948,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622461948,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTk0OA==,9599,2020-05-01T16:36:42Z,2020-05-01T16:36:42Z,MEMBER,It should only create views if the underlying tables exist.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136, https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622978173,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622978173,MDEyOklzc3VlQ29tbWVudDYyMjk3ODE3Mw==,9599,2020-05-02T16:19:31Z,2020-05-02T16:19:47Z,MEMBER,"I can use the new `.create_view(..., replace=True)` parameter in `sqlite-utils` 2.7.2 for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136, https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622980203,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10,622980203,MDEyOklzc3VlQ29tbWVudDYyMjk4MDIwMw==,9599,2020-05-02T16:34:29Z,2020-05-02T16:34:29Z,MEMBER,"Fixed definition: ```sql select stars.starred_at, starring_user.login as starred_by, repos.* from repos join stars on repos.id = stars.repo join users as starring_user on stars.user = starring_user.id join users on repos.owner = users.id order by starred_at desc; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516967682, https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622982346,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36,622982346,MDEyOklzc3VlQ29tbWVudDYyMjk4MjM0Ng==,9599,2020-05-02T16:50:31Z,2020-05-02T16:50:31Z,MEMBER,Demo: https://github-to-sqlite.dogsheep.net/github/dependent_repos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610842926, https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622982667,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622982667,MDEyOklzc3VlQ29tbWVudDYyMjk4MjY2Nw==,9599,2020-05-02T16:52:53Z,2020-05-02T16:52:53Z,MEMBER,Easiest option: use `db.index_foreign_keys()`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#adding-indexes-for-all-foreign-keys,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450, https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,622989874,MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA==,9599,2020-05-02T17:46:14Z,2020-05-02T17:46:14Z,MEMBER,Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546, https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-622990947,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,622990947,MDEyOklzc3VlQ29tbWVudDYyMjk5MDk0Nw==,9599,2020-05-02T17:54:16Z,2020-05-02T17:54:16Z,MEMBER,"I could add that window function query as a view, but only if the detected version of SQLite supports window functions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730, https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622997410,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622997410,MDEyOklzc3VlQ29tbWVudDYyMjk5NzQxMA==,9599,2020-05-02T18:46:10Z,2020-05-02T18:46:10Z,MEMBER,Documented here: https://github.com/dogsheep/github-to-sqlite/blob/10fb34de41aaa35681f08b5991540d65bfcf2e2e/README.md#authentication,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090, https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,622998813,MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw==,9599,2020-05-02T18:58:17Z,2020-05-02T18:58:17Z,MEMBER,Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546, https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623000814,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623000814,MDEyOklzc3VlQ29tbWVudDYyMzAwMDgxNA==,9599,2020-05-02T19:15:23Z,2020-05-02T19:15:23Z,MEMBER,"I'm not going to do the `--sql` bit just yet. I have patterns for working around this for other commands which are working fine: https://github.com/dogsheep/github-to-sqlite/blob/d00a53061556dc403c166b443d141c4e1adbd64a/.github/workflows/deploy-demo.yml#L53-L70","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730, https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623004836,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623004836,MDEyOklzc3VlQ29tbWVudDYyMzAwNDgzNg==,9599,2020-05-02T19:49:51Z,2020-05-02T19:49:51Z,MEMBER,"Alternative pattern: ``` sqlite-utils releases.db 'select full_name from repos' --csv --no-headers \ | tr -d '\r' \ | xargs github-to-sqlite stargazers stars.db ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730, https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-623006004,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8,623006004,MDEyOklzc3VlQ29tbWVudDYyMzAwNjAwNA==,9599,2020-05-02T20:00:26Z,2020-05-02T20:00:26Z,MEMBER,I'm abandoning this in favour of a new implementation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516763727, https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623006154,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623006154,MDEyOklzc3VlQ29tbWVudDYyMzAwNjE1NA==,9599,2020-05-02T20:01:39Z,2020-05-02T20:01:54Z,MEMBER,Needs tests and documentation. I shipped it early to check that the live demo works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730, https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623007441,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623007441,MDEyOklzc3VlQ29tbWVudDYyMzAwNzQ0MQ==,9599,2020-05-02T20:13:37Z,2020-05-02T20:13:37Z,MEMBER,Datasette cumulative stars over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730, https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623010272,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623010272,MDEyOklzc3VlQ29tbWVudDYyMzAxMDI3Mg==,9599,2020-05-02T20:39:14Z,2020-05-02T20:39:14Z,MEMBER,"Graph of cumulative stars for Datasette over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29#g.mark=line&g.x_column=yyyymmdd&g.x_type=temporal&g.y_column=cumulative_count&g.y_type=quantitative Stars per day (as a label bar chart, so very wide): https://github-to-sqlite.dogsheep.net/github?sql=%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++#g.mark=bar&g.x_column=yyyymmdd&g.x_type=ordinal&g.y_column=n&g.y_type=quantitative ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730, https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623027889,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623027889,MDEyOklzc3VlQ29tbWVudDYyMzAyNzg4OQ==,9599,2020-05-02T23:15:11Z,2020-05-02T23:15:11Z,MEMBER,"This is one of the use-cases for the `repos_starred` view: it allows you to easily run this kid of query without having to construct the SQL by hand. Here's a demo: https://github-to-sqlite.dogsheep.net/github/repos_starred?name__contains=twitter My philosophy here is to keep the raw tables (like `stars`) as normalized as possible, then use SQL views which expose the data in a form that's easier to query.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481, https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038378,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623038378,MDEyOklzc3VlQ29tbWVudDYyMzAzODM3OA==,9599,2020-05-03T01:21:13Z,2020-05-03T01:21:13Z,MEMBER,No this is really useful feedback! I'm so close to this project that I miss what's not obvious to people dropping in for the first time.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481, https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623193947,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623193947,MDEyOklzc3VlQ29tbWVudDYyMzE5Mzk0Nw==,9599,2020-05-03T22:36:17Z,2020-05-03T22:36:17Z,MEMBER,"I'm going to use [osxphotos](https://github.com/RhetTbull/osxphotos) for this. Since I've already got code to upload photos and insert them into a table based on their `sha256` hash, my first go at this will be to import data using the tool and foreign-key it to the `sha256` hash in the existing table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300, https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623195197,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623195197,MDEyOklzc3VlQ29tbWVudDYyMzE5NTE5Nw==,9599,2020-05-03T22:44:33Z,2020-05-03T22:44:33Z,MEMBER,"Command will be this: $ photos-to-sqlite apple-photos photos.db This will populate a `apple_photos` table with the data imported by the `osxphotos` library, plus the calculated sha256.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300, https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198653,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623198653,MDEyOklzc3VlQ29tbWVudDYyMzE5ODY1Mw==,9599,2020-05-03T23:09:57Z,2020-05-03T23:09:57Z,MEMBER,"For locations: I'll add `place_x` columns for all of these: ``` (Pdb) photo.place.address._asdict() {'street': None, 'sub_locality': None, 'city': 'Loreto', 'sub_administrative_area': 'Loreto', 'state_province': 'BCS', 'postal_code': None, 'country': 'Mexico', 'iso_country_code': 'MX'} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300, https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198986,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623198986,MDEyOklzc3VlQ29tbWVudDYyMzE5ODk4Ng==,9599,2020-05-03T23:12:31Z,2020-05-03T23:12:46Z,MEMBER,"To get the taken date in UTC: ``` from datetime import timezone (Pdb) photo.date.astimezone(timezone.utc).isoformat() '2018-02-13T20:21:31.620000+00:00' (Pdb) photo.date.astimezone(timezone.utc).isoformat().split(""."") ['2018-02-13T20:21:31', '620000+00:00'] (Pdb) photo.date.astimezone(timezone.utc).isoformat().split(""."")[0] '2018-02-13T20:21:31' (Pdb) photo.date.astimezone(timezone.utc).isoformat().split(""."")[0] + ""+00:00"" '2018-02-13T20:21:31+00:00' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300, https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199214,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623199214,MDEyOklzc3VlQ29tbWVudDYyMzE5OTIxNA==,9599,2020-05-03T23:14:08Z,2020-05-03T23:14:08Z,MEMBER,"Albums have UUIDs: ``` (Pdb) photo.album_info[0].__dict__ {'_uuid': '17816791-ABF3-447B-942C-9FA8065EEBBA', '_db': osxphotos.PhotosDB(dbfile='/Users/simon/Pictures/Photos Library.photoslibrary/database/photos.db'), '_title': 'Geotaggable Photos geotagged'} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300, https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199701,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623199701,MDEyOklzc3VlQ29tbWVudDYyMzE5OTcwMQ==,9599,2020-05-03T23:17:38Z,2020-05-03T23:17:38Z,MEMBER,"Record burst_uuid as a column: ``` (Pdb) with_bursts[0]._info[""burstUUID""] '703FAA23-57BF-40B4-8A33-D9CEB143391B' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300, https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199750,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623199750,MDEyOklzc3VlQ29tbWVudDYyMzE5OTc1MA==,9599,2020-05-03T23:17:58Z,2020-05-03T23:17:58Z,MEMBER,Reading this source code is really useful for figuring out how to store a photo in a DB table: https://github.com/RhetTbull/osxphotos/blob/7444b6d173918a3ad2a07aefce5ecf054786c787/osxphotos/photoinfo.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300, https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623232984,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623232984,MDEyOklzc3VlQ29tbWVudDYyMzIzMjk4NA==,9599,2020-05-04T02:41:32Z,2020-05-04T02:41:32Z,MEMBER,Needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300, https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723026,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,623723026,MDEyOklzc3VlQ29tbWVudDYyMzcyMzAyNg==,9599,2020-05-04T21:41:30Z,2020-05-04T21:41:30Z,MEMBER,"I'm going to put these in a table called `apple_photos_scores` - I'll also pull in the following columns from the `ZGENERICASSET` table: * `ZOVERALLAESTHETICSCORE` * `ZCURATIONSCORE` * `ZHIGHLIGHTVISIBILITYSCORE` * `ZPROMOTIONSCORE`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767, https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723687,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,623723687,MDEyOklzc3VlQ29tbWVudDYyMzcyMzY4Nw==,9599,2020-05-04T21:43:06Z,2020-05-04T21:43:06Z,MEMBER,It looks like I can map the photos I'm importing to these tables using the `ZUUID` column on `ZGENERICASSET` to get a `Z_PK` which then maps to the rows in `ZGENERICASSET`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767, https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623730934,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,623730934,MDEyOklzc3VlQ29tbWVudDYyMzczMDkzNA==,9599,2020-05-04T22:00:38Z,2020-05-04T22:00:48Z,MEMBER,"Here's the query to create the new table: ```sql create table apple_photos_scores as select ZGENERICASSET.ZUUID, ZGENERICASSET.ZOVERALLAESTHETICSCORE, ZGENERICASSET.ZCURATIONSCORE, ZGENERICASSET.ZPROMOTIONSCORE, ZGENERICASSET.ZHIGHLIGHTVISIBILITYSCORE, ZCOMPUTEDASSETATTRIBUTES.ZBEHAVIORALSCORE, ZCOMPUTEDASSETATTRIBUTES.ZFAILURESCORE, ZCOMPUTEDASSETATTRIBUTES.ZHARMONIOUSCOLORSCORE, ZCOMPUTEDASSETATTRIBUTES.ZIMMERSIVENESSSCORE, ZCOMPUTEDASSETATTRIBUTES.ZINTERACTIONSCORE, ZCOMPUTEDASSETATTRIBUTES.ZINTERESTINGSUBJECTSCORE, ZCOMPUTEDASSETATTRIBUTES.ZINTRUSIVEOBJECTPRESENCESCORE, ZCOMPUTEDASSETATTRIBUTES.ZLIVELYCOLORSCORE, ZCOMPUTEDASSETATTRIBUTES.ZLOWLIGHT, ZCOMPUTEDASSETATTRIBUTES.ZNOISESCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCAMERATILTSCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCOMPOSITIONSCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTLIGHTINGSCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPATTERNSCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPERSPECTIVESCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPOSTPROCESSINGSCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTREFLECTIONSSCORE, ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTSYMMETRYSCORE, ZCOMPUTEDASSETATTRIBUTES.ZSHARPLYFOCUSEDSUBJECTSCORE, ZCOMPUTEDASSETATTRIBUTES.ZTASTEFULLYBLURREDSCORE, ZCOMPUTEDASSETATTRIBUTES.ZWELLCHOSENSUBJECTSCORE, ZCOMPUTEDASSETATTRIBUTES.ZWELLFRAMEDSUBJECTSCORE, ZCOMPUTEDASSETATTRIBUTES.ZWELLTIMEDSHOTSCORE from attached.ZGENERICASSET join attached.ZCOMPUTEDASSETATTRIBUTES on attached.ZGENERICASSET.Z_PK = attached.ZCOMPUTEDASSETATTRIBUTES.Z_PK; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767, https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623739934,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,623739934,MDEyOklzc3VlQ29tbWVudDYyMzczOTkzNA==,9599,2020-05-04T22:24:26Z,2020-05-04T22:24:26Z,MEMBER,Twitter thread with some examples of photos that are coming up from queries against these scores: https://twitter.com/simonw/status/1257434670750408705,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767, https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623805823,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623805823,MDEyOklzc3VlQ29tbWVudDYyMzgwNTgyMw==,9599,2020-05-05T02:45:56Z,2020-05-05T02:45:56Z,MEMBER,I filed an issue with `osxphotos` about this here: https://github.com/RhetTbull/osxphotos/issues/121,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234, https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806085,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623806085,MDEyOklzc3VlQ29tbWVudDYyMzgwNjA4NQ==,9599,2020-05-05T02:47:18Z,2020-05-05T02:47:18Z,MEMBER,"In https://github.com/RhetTbull/osxphotos/issues/121#issuecomment-623249263 Rhet Turnbull spotted a table called `ZSCENEIDENTIFIER` which looked like it might have the right data, but the columns in it aren't particularly helpful: ``` Z_PK,Z_ENT,Z_OPT,ZSCENEIDENTIFIER,ZASSETATTRIBUTES,ZCONFIDENCE 8,49,1,731,5,0.11834716796875 9,49,1,684,6,0.0233648251742125 10,49,1,1702,1,0.026153564453125 ``` I love the look of those confidence scores, but what do the numbers mean?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234, https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806533,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623806533,MDEyOklzc3VlQ29tbWVudDYyMzgwNjUzMw==,9599,2020-05-05T02:50:16Z,2020-05-05T02:50:16Z,MEMBER,"I figured there must be a separate database that Photos uses to store the text of the identified labels. I used ""Open Files and Ports"" in Activity Monitor against the Photos app to try and spot candidates... and found `/Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite` - a 53MB SQLite database file. Here's the schema of that file: ``` $ sqlite3 psi.sqlite .schema CREATE TABLE word_embedding(word TEXT, extended_word TEXT, score DOUBLE); CREATE INDEX word_embedding_index ON word_embedding(word); CREATE VIRTUAL TABLE word_embedding_prefix USING fts5(extended_word) /* word_embedding_prefix(extended_word) */; CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_data'(id INTEGER PRIMARY KEY, block BLOB); CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID; CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_content'(id INTEGER PRIMARY KEY, c0); CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB); CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID; CREATE TABLE groups(category INT2, owning_groupid INT, content_string TEXT, normalized_string TEXT, lookup_identifier TEXT, token_ranges_0 INT8, token_ranges_1 INT8, UNIQUE(category, owning_groupid, content_string, lookup_identifier, token_ranges_0, token_ranges_1)); CREATE TABLE assets(uuid_0 INT, uuid_1 INT, creationDate INT, UNIQUE(uuid_0, uuid_1)); CREATE TABLE ga(groupid INT, assetid INT, PRIMARY KEY(groupid, assetid)); CREATE TABLE collections(uuid_0 INT, uuid_1 INT, startDate INT, endDate INT, title TEXT, subtitle TEXT, keyAssetUUID_0 INT, keyAssetUUID_1 INT, typeAndNumberOfAssets INT32, sortDate DOUBLE, UNIQUE(uuid_0, uuid_1)); CREATE TABLE gc(groupid INT, collectionid INT, PRIMARY KEY(groupid, collectionid)); CREATE VIRTUAL TABLE prefix USING fts5(content='groups', normalized_string, category UNINDEXED, tokenize = 'PSITokenizer'); CREATE TABLE IF NOT EXISTS 'prefix_data'(id INTEGER PRIMARY KEY, block BLOB); CREATE TABLE IF NOT EXISTS 'prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID; CREATE TABLE IF NOT EXISTS 'prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB); CREATE TABLE IF NOT EXISTS 'prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID; CREATE TABLE lookup(identifier TEXT PRIMARY KEY, category INT2); CREATE TRIGGER trigger_groups_insert AFTER INSERT ON groups BEGIN INSERT INTO prefix(rowid, normalized_string, category) VALUES (new.rowid, new.normalized_string, new.category); END; CREATE TRIGGER trigger_groups_delete AFTER DELETE ON groups BEGIN INSERT INTO prefix(prefix, rowid, normalized_string, category) VALUES('delete', old.rowid, old.normalized_string, old.category); END; CREATE INDEX group_pk ON groups(category, content_string, normalized_string, lookup_identifier); CREATE INDEX asset_pk ON assets(uuid_0, uuid_1); CREATE INDEX ga_assetid ON ga(assetid, groupid); CREATE INDEX collection_pk ON collections(uuid_0, uuid_1); CREATE INDEX gc_collectionid ON gc(collectionid); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234, https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806687,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623806687,MDEyOklzc3VlQ29tbWVudDYyMzgwNjY4Nw==,9599,2020-05-05T02:51:16Z,2020-05-05T02:51:16Z,MEMBER,"Running datasette against it directly doesn't work: ``` simon@Simons-MacBook-Pro search % datasette psi.sqlite Serve! files=('psi.sqlite',) (immutables=()) on port 8001 Usage: datasette serve [OPTIONS] [FILES]... Error: Connection to psi.sqlite failed check: no such tokenizer: PSITokenizer ``` Instead, I created a new SQLite database with a copy of some of the key tables, like this: ``` sqlite-utils rows psi.sqlite groups | sqlite-utils insert /tmp/search.db groups - sqlite-utils rows psi.sqlite assets | sqlite-utils insert /tmp/search.db assets - sqlite-utils rows psi.sqlite ga | sqlite-utils insert /tmp/search.db ga - sqlite-utils rows psi.sqlite collections | sqlite-utils insert /tmp/search.db collections - sqlite-utils rows psi.sqlite gc | sqlite-utils insert /tmp/search.db gc - sqlite-utils rows psi.sqlite lookup | sqlite-utils insert /tmp/search.db lookup - ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234, https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623807568,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623807568,MDEyOklzc3VlQ29tbWVudDYyMzgwNzU2OA==,9599,2020-05-05T02:56:06Z,2020-05-05T02:56:06Z,MEMBER,"I'm pretty sure this is what I'm after. The `groups` table has what looks like identified labels in the rows with category = 2025: Then there's a `ga` table that maps groups to assets: And an `assets` table which looks like it has one row for every one of my photos: One major challenge: these UUIDs are split into two integer numbers, `uuid_0` and `uuid_1` - but the main photos database uses regular UUIDs like this: ![image](https://user-images.githubusercontent.com/9599/81031481-39164280-8e41-11ea-983b-005ced641a18.png) I need to figure out how to match up these two different UUID representations. I asked on Twitter if anyone has any ideas: https://twitter.com/simonw/status/1257500689019703296","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234, https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623811131,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623811131,MDEyOklzc3VlQ29tbWVudDYyMzgxMTEzMQ==,9599,2020-05-05T03:16:18Z,2020-05-05T03:16:18Z,MEMBER,"Here's how to convert two integers unto a UUID using Java. Not sure if it's the solution I need though (or how to do the same thing in Python): https://repl.it/repls/EuphoricSomberClasslibrary ```java import java.util.UUID; class Main { public static void main(String[] args) { java.util.UUID uuid = new java.util.UUID( 2544182952487526660L, -3640314103732024685L ); System.out.println( uuid ); } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234, https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623846880,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623846880,MDEyOklzc3VlQ29tbWVudDYyMzg0Njg4MA==,9599,2020-05-05T04:06:08Z,2020-05-05T04:06:08Z,MEMBER,"This function seems to convert them into UUIDs that match my photos: ```python def to_uuid(uuid_0, uuid_1): b = uuid_0.to_bytes(8, 'little', signed=True) + uuid_1.to_bytes(8, 'little', signed=True) return str(uuid.UUID(bytes=b)).upper() ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234, https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855841,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623855841,MDEyOklzc3VlQ29tbWVudDYyMzg1NTg0MQ==,9599,2020-05-05T04:54:28Z,2020-05-05T04:54:28Z,MEMBER,"Things were not matching up for me correctly: I think that's because my import script didn't correctly import the existing `rowid` values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234, https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855885,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623855885,MDEyOklzc3VlQ29tbWVudDYyMzg1NTg4NQ==,9599,2020-05-05T04:54:39Z,2020-05-05T04:54:53Z,MEMBER,"Trying this import mechanism instead: `sqlite3 /Users/simon/Pictures/Photos\ Library.photoslibrary/database/search/psi.sqlite .dump | grep -v 'CREATE INDEX' | grep -v 'CREATE TRIGGER' | grep -v 'CREATE VIRTUAL TABLE' | sqlite3 search.db`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234, https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623857417,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623857417,MDEyOklzc3VlQ29tbWVudDYyMzg1NzQxNw==,9599,2020-05-05T05:01:47Z,2020-05-05T05:01:47Z,MEMBER,"Even that didn't work - it didn't copy across the rowid values. I'm pretty sure that's what's wrong here: ``` sqlite3 /Users/simon/Pictures/Photos\ Library.photoslibrary/database/search/psi.sqlite 'select rowid, uuid_0, uuid_1 from assets limit 10' 1619605|-9205353363298198838|4814875488794983828 1641378|-9205348195631362269|390804289838822030 1634974|-9205331524553603243|-3834026796261633148 1619083|-9205326176986145401|7563404215614709654 22131|-9205315724827218763|8370531509591906734 1645633|-9205247376092758131|-1311540150497601346 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234, https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623863902,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623863902,MDEyOklzc3VlQ29tbWVudDYyMzg2MzkwMg==,9599,2020-05-05T05:31:53Z,2020-05-05T05:31:53Z,MEMBER,"Yes! Turning those `rowid` values into `id` with this script did the job: ```python import sqlite3 import sqlite_utils conn = sqlite3.connect( ""/Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite"" ) def all_rows(table): result = conn.execute(""select rowid as id, * from {}"".format(table)) cols = [c[0] for c in result.description] for row in result.fetchall(): yield dict(zip(cols, row)) if __name__ == ""__main__"": db = sqlite_utils.Database(""psi_copy.db"") for table in (""assets"", ""collections"", ""ga"", ""gc"", ""groups""): db[table].upsert_all(all_rows(table), pk=""id"", alter=True) ``` Then I ran this query: ```sql select json_object('img_src', 'https://photos.simonwillison.net/i/' || photos.sha256 || '.' || photos.ext || '?w=400') as photo, group_concat(strip_null_chars(groups.content_string), ' ') as words, assets.uuid_0, assets.uuid_1, to_uuid(assets.uuid_0, assets.uuid_1) as uuid from assets join ga on assets.id = ga.assetid join groups on ga.groupid = groups.id join photos on photos.uuid = to_uuid(assets.uuid_0, assets.uuid_1) where groups.category = 2024 group by assets.id order by random() limit 10 ``` And got these results! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234, https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623865250,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623865250,MDEyOklzc3VlQ29tbWVudDYyMzg2NTI1MA==,9599,2020-05-05T05:38:16Z,2020-05-05T05:38:16Z,MEMBER,It looks like `groups.content_string` often has a null byte in it. I should clean this up as part of the import.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234, https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624278090,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17,624278090,MDEyOklzc3VlQ29tbWVudDYyNDI3ODA5MA==,9599,2020-05-05T20:06:01Z,2020-05-05T20:06:01Z,MEMBER,https://www.python.org/dev/peps/pep-0508/#environment-markers I think I want `sys_platform` of `darwin`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612860531, https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624278714,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17,624278714,MDEyOklzc3VlQ29tbWVudDYyNDI3ODcxNA==,9599,2020-05-05T20:07:19Z,2020-05-05T20:07:19Z,MEMBER,"From https://hynek.me/articles/conditional-python-dependencies/ I think this will look like: ```python setup( # ... install_requires=[ # ... ""osxphotos>=0.28.13 ; sys_platform=='darwin'"", ] ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612860531, https://github.com/dogsheep/dogsheep-photos/issues/18#issuecomment-624364557,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18,624364557,MDEyOklzc3VlQ29tbWVudDYyNDM2NDU1Nw==,9599,2020-05-05T23:49:18Z,2020-05-05T23:49:18Z,MEMBER,Label is `macos-latest`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612860758, https://github.com/dogsheep/dogsheep-photos/issues/19#issuecomment-624406285,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/19,624406285,MDEyOklzc3VlQ29tbWVudDYyNDQwNjI4NQ==,9599,2020-05-06T02:10:03Z,2020-05-06T02:10:03Z,MEMBER,"Most annoying part of this is the difficulty of actually showing a photo. Maybe I need to run a local proxy that I can link to? A custom Datasette plugin perhaps?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613002220, https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408220,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,624408220,MDEyOklzc3VlQ29tbWVudDYyNDQwODIyMA==,9599,2020-05-06T02:18:47Z,2020-05-06T02:18:47Z,MEMBER,"The `apple_photos` table has an indexed `uuid` column and a `path` column which stores the full path to that photo file on disk. I can write a custom Datasette plugin which takes the `uuid` from the URL, looks up the path, then serves up a thumbnail of the jpeg or heic image file. I'll prototype this is a one-off plugin first, then package it on PyPI for other people to install.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393, https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408370,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,624408370,MDEyOklzc3VlQ29tbWVudDYyNDQwODM3MA==,9599,2020-05-06T02:19:27Z,2020-05-06T02:19:27Z,MEMBER,"The plugin can be generalized: it can be configured to know how to take the URL path, look it up in ANY table (via a custom SQL query) to get a path on disk and then serve that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393, https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408738,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,624408738,MDEyOklzc3VlQ29tbWVudDYyNDQwODczOA==,9599,2020-05-06T02:21:05Z,2020-05-06T02:21:32Z,MEMBER,"Here's rendering code from my hacked-together not-yet-released S3 image proxy: ```python from starlette.responses import Response from PIL import Image, ExifTags import pyheif for ORIENTATION_TAG in ExifTags.TAGS.keys(): if ExifTags.TAGS[ORIENTATION_TAG] == ""Orientation"": break ... # Load it into Pillow if ext == ""heic"": heic = pyheif.read_heif(image_response.content) image = Image.frombytes(mode=heic.mode, size=heic.size, data=heic.data) else: image = Image.open(io.BytesIO(image_response.content)) # Does EXIF tell us to rotate it? try: exif = dict(image._getexif().items()) if exif[ORIENTATION_TAG] == 3: image = image.rotate(180, expand=True) elif exif[ORIENTATION_TAG] == 6: image = image.rotate(270, expand=True) elif exif[ORIENTATION_TAG] == 8: image = image.rotate(90, expand=True) except (AttributeError, KeyError, IndexError): pass # Resize based on ?w= and ?h=, if set width, height = image.size w = request.query_params.get(""w"") h = request.query_params.get(""h"") if w is not None or h is not None: if h is None: # Set h based on w w = int(w) h = int((float(height) / width) * w) elif w is None: h = int(h) # Set w based on h w = int((float(width) / height) * h) w = int(w) h = int(h) image.thumbnail((w, h)) # ?bw= converts to black and white if request.query_params.get(""bw""): image = image.convert(""L"") # ?q= sets the quality - defaults to 75 quality = 75 q = request.query_params.get(""q"") if q and q.isdigit() and 1 <= int(q) <= 100: quality = int(q) # Output as JPEG or PNG output_image = io.BytesIO() image_type = ""JPEG"" kwargs = {""quality"": quality} if image.format == ""PNG"": image_type = ""PNG"" kwargs = {} image.save(output_image, image_type, **kwargs) return Response( output_image.getvalue(), media_type=""image/jpeg"", headers={""cache-control"": ""s-maxage={}, public"".format(365 * 24 * 60 * 60)}, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393, https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-625947133,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,625947133,MDEyOklzc3VlQ29tbWVudDYyNTk0NzEzMw==,9599,2020-05-08T18:13:06Z,2020-05-08T18:13:06Z,MEMBER,`datasette-media` will be able to handle this once I implement https://github.com/simonw/datasette-media/issues/3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393, https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388764,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626388764,MDEyOklzc3VlQ29tbWVudDYyNjM4ODc2NA==,9599,2020-05-10T20:58:52Z,2020-05-10T20:58:52Z,MEMBER,"More from the debugger: ``` > /Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py(614)place() -> self._place = PlaceInfo5(self._info[""reverse_geolocation""]) ``` And: ``` > /Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/utils.py(91)osxphoto_to_row() -> place = photo.place ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990, https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388837,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626388837,MDEyOklzc3VlQ29tbWVudDYyNjM4ODgzNw==,9599,2020-05-10T20:59:32Z,2020-05-10T20:59:32Z,MEMBER,So it appears it's possible for `photo.place` to raise that exception. A workaround could be to catch that and treat those photos as not having a place.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990, https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626394989,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626394989,MDEyOklzc3VlQ29tbWVudDYyNjM5NDk4OQ==,9599,2020-05-10T21:50:36Z,2020-05-10T21:50:36Z,MEMBER,https://github.com/Marketcircle/bpylist/pull/2 looks relevant here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990, https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395103,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395103,MDEyOklzc3VlQ29tbWVudDYyNjM5NTEwMw==,9599,2020-05-10T21:51:36Z,2020-05-10T21:51:36Z,MEMBER,"@RhetTbull I tried that workaround and it turns out I'm getting this error on ALL of my photos now! It's weird: a few day ago this wasn't happening. Now it's happening to everything. I'm not sure what I might have changed. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990, https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395209,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395209,MDEyOklzc3VlQ29tbWVudDYyNjM5NTIwOQ==,9599,2020-05-10T21:52:42Z,2020-05-10T21:52:42Z,MEMBER,"Aha! It looks like I accidentally installed the old bplist into the same environment: ``` $ pip freeze | grep bpylist bpylist==0.1.4 bpylist2==3.0.0 ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990, https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395781,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395781,MDEyOklzc3VlQ29tbWVudDYyNjM5NTc4MQ==,9599,2020-05-10T21:57:09Z,2020-05-10T21:57:09Z,MEMBER,"Yes, I just recreated my virtual environment from scratch and the error went away. The problem occurred when I ran `pip install datasette-bplist` in the same virtual environment - https://github.com/simonw/datasette-bplist/blob/master/setup.py depends on `bpylist` which is incompatible with `bpylist2`.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990, https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-626941278,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,626941278,MDEyOklzc3VlQ29tbWVudDYyNjk0MTI3OA==,9599,2020-05-11T20:25:58Z,2020-05-11T20:25:58Z,MEMBER,"Interesting - do you know if there's anything the `exiftool` process handles that `ExifReader` doesn't? I'm actually just going to extract a subset of the EXIF data at first - since the original photo files will always be available I don't feel the need to get everything out for the first step. My plan is to use EXIF to help support photo collections that aren't in Apple Photos - I'm going to build a database table keyed by the `sha256` of each photo that extracts the camera make, lens, a few settings (ISO, aperture etc) and the GPS lat/lon.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118, https://github.com/dogsheep/dogsheep-photos/issues/23#issuecomment-631120771,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/23,631120771,MDEyOklzc3VlQ29tbWVudDYzMTEyMDc3MQ==,9599,2020-05-19T22:32:48Z,2020-05-19T22:32:48Z,MEMBER,Documentation: https://github.com/dogsheep/photos-to-sqlite/blob/e2fab012551eed05278040b5d57e7373a1b9a0bf/README.md#creating-a-subset-database,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621280529, https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631127454,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631127454,MDEyOklzc3VlQ29tbWVudDYzMTEyNzQ1NA==,9599,2020-05-19T22:48:00Z,2020-05-21T15:58:32Z,MEMBER,"I built #23 to help with this. $ dogsheep-photos create-subset photos.db public.db \ ""select sha256 from apple_photos where albums like '%Public%'"" And publish with Vercel: $ datasette publish now public.db --project dogsheep-photos \ --about=dogsheep/dogsheep-photos \ --about_url=""https://github.com/dogsheep/dogsheep-photos"" \ --install=datasette-json-html \ --install=datasette-cluster-map","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242, https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226481,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631226481,MDEyOklzc3VlQ29tbWVudDYzMTIyNjQ4MQ==,9599,2020-05-20T04:18:29Z,2020-05-20T04:18:29Z,MEMBER,I just renamed the repository.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763, https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226572,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631226572,MDEyOklzc3VlQ29tbWVudDYzMTIyNjU3Mg==,9599,2020-05-20T04:18:52Z,2020-05-20T04:18:52Z,MEMBER,Need to reconfigure Circle CI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763, https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226953,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631226953,MDEyOklzc3VlQ29tbWVudDYzMTIyNjk1Mw==,9599,2020-05-20T04:20:34Z,2020-05-20T04:20:34Z,MEMBER,"Huh, it looks like Circle CI picked up the name change automatically. https://app.circleci.com/pipelines/github/dogsheep/dogsheep-photos","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763, https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227020,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631227020,MDEyOklzc3VlQ29tbWVudDYzMTIyNzAyMA==,9599,2020-05-20T04:20:48Z,2020-05-20T04:21:16Z,MEMBER,Next time I push a release it will create `dogsheep-photos` on PyPI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763, https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227105,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631227105,MDEyOklzc3VlQ29tbWVudDYzMTIyNzEwNQ==,9599,2020-05-20T04:21:06Z,2020-05-20T04:21:06Z,MEMBER,Then I just need to push a final photos-to-sqlite release that updates the README to tell people about the name change.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763, https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227245,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631227245,MDEyOklzc3VlQ29tbWVudDYzMTIyNzI0NQ==,9599,2020-05-20T04:21:38Z,2020-05-20T04:21:38Z,MEMBER,I'm going to release 0.4 now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763, https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229409,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631229409,MDEyOklzc3VlQ29tbWVudDYzMTIyOTQwOQ==,9599,2020-05-20T04:30:40Z,2020-05-20T04:30:40Z,MEMBER,https://pypi.org/project/photos-to-sqlite/ now links to dogsheep-photos.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763, https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229485,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631229485,MDEyOklzc3VlQ29tbWVudDYzMTIyOTQ4NQ==,9599,2020-05-20T04:31:02Z,2020-05-20T04:31:02Z,MEMBER,https://pypi.org/project/dogsheep-photos/ is live.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763, https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631251707,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631251707,MDEyOklzc3VlQ29tbWVudDYzMTI1MTcwNw==,9599,2020-05-20T05:49:27Z,2020-05-21T15:58:42Z,MEMBER,Renaming this demo to `dogsheep-photos.dogsheep.net`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242, https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253136,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631253136,MDEyOklzc3VlQ29tbWVudDYzMTI1MzEzNg==,9599,2020-05-20T05:53:58Z,2020-05-20T05:53:58Z,MEMBER,"Updated deploy command: ``` datasette publish now public.db --project dogsheep-photos \ --about=dogsheep/dogsheep-photos \ --about_url=""https://github.com/dogsheep/dogsheep-photos"" \ --install=datasette-json-html \ --install=datasette-cluster-map \ --title ""Dogsheep Photos demo"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242, https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253248,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631253248,MDEyOklzc3VlQ29tbWVudDYzMTI1MzI0OA==,9599,2020-05-20T05:54:18Z,2020-05-20T05:54:18Z,MEMBER,https://dogsheep-photos.dogsheep.net/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242, https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253852,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631253852,MDEyOklzc3VlQ29tbWVudDYzMTI1Mzg1Mg==,9599,2020-05-20T05:56:17Z,2020-05-21T22:26:16Z,MEMBER,"I have a `deploy-demo.sh` script now: ```bash #!/bin/bash if [ -f public.db ]; then rm public.db fi pipenv run dogsheep-photos create-subset photos.db public.db \ ""select sha256 from apple_photos where albums like '%Public%'"" pipenv run sqlite-utils create-view public.db photos_on_a_map \ ""select date, latitude, longitude, apple_photos.sha256, uploads.ext, json_object( 'title', 'Taken on ' || date, 'image', 'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=400', 'link', 'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=1200' ) as popup from apple_photos join uploads on apple_photos.sha256 = uploads.sha256 where latitude is not null order by date desc"" \ --replace pipenv run datasette publish now public.db --project dogsheep-photos \ --about=dogsheep/dogsheep-photos \ --about_url=""https://github.com/dogsheep/dogsheep-photos"" \ --install=datasette-json-html \ --install=datasette-pretty-json \ --install=datasette-cluster-map>=0.10 \ --title ""Dogsheep Photos demo"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,