{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/18#issuecomment-624364557", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18", "id": 624364557, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNDM2NDU1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-05T23:49:18Z", "updated_at": "2020-05-05T23:49:18Z", "author_association": "MEMBER", "body": "Label is `macos-latest`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612860758, "label": "Switch CI solution to GitHub Actions with a macOS runner"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/19#issuecomment-624406285", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/19", "id": 624406285, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNDQwNjI4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-06T02:10:03Z", "updated_at": "2020-05-06T02:10:03Z", "author_association": "MEMBER", "body": "Most annoying part of this is the difficulty of actually showing a photo.\r\n\r\nMaybe I need to run a local proxy that I can link to? A custom Datasette plugin perhaps?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613002220, "label": "apple-photos command should work even if upload has not run"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/2#issuecomment-615931488", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/2", "id": 615931488, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMTQ4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:24:02Z", "updated_at": "2020-04-18T19:24:02Z", "author_association": "MEMBER", "body": "I made a start on this last week with a https://github.com/simonw/heic-to-jpeg proxy.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533352, "label": "Ability to convert HEIC images to JPEG"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408220", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 624408220, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNDQwODIyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-06T02:18:47Z", "updated_at": "2020-05-06T02:18:47Z", "author_association": "MEMBER", "body": "The `apple_photos` table has an indexed `uuid` column and a `path` column which stores the full path to that photo file on disk.\r\n\r\nI can write a custom Datasette plugin which takes the `uuid` from the URL, looks up the path, then serves up a thumbnail of the jpeg or heic image file.\r\n\r\nI'll prototype this is a one-off plugin first, then package it on PyPI for other people to install.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408370", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 624408370, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNDQwODM3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-06T02:19:27Z", "updated_at": "2020-05-06T02:19:27Z", "author_association": "MEMBER", "body": "The plugin can be generalized: it can be configured to know how to take the URL path, look it up in ANY table (via a custom SQL query) to get a path on disk and then serve that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408738", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 624408738, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNDQwODczOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-06T02:21:05Z", "updated_at": "2020-05-06T02:21:32Z", "author_association": "MEMBER", "body": "Here's rendering code from my hacked-together not-yet-released S3 image proxy:\r\n```python\r\nfrom starlette.responses import Response\r\nfrom PIL import Image, ExifTags\r\nimport pyheif\r\n\r\nfor ORIENTATION_TAG in ExifTags.TAGS.keys():\r\n    if ExifTags.TAGS[ORIENTATION_TAG] == \"Orientation\":\r\n        break\r\n    ...\r\n    # Load it into Pillow\r\n    if ext == \"heic\":\r\n        heic = pyheif.read_heif(image_response.content)\r\n        image = Image.frombytes(mode=heic.mode, size=heic.size, data=heic.data)\r\n    else:\r\n        image = Image.open(io.BytesIO(image_response.content))\r\n\r\n    # Does EXIF tell us to rotate it?\r\n    try:\r\n        exif = dict(image._getexif().items())\r\n        if exif[ORIENTATION_TAG] == 3:\r\n            image = image.rotate(180, expand=True)\r\n        elif exif[ORIENTATION_TAG] == 6:\r\n            image = image.rotate(270, expand=True)\r\n        elif exif[ORIENTATION_TAG] == 8:\r\n            image = image.rotate(90, expand=True)\r\n    except (AttributeError, KeyError, IndexError):\r\n        pass\r\n\r\n    # Resize based on ?w= and ?h=, if set\r\n    width, height = image.size\r\n    w = request.query_params.get(\"w\")\r\n    h = request.query_params.get(\"h\")\r\n    if w is not None or h is not None:\r\n        if h is None:\r\n            # Set h based on w\r\n            w = int(w)\r\n            h = int((float(height) / width) * w)\r\n        elif w is None:\r\n            h = int(h)\r\n            # Set w based on h\r\n            w = int((float(width) / height) * h)\r\n        w = int(w)\r\n        h = int(h)\r\n        image.thumbnail((w, h))\r\n\r\n    # ?bw= converts to black and white\r\n    if request.query_params.get(\"bw\"):\r\n        image = image.convert(\"L\")\r\n\r\n    # ?q= sets the quality - defaults to 75\r\n    quality = 75\r\n    q = request.query_params.get(\"q\")\r\n    if q and q.isdigit() and 1 <= int(q) <= 100:\r\n        quality = int(q)\r\n\r\n    # Output as JPEG or PNG\r\n    output_image = io.BytesIO()\r\n    image_type = \"JPEG\"\r\n    kwargs = {\"quality\": quality}\r\n    if image.format == \"PNG\":\r\n        image_type = \"PNG\"\r\n        kwargs = {}\r\n\r\n    image.save(output_image, image_type, **kwargs)\r\n    return Response(\r\n        output_image.getvalue(),\r\n        media_type=\"image/jpeg\",\r\n        headers={\"cache-control\": \"s-maxage={}, public\".format(365 * 24 * 60 * 60)},\r\n    )\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-625947133", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 625947133, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNTk0NzEzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-08T18:13:06Z", "updated_at": "2020-05-08T18:13:06Z", "author_association": "MEMBER", "body": "`datasette-media` will be able to handle this once I implement https://github.com/simonw/datasette-media/issues/3", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633626741", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 633626741, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMzYyNjc0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-25T15:38:55Z", "updated_at": "2020-05-25T15:38:55Z", "author_association": "MEMBER", "body": "Sure, I should absolutely document this!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633629944", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 633629944, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMzYyOTk0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-25T15:47:42Z", "updated_at": "2020-05-25T15:47:42Z", "author_association": "MEMBER", "body": "I'll add a proper section to the README, but for the moment here's how I do this.\r\n\r\nFirst, install `datasette` and the `datasette-media` plugin.\r\n\r\nCreate a `metadata.yaml` file with the following content:\r\n\r\n```yaml\r\nplugins:\r\n  datasette-media:\r\n    photo:\r\n      sql: |-\r\n        select path as filepath, 200 as resize_height from apple_photos where uuid = :key\r\n    photo-big:\r\n      sql: |-\r\n        select path as filepath, 1024 as resize_height from apple_photos where uuid = :key\r\n```\r\nNow run `datasette -m metadata.yaml photos.db` - thumbnails will be served at http://127.0.0.1:8001/-/media/photo/F4469918-13F3-43D8-9EC1-734C0E6B60AD and larger sizes of the image at http://127.0.0.1:8001/-/media/photo-big/A8B02C7D-365E-448B-9510-69F80C26304D\r\n\r\nI also made myself two custom pages, one showing recent images and one showing random images.\r\n\r\nTo do this, install the `datasette-template-sql` plugin and then create a `templates/pages` directory and add these files:\r\n\r\n`recent-photos.html`\r\n```html\r\n<h1>Recent photos</h1>\r\n\r\n<div>\r\n{% for photo in sql(\"select * from apple_photos order by date desc limit 100\") %}\r\n    <img src=\"/-/media/photo/{{ photo['uuid'] }}\">\r\n{% endfor %}\r\n</div>\r\n```\r\n\r\n`random-photos.html`\r\n```html\r\n<h1>Random photos</h1>\r\n\r\n<div>\r\n{% for photo in sql(\"with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 100\") %}\r\n    <img src=\"/-/media/photo/{{ photo['uuid'] }}\">\r\n{% endfor %}\r\n</div>\r\n```\r\n\r\nNow run `datasette -m metadata.yaml photos.db --template-dir=templates/`\r\n\r\nVisit http://127.0.0.1:8001/random-photos to see some random photos or http://127.0.0.1:8002/recent-photos for recent photos.\r\n\r\nThis is using this mechanism: https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633643921", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 633643921, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMzY0MzkyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-25T16:29:44Z", "updated_at": "2020-05-25T16:29:44Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/dogsheep-photos/blob/dc43fa8653cb9c7238a36f52239b91d1ec916d5c/README.md#serving-photos-locally-with-datasette-media", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633644225", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 633644225, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMzY0NDIyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-25T16:30:44Z", "updated_at": "2020-05-25T16:30:44Z", "author_association": "MEMBER", "body": "I'll add docs on using `datasette-json-html` too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633704127", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 633704127, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMzcwNDEyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-25T20:14:22Z", "updated_at": "2020-05-25T20:14:22Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/dogsheep-photos/blob/0.4.1/README.md#serving-photos-locally-with-datasette-media", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388764", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21", "id": 626388764, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNjM4ODc2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-10T20:58:52Z", "updated_at": "2020-05-10T20:58:52Z", "author_association": "MEMBER", "body": "More from the debugger:\r\n```\r\n> /Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py(614)place()\r\n-> self._place = PlaceInfo5(self._info[\"reverse_geolocation\"])\r\n```\r\nAnd:\r\n```\r\n> /Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/utils.py(91)osxphoto_to_row()\r\n-> place = photo.place\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 615474990, "label": "bpylist.archiver.CircularReference: archive has a cycle with uid(13)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388837", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21", "id": 626388837, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNjM4ODgzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-10T20:59:32Z", "updated_at": "2020-05-10T20:59:32Z", "author_association": "MEMBER", "body": "So it appears it's possible for `photo.place` to raise that exception. A workaround could be to catch that and treat those photos as not having a place.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 615474990, "label": "bpylist.archiver.CircularReference: archive has a cycle with uid(13)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626394989", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21", "id": 626394989, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNjM5NDk4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-10T21:50:36Z", "updated_at": "2020-05-10T21:50:36Z", "author_association": "MEMBER", "body": "https://github.com/Marketcircle/bpylist/pull/2 looks relevant here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 615474990, "label": "bpylist.archiver.CircularReference: archive has a cycle with uid(13)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395103", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21", "id": 626395103, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNjM5NTEwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-10T21:51:36Z", "updated_at": "2020-05-10T21:51:36Z", "author_association": "MEMBER", "body": "@RhetTbull I tried that workaround and it turns out I'm getting this error on ALL of my photos now!\r\n\r\nIt's weird: a few day ago this wasn't happening. Now it's happening to everything. I'm not sure what I might have changed. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 615474990, "label": "bpylist.archiver.CircularReference: archive has a cycle with uid(13)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395209", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21", "id": 626395209, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNjM5NTIwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-10T21:52:42Z", "updated_at": "2020-05-10T21:52:42Z", "author_association": "MEMBER", "body": "Aha! It looks like I accidentally installed the old bplist into the same environment:\r\n```\r\n$ pip freeze | grep bpylist\r\nbpylist==0.1.4\r\nbpylist2==3.0.0\r\n```", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 615474990, "label": "bpylist.archiver.CircularReference: archive has a cycle with uid(13)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395781", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21", "id": 626395781, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNjM5NTc4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-10T21:57:09Z", "updated_at": "2020-05-10T21:57:09Z", "author_association": "MEMBER", "body": "Yes, I just recreated my virtual environment from scratch and the error went away.\r\n\r\nThe problem occurred when I ran `pip install datasette-bplist` in the same virtual environment - https://github.com/simonw/datasette-bplist/blob/master/setup.py depends on `bpylist` which is incompatible with `bpylist2`.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 615474990, "label": "bpylist.archiver.CircularReference: archive has a cycle with uid(13)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-626941278", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22", "id": 626941278, "node_id": "MDEyOklzc3VlQ29tbWVudDYyNjk0MTI3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-11T20:25:58Z", "updated_at": "2020-05-11T20:25:58Z", "author_association": "MEMBER", "body": "Interesting - do you know if there's anything the `exiftool` process handles that `ExifReader` doesn't?\r\n\r\nI'm actually just going to extract a subset of the EXIF data at first - since the original photo files will always be available I don't feel the need to get everything out for the first step.\r\n\r\nMy plan is to use EXIF to help support photo collections that aren't in Apple Photos - I'm going to build a database table keyed by the `sha256` of each photo that extracts the camera make, lens, a few settings (ISO, aperture etc) and the GPS lat/lon.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 615626118, "label": "Try out ExifReader"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/23#issuecomment-631120771", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/23", "id": 631120771, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTEyMDc3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-19T22:32:48Z", "updated_at": "2020-05-19T22:32:48Z", "author_association": "MEMBER", "body": "Documentation: https://github.com/dogsheep/photos-to-sqlite/blob/e2fab012551eed05278040b5d57e7373a1b9a0bf/README.md#creating-a-subset-database", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621280529, "label": "create-subset command for creating a publishable subset of a photos database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/24#issuecomment-631255206", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/24", "id": 631255206, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTI1NTIwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T06:00:25Z", "updated_at": "2020-05-20T06:00:25Z", "author_association": "MEMBER", "body": "This needs documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621323348, "label": "Configurable URL for images"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631127454", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25", "id": 631127454, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTEyNzQ1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-19T22:48:00Z", "updated_at": "2020-05-21T15:58:32Z", "author_association": "MEMBER", "body": "I built #23 to help with this.\r\n\r\n    $ dogsheep-photos create-subset photos.db public.db \\\r\n        \"select sha256 from apple_photos where albums like '%Public%'\"\r\n\r\nAnd publish with Vercel:\r\n\r\n    $ datasette publish now public.db --project dogsheep-photos \\\r\n        --about=dogsheep/dogsheep-photos \\\r\n        --about_url=\"https://github.com/dogsheep/dogsheep-photos\" \\\r\n        --install=datasette-json-html \\\r\n        --install=datasette-cluster-map", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621332242, "label": "Create a public demo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631251707", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25", "id": 631251707, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTI1MTcwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T05:49:27Z", "updated_at": "2020-05-21T15:58:42Z", "author_association": "MEMBER", "body": "Renaming this demo to `dogsheep-photos.dogsheep.net`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621332242, "label": "Create a public demo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253136", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25", "id": 631253136, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTI1MzEzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T05:53:58Z", "updated_at": "2020-05-20T05:53:58Z", "author_association": "MEMBER", "body": "Updated deploy command:\r\n```\r\ndatasette publish now public.db --project dogsheep-photos \\\r\n    --about=dogsheep/dogsheep-photos \\\r\n    --about_url=\"https://github.com/dogsheep/dogsheep-photos\" \\\r\n    --install=datasette-json-html \\\r\n    --install=datasette-cluster-map \\\r\n    --title \"Dogsheep Photos demo\"\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621332242, "label": "Create a public demo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253248", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25", "id": 631253248, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTI1MzI0OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T05:54:18Z", "updated_at": "2020-05-20T05:54:18Z", "author_association": "MEMBER", "body": "https://dogsheep-photos.dogsheep.net/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621332242, "label": "Create a public demo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253852", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25", "id": 631253852, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTI1Mzg1Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T05:56:17Z", "updated_at": "2020-05-21T22:26:16Z", "author_association": "MEMBER", "body": "I have a `deploy-demo.sh` script now:\r\n```bash\r\n#!/bin/bash\r\nif [ -f public.db ]; then\r\n  rm public.db\r\nfi\r\npipenv run dogsheep-photos create-subset photos.db public.db \\\r\n    \"select sha256 from apple_photos where albums like '%Public%'\"\r\npipenv run sqlite-utils create-view public.db photos_on_a_map \\\r\n    \"select\r\n      date,\r\n      latitude,\r\n      longitude,\r\n      apple_photos.sha256,\r\n      uploads.ext,\r\n      json_object(\r\n        'title',\r\n        'Taken on ' || date,\r\n        'image',\r\n        'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=400',\r\n        'link',\r\n        'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=1200'\r\n      ) as popup\r\n    from\r\n      apple_photos\r\n      join uploads on apple_photos.sha256 = uploads.sha256\r\n    where\r\n      latitude is not null\r\n    order by\r\n      date desc\" \\\r\n    --replace\r\npipenv run datasette publish now public.db --project dogsheep-photos \\\r\n    --about=dogsheep/dogsheep-photos \\\r\n    --about_url=\"https://github.com/dogsheep/dogsheep-photos\" \\\r\n    --install=datasette-json-html \\\r\n    --install=datasette-pretty-json \\\r\n    --install=datasette-cluster-map>=0.10 \\\r\n    --title \"Dogsheep Photos demo\"\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621332242, "label": "Create a public demo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226481", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26", "id": 631226481, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTIyNjQ4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T04:18:29Z", "updated_at": "2020-05-20T04:18:29Z", "author_association": "MEMBER", "body": "I just renamed the repository.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621444763, "label": "Rename project to dogsheep-photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226572", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26", "id": 631226572, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTIyNjU3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T04:18:52Z", "updated_at": "2020-05-20T04:18:52Z", "author_association": "MEMBER", "body": "Need to reconfigure Circle CI.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621444763, "label": "Rename project to dogsheep-photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226953", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26", "id": 631226953, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTIyNjk1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T04:20:34Z", "updated_at": "2020-05-20T04:20:34Z", "author_association": "MEMBER", "body": "Huh, it looks like Circle CI picked up the name change automatically. https://app.circleci.com/pipelines/github/dogsheep/dogsheep-photos", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621444763, "label": "Rename project to dogsheep-photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227020", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26", "id": 631227020, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTIyNzAyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T04:20:48Z", "updated_at": "2020-05-20T04:21:16Z", "author_association": "MEMBER", "body": "Next time I push a release it will create `dogsheep-photos` on PyPI.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621444763, "label": "Rename project to dogsheep-photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227105", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26", "id": 631227105, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTIyNzEwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T04:21:06Z", "updated_at": "2020-05-20T04:21:06Z", "author_association": "MEMBER", "body": "Then I just need to push a final photos-to-sqlite release that updates the README to tell people about the name change.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621444763, "label": "Rename project to dogsheep-photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227245", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26", "id": 631227245, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTIyNzI0NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T04:21:38Z", "updated_at": "2020-05-20T04:21:38Z", "author_association": "MEMBER", "body": "I'm going to release 0.4 now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621444763, "label": "Rename project to dogsheep-photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229409", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26", "id": 631229409, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTIyOTQwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T04:30:40Z", "updated_at": "2020-05-20T04:30:40Z", "author_association": "MEMBER", "body": "https://pypi.org/project/photos-to-sqlite/ now links to dogsheep-photos.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621444763, "label": "Rename project to dogsheep-photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229485", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26", "id": 631229485, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMTIyOTQ4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-20T04:31:02Z", "updated_at": "2020-05-20T04:31:02Z", "author_association": "MEMBER", "body": "https://pypi.org/project/dogsheep-photos/ is live.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621444763, "label": "Rename project to dogsheep-photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/pull/29#issuecomment-739058820", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/29", "id": 739058820, "node_id": "MDEyOklzc3VlQ29tbWVudDczOTA1ODgyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-12-04T22:32:35Z", "updated_at": "2020-12-04T22:32:35Z", "author_association": "MEMBER", "body": "Thanks for this!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 638375985, "label": "Fixed bug in SQL query for photo scores"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932007", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615932007, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMjAwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:27:55Z", "updated_at": "2020-04-18T19:27:55Z", "author_association": "MEMBER", "body": "Research thread: https://twitter.com/simonw/status/1249049694984011776\r\n> I want to build some software that lets people store their own data in their own S3 bucket, but if possible I'd like not to have to teach people the incantations needed to get their bucket setup and minimum-permission credentials figures out\r\n\r\nhttps://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/ looks useful", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932204", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615932204, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMjIwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:29:22Z", "updated_at": "2020-04-18T19:34:44Z", "author_association": "MEMBER", "body": "I'm going to call my bucket `dogsheep-photos-simon`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615933273", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615933273, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMzI3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:37:33Z", "updated_at": "2020-04-18T19:37:33Z", "author_association": "MEMBER", "body": "https://console.aws.amazon.com/s3/bucket/create?region=us-west-1\r\n\r\n![S3_Management_Console](https://user-images.githubusercontent.com/9599/79669552-33e2a380-8171-11ea-9ab5-5785d34f652a.png)\r\n\r\nI created it with no public read-write access. I plan to use signed URLs via a transforming proxy to access images for display on the web.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615935577", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615935577, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzNTU3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:54:59Z", "updated_at": "2020-04-18T19:55:30Z", "author_association": "MEMBER", "body": "Creating IAM groups called `dogsheep-photos-simon-read-write` and `dogsheep-photos-simon-read`: https://console.aws.amazon.com/iam/home#/groups - I created them with no attached policies.\r\n\r\nNow I can attach an \"inline policy\" to each one. For the read-write group I go here:\r\n\r\nhttps://console.aws.amazon.com/iam/home#/groups/dogsheep-photos-simon-read-write\r\n\r\n![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669703-2d086080-8172-11ea-9597-83e0b155193e.png)\r\n\r\nExample policies are here: https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html\r\n\r\nFor the read-write one I went with:\r\n```json\r\n{\r\n    \"Version\": \"2012-10-17\",\r\n    \"Statement\": [\r\n        {\r\n            \"Effect\": \"Allow\",\r\n            \"Action\": \"s3:*\",\r\n            \"Resource\": [\r\n                \"arn:aws:s3:::dogsheep-photos-simon/*\"\r\n            ]\r\n        }\r\n    ]\r\n}\r\n```\r\nFor the read-only policy I'm going to guess that this is appropriate:\r\n\r\n```json\r\n{\r\n    \"Version\": \"2012-10-17\",\r\n    \"Statement\": [\r\n        {\r\n            \"Effect\": \"Allow\",\r\n            \"Action\": [\r\n                \"s3:GetObject*\",\r\n                \"s3:ListBucket\"\r\n            ],\r\n            \"Resource\": [\r\n                \"arn:aws:s3:::dogsheep-photos-simon/*\"\r\n            ]\r\n        }\r\n    ]\r\n}\r\n```\r\nI tried the policy simulator to test this out: https://policysim.aws.amazon.com/home/index.jsp?#groups/dogsheep-photos-simon-read - this worked:\r\n\r\n![IAM_Policy_Simulator](https://user-images.githubusercontent.com/9599/79669893-cd12b980-8173-11ea-8dfb-5660ce3652da.png)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615936880", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615936880, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzNjg4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:04:31Z", "updated_at": "2020-04-18T20:04:31Z", "author_association": "MEMBER", "body": "Next step: create two IAM users, one for each of those groups.\r\n\r\nhttps://console.aws.amazon.com/iam/home#/users$new?step=details\r\n\r\n![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669931-1bc05380-8174-11ea-9657-0e0c6a692d42.png)\r\n\r\n![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669941-27137f00-8174-11ea-8ce7-249f0d4f96f6.png)\r\n\r\nI copied the keys into a secure note in 1password.\r\n\r\nCouldn't get into Transmit with them though! https://library.panic.com/transmit/transmit5/iam-roles/ may help.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615941746", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615941746, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0MTc0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:29:36Z", "updated_at": "2020-04-18T20:29:36Z", "author_association": "MEMBER", "body": "I'm going to create another user just for Transmit, with full S3 access.\r\n\r\nname: `dogsheep-photos-simon-s3-all-access`\r\n\r\nRather than creating a group for that user, I'm trying the \"Attach existing policies directly\" option:\r\n\r\n![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79670182-03513880-8176-11ea-811a-c80aefb4538a.png)\r\n\r\nThat user DID work with Transmit. I uploaded a test HEIC image. I used Transmit to copy a signed URL for it.\r\n\r\n```\r\n~ $ curl -i 'https://dogsheep-photos-simon.s3.us-west-1.amazonaws.com/IMG_7195.HEIC?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAWXFXAI...' | head -n 100\r\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\n                                 Dload  Upload   Total   Spent    Left  Speed\r\n  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/1.1 200 OK\r\nx-amz-id-2: gBOCYqZfbNAnv0R/uJ++qm2NbW5SgD4TapgF9RQjzzeDIThcCz/BkKU+YoxlG4NJHlcmMgAHyh4=\r\nx-amz-request-id: C2FE7FCC3BD53A84\r\nDate: Sat, 18 Apr 2020 20:28:54 GMT\r\nLast-Modified: Sat, 18 Apr 2020 20:13:49 GMT\r\nETag: \"fe3e081239a123ef745517878c53b854\"\r\nAccept-Ranges: bytes\r\nContent-Type: image/heic\r\nContent-Length: 1913097\r\nServer: AmazonS3\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615942116", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615942116, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0MjExNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:30:56Z", "updated_at": "2020-04-18T20:30:56Z", "author_association": "MEMBER", "body": "Next step: attempt a programmatic upload using the `dogsheep-photos-simon-read-write` credentials from a Jupyter notebook.\r\n\r\nAlso attempt a programmatic bucket listing and read using `dogsheep-photos-simon-read` credentials.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615944806", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615944806, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NDgwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:41:39Z", "updated_at": "2020-04-18T20:41:39Z", "author_association": "MEMBER", "body": "This worked!\r\n\r\n![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670712-d868e380-8179-11ea-82a5-5dfd17356113.png)\r\n\r\nAnd this worked:\r\n\r\n![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670777-50370e00-817a-11ea-83cd-18ebf5702878.png)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615945056", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615945056, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NTA1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:42:41Z", "updated_at": "2020-04-18T20:42:41Z", "author_association": "MEMBER", "body": "But... `list_objects` failed for both of my keys (read and write):\r\n\r\n![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670798-75c41780-817a-11ea-9907-2cbc4a2e497c.png)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615946537", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615946537, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NjUzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:48:13Z", "updated_at": "2020-04-18T20:48:13Z", "author_association": "MEMBER", "body": "How about generating a signed URL?\r\n```python\r\nread_client.generate_presigned_url(\r\n    \"get_object\",\r\n    Params={\r\n        \"Bucket\": \"dogsheep-photos-simon\",\r\n        \"Key\": \"this_is_fine.jpg\",\r\n    },\r\n    ExpiresIn=600\r\n)\r\n```\r\nGave me https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398\r\n\r\nWhich does this:\r\n\r\n```\r\n~ $ curl -i 'https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'\r\nHTTP/1.1 307 Temporary Redirect\r\nx-amz-bucket-region: us-west-1\r\nx-amz-request-id: E78CD859AEE21D33\r\nx-amz-id-2: 648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=\r\nLocation: https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398\r\nContent-Type: application/xml\r\nTransfer-Encoding: chunked\r\nDate: Sat, 18 Apr 2020 20:47:21 GMT\r\nServer: AmazonS3\r\n\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<Error><Code>TemporaryRedirect</Code><Message>Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.</Message><Endpoint>dogsheep-photos-simon.s3-us-west-1.amazonaws.com</Endpoint><Bucket>dogsheep-photos-simon</Bucket><RequestId>E78CD859AEE21D33</RequestId><HostId>648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=</HostId></Error>~ $ \r\n```\r\nSo it redirects to another URL... which returns this:\r\n```\r\n~ $ curl -i 'https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'\r\nHTTP/1.1 200 OK\r\nx-amz-id-2: XafOl6mswj3yz0GJC9+Ptot1ll5sROVwqsMc10CUUfgpaUANTdIx2GhnONb5d1GVFJ6wlS2j3UY=\r\nx-amz-request-id: 258387C180411AFE\r\nDate: Sat, 18 Apr 2020 20:47:52 GMT\r\nLast-Modified: Sat, 18 Apr 2020 20:37:35 GMT\r\nETag: \"ee04081c3182a44a1c6944e94012e977\"\r\nAccept-Ranges: bytes\r\nContent-Type: binary/octet-stream\r\nContent-Length: 53072\r\nServer: AmazonS3\r\n\r\n????JFIF??C\r\n```\r\nSo that worked! It did come back with `Content-Type: binary/octet-stream` though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947229", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615947229, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NzIyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:51:26Z", "updated_at": "2020-04-18T20:51:26Z", "author_association": "MEMBER", "body": "Running the upload again like this resulted in the correct content-type:\r\n```python\r\nclient.upload_file(\r\n    \"/Users/simonw/Desktop/this_is_fine.jpg\",\r\n    \"dogsheep-photos-simon\",\r\n    \"this_is_fine.jpg\",\r\n    ExtraArgs={\r\n        \"ContentType\": \"image/jpeg\"\r\n    }\r\n)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947370", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615947370, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NzM3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:52:13Z", "updated_at": "2020-04-18T20:52:13Z", "author_association": "MEMBER", "body": "This is great! I now have a key that can upload photos, and a separate key that can download photos OR generate signed URLs to access those photos.\r\n\r\nNext step: a script that starts uploading my photos.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615948102", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615948102, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0ODEwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:56:59Z", "updated_at": "2020-04-18T20:56:59Z", "author_association": "MEMBER", "body": "I'm going to start with this:\r\n\r\n`photos-to-sqlite upload photos.db ~/path/to/directory`\r\n\r\nThis will scan the provided directory (and all sub-directories) for image files. It will then:\r\n\r\n* Calculate a sha256 of the contents of that file\r\n* Upload the file to a key that's `sha256.jpg` or `.heic`\r\n* Upload a `sha256.json` file with the original path to the image\r\n* Add that image to a `uploads` table in `photos.db`\r\n\r\nStretch goal: grab the EXIF data and include that in the `.json` upload AND the `uploads` database table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615957385", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615957385, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk1NzM4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T21:56:16Z", "updated_at": "2020-04-18T21:58:11Z", "author_association": "MEMBER", "body": "Got this working! I'll do EXIF in a separate ticket #3.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/5#issuecomment-615949574", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/5", "id": 615949574, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0OTU3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T21:06:07Z", "updated_at": "2020-04-18T21:06:07Z", "author_association": "MEMBER", "body": "```\r\n$ photos-to-sqlite s3-auth\r\nCreate S3 credentials and paste them here:\r\n\r\nAccess key ID: xxx\r\nSecret access key: yyy\r\n$ cat auth.json \r\n{\r\n    \"access_key_id\": \"xxx\",\r\n    \"secret_access_key\": \"yyy\"\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602551638, "label": "photos-to-sqlite s3-auth command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615979923", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6", "id": 615979923, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk3OTkyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T23:36:02Z", "updated_at": "2020-04-18T23:36:02Z", "author_association": "MEMBER", "body": "I'll use a Click progress bar. To do this I need to first calculate the sum number of bytes in the photos that are going to be uploaded, then run the upload.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602575575, "label": "Add progress bar to upload command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615983393", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6", "id": 615983393, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk4MzM5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T23:53:10Z", "updated_at": "2020-04-18T23:53:10Z", "author_association": "MEMBER", "body": "```\r\n$ photos-to-sqlite upload photos3.db ~/Pictures/Photos\\ Library.photoslibrary/Masters/2020\r\nUploading 2.09 GB  [##----------------------------------]    6%  00:36:37\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602575575, "label": "Add progress bar to upload command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/7#issuecomment-615993178", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7", "id": 615993178, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk5MzE3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-19T00:37:08Z", "updated_at": "2020-04-19T00:37:08Z", "author_association": "MEMBER", "body": "https://pypi.org/project/ImageHash/ Is one option.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602585497, "label": "Integrate image content hashing"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100434", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8", "id": 618100434, "node_id": "MDEyOklzc3VlQ29tbWVudDYxODEwMDQzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-23T00:02:53Z", "updated_at": "2020-04-23T00:02:53Z", "author_association": "MEMBER", "body": "I don't think it matters one way or the other - I'm storing the sha256 in the filename, so the fact that I could read the MD5 back from the list bucket operation doesn't give me any benefits.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 605147638, "label": "Should I have used MD5 instead of SHA256?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100658", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8", "id": 618100658, "node_id": "MDEyOklzc3VlQ29tbWVudDYxODEwMDY1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-23T00:03:35Z", "updated_at": "2020-04-23T00:03:35Z", "author_association": "MEMBER", "body": "Also MD5 isn't guaranteed for the ETag:\r\n> If an object is created by either the Multipart Upload or Part Copy operation, the ETag is not an MD5 digest, regardless of the method of encryption.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 605147638, "label": "Should I have used MD5 instead of SHA256?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618724149", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9", "id": 618724149, "node_id": "MDEyOklzc3VlQ29tbWVudDYxODcyNDE0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-23T23:35:29Z", "updated_at": "2020-04-23T23:35:29Z", "author_association": "MEMBER", "body": "```\r\n% photos-to-sqlite upload photos.db ~/Pictures/Photos\\ Library.photoslibrary/originals\r\nFetching existing keys from S3...\r\nGot 22,446 existing keys\r\nCalculating hashes  [####--------------------------------]   13%  00:04:14\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 605938063, "label": "upload command should be resumable, should only upload photos not already uploaded"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618725155", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9", "id": 618725155, "node_id": "MDEyOklzc3VlQ29tbWVudDYxODcyNTE1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-23T23:39:14Z", "updated_at": "2020-04-23T23:39:14Z", "author_association": "MEMBER", "body": "A few minutes later...\r\n```\r\nFetching existing keys from S3...\r\nGot 22,446 existing keys\r\nCalculating hashes  [####################################]  100%          \r\n22,441 hashed files, 610 are not yet in S3\r\nUploading 0.99 GB\r\nUploading 610 photos  [------------------------------------]  1/610  03:10:35\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 605938063, "label": "upload command should be resumable, should only upload photos not already uploaded"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/1#issuecomment-706775706", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/1", "id": 706775706, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc3NTcwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T22:14:00Z", "updated_at": "2020-10-11T22:14:00Z", "author_association": "MEMBER", "body": "A live demo would be good too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718934942, "label": "Documentation on how to use this with Datasette"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/pull/10#issuecomment-777839351", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10", "id": 777839351, "node_id": "MDEyOklzc3VlQ29tbWVudDc3NzgzOTM1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-11T22:37:55Z", "updated_at": "2021-02-11T22:37:55Z", "author_association": "MEMBER", "body": "I've merged these changes by hand now, thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770712149, "label": "BugFix for encoding and not update info."}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777798330", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11", "id": 777798330, "node_id": "MDEyOklzc3VlQ29tbWVudDc3Nzc5ODMzMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-11T21:18:58Z", "updated_at": "2021-02-11T21:18:58Z", "author_association": "MEMBER", "body": "Thanks for the fix!", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 792851444, "label": "XML parse error"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905203570", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 905203570, "node_id": "IC_kwDOEhK-wc419E9y", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-25T05:51:22Z", "updated_at": "2021-08-25T05:53:27Z", "author_association": "MEMBER", "body": "The debugger showed me that it broke on a string that looked like this:\r\n```xml\r\n<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n<!DOCTYPE en-note SYSTEM \"http://xml.evernote.com/pub/enml2.dtd\">\r\n<en-note>\r\n  <h1 title=\"Q3 2018 Reflection & Development\">\r\n    <span title=Q3 2018 Reflection & Development\">\r\n      Q3 2018 Reflection & Development\r\n    </span>\r\n  </h1>\r\n  ...\r\n```\r\nYeah that is not valid XML!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978743426, "label": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905206234", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 905206234, "node_id": "IC_kwDOEhK-wc419Fna", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-25T05:58:42Z", "updated_at": "2021-08-25T05:58:42Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/evernote-to-sqlite/blob/36a466f142e5bad52719851c2fbda0c05cd35b99/evernote_to_sqlite/utils.py#L34-L42\r\n\r\nNot sure why I was round-tripping the `content_xml` like that - I will try not doing that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978743426, "label": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906635938", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 906635938, "node_id": "IC_kwDOEhK-wc42Ciqi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-26T18:18:27Z", "updated_at": "2021-08-26T18:18:27Z", "author_association": "MEMBER", "body": "It looks like I was using the round-trip to dump the `<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>` and `<!DOCTYPE` prefixes.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978743426, "label": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906646452", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 906646452, "node_id": "IC_kwDOEhK-wc42ClO0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-26T18:34:34Z", "updated_at": "2021-08-26T18:35:20Z", "author_association": "MEMBER", "body": "I tried this ampersand fix: https://regex101.com/r/ojU2H9/1\r\n```python\r\n\r\n# https://regex101.com/r/ojU2H9/1\r\n_invalid_ampersand_re = re.compile(r'&(?![a-z0-9]+;)')\r\n\r\ndef fix_bad_xml(xml):\r\n    # More fixes for things like '&' not as part of an entity\r\n    return _invalid_ampersand_re.sub('&amp;', xml)\r\n```\r\n\r\nEven with that I'm still getting total garbage in the `<en-note>` content - it's just HTML, not even trying to be XML.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978743426, "label": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706784028", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4", "id": 706784028, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc4NDAyOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T23:20:32Z", "updated_at": "2020-10-11T23:20:32Z", "author_association": "MEMBER", "body": "I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938508, "label": "Configure FTS + add an index on the date columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706786548", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4", "id": 706786548, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc4NjU0OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T23:39:46Z", "updated_at": "2020-10-11T23:39:46Z", "author_association": "MEMBER", "body": "Should have used porter stemming for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938508, "label": "Configure FTS + add an index on the date columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776180", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5", "id": 706776180, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc3NjE4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T22:17:55Z", "updated_at": "2020-10-11T22:17:55Z", "author_association": "MEMBER", "body": "We could even do server-side thumbnailing for some of these images, but I'm inclined to serve up the full size ones and set a width on the image element based on the `width` attribute on `<en-media>`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938889, "label": "Figure out how to display images from <en-media> tags inline in Datasette"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776242", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5", "id": 706776242, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc3NjI0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T22:18:30Z", "updated_at": "2020-10-11T22:19:48Z", "author_association": "MEMBER", "body": "Alternatively, rather than relying on `datasette-media` this could base64-embed the images. `evernote-to-sqlite` could register itself as a Datasette plugin that knows how to do this.\r\n\r\nMaybe rename the column to `evernote_content` and register a render cell hook that knows how to rewrite those note bodies so that they are visible?\r\n\r\nMight need to feed them through Bleach too, just in case any nasty code can get into them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938889, "label": "Figure out how to display images from <en-media> tags inline in Datasette"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776447", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5", "id": 706776447, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc3NjQ0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T22:20:32Z", "updated_at": "2020-10-11T22:20:32Z", "author_association": "MEMBER", "body": "Or... I could do this client-side. JavaScript that looks for `<en-media>` tags and fetches the data using `fetch()` wouldn't be too hard to write.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938889, "label": "Figure out how to display images from <en-media> tags inline in Datasette"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776680", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5", "id": 706776680, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc3NjY4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T22:22:16Z", "updated_at": "2020-10-11T22:22:16Z", "author_association": "MEMBER", "body": "Maybe the best way do this is with a custom route, `/-/evernote/note-id` - that way I can clean the HTML and resolve the other things in the `<en-note>` structure without using `render_cell()` and the like. My concern about using `render_cell()` is that it could lead to weird security problems when combined with `?sql=` queries.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938889, "label": "Figure out how to display images from <en-media> tags inline in Datasette"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776808", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5", "id": 706776808, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc3NjgwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T22:23:14Z", "updated_at": "2020-10-11T22:23:14Z", "author_association": "MEMBER", "body": "... but it's still important to be able to get to the rendered note directly from the browse notes `/evernote/notes` page. Maybe use a simple `render_cell()` hook that just knows how to generate the link to the rendered note page?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938889, "label": "Figure out how to display images from <en-media> tags inline in Datasette"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706834800", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5", "id": 706834800, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjgzNDgwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-12T03:24:57Z", "updated_at": "2020-10-16T20:16:28Z", "author_association": "MEMBER", "body": "Here's my first attempt at a plugin for this:\r\n```python\r\nfrom datasette import hookimpl\r\nimport jinja2\r\n\r\nSTART = \"<en-note\"\r\nEND = \"</en-note>\"\r\nTEMPLATE = \"\"\"\r\n<div style=\"max-width: 500px; white-space: normal; overflow-wrap: break-word;\">{}</div>\r\n\"\"\".strip()\r\n\r\nEN_MEDIA_SCRIPT = \"\"\"\r\nArray.from(document.querySelectorAll('en-media')).forEach(el => {\r\n    let hash = el.getAttribute('hash');\r\n    let type = el.getAttribute('type');\r\n    let path = `/evernote/resources_data/${hash}.json?_shape=array`;\r\n    fetch(path).then(r => r.json()).then(rows => {\r\n        let b64 = rows[0].data.encoded;\r\n        let data = `data:${type};base64,${b64}`;\r\n        el.innerHTML = `<img style=\"max-width: 300px\" src=\"${data}\">`;\r\n    });\r\n});\r\n\"\"\"\r\n\r\n\r\n@hookimpl\r\ndef render_cell(value, table):\r\n    if not table:\r\n        # Don't render content from arbitrary SQL queries, could be XSS hole\r\n        return\r\n    if not value or not isinstance(value, str):\r\n        return\r\n    value = value.strip()\r\n    if value.startswith(START) and value.endswith(END):\r\n        trimmed = value[len(START) : -len(END)]\r\n        trimmed = trimmed.split(\">\", 1)[1]\r\n        # Replace those horrible double newlines\r\n        trimmed = trimmed.replace(\"<div><br /></div>\", \"<br>\")\r\n        return jinja2.Markup(TEMPLATE.format(trimmed))\r\n\r\n\r\n@hookimpl\r\ndef extra_body_script():\r\n    return EN_MEDIA_SCRIPT\r\n```\r\nIt works!\r\n\r\nIt does however demonstrate that Evernote's \"clip this webpage\" feature means there is a LOT of weird HTML that can get into a note. It looks like they've filtered out the scripts but I wouldn't bet on it - they certainly don't filter out many of the inline styles. So running Bleach is almost certainly a good idea.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718938889, "label": "Figure out how to display images from <en-media> tags inline in Datasette"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785086", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6", "id": 706785086, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc4NTA4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T23:28:50Z", "updated_at": "2020-10-11T23:28:50Z", "author_association": "MEMBER", "body": "The XML for the OCR stuff is a bit weird. Currently I'm doing this to it:\r\n\r\nhttps://github.com/dogsheep/evernote-to-sqlite/blob/c33d7b043a45eb3e88676e5fa3ce31755199d9f8/evernote_to_sqlite/utils.py#L70-L78\r\n\r\nThis can produce some odd results, for example:\r\n\r\n> Sure 'Sure, 'Sure. Sure, Sure. sure sure. sure ? If you If Yau [you live jive In m 1n an area devoid of natural wonders, wanders, wonders ? wonders wonders. your mind will be blown, blown' blown. blown ? -e i ? ,1 IL it ? at ? KY ? fl ft bat at\r\n\r\nWhich came from this image:\r\n\r\n![image](https://user-images.githubusercontent.com/9599/95692952-5dd7c880-0bde-11eb-939a-d10b800a4105.png)\r\n\r\nThe XML for that is:\r\n\r\n```xml\r\n<recoIndex docType=\"unknown\" objType=\"image\" objID=\"05ffb72b307bf495f064243c7099d94f\" engineVersion=\"6.5.17.7\" recoType=\"service\" lang=\"en\" objWidth=\"1000\" objHeight=\"1504\">\r\n  <item x=\"68\" y=\"75\" w=\"104\" h=\"37\">\r\n    <t w=\"60\">Sure</t>\r\n    <t w=\"52\">'Sure,</t>\r\n    <t w=\"47\">'Sure.</t>\r\n    <t w=\"33\">Sure,</t>\r\n    <t w=\"26\">Sure.</t>\r\n  </item>\r\n  <item x=\"182\" y=\"83\" w=\"92\" h=\"26\">\r\n    <t w=\"62\">sure</t>\r\n    <t w=\"58\">sure.</t>\r\n    <t w=\"46\">sure ?</t>\r\n  </item>\r\n  <item x=\"69\" y=\"132\" w=\"107\" h=\"45\">\r\n    <t w=\"81\">If you</t>\r\n    <t w=\"64\">If Yau</t>\r\n    <t w=\"31\">[you</t>\r\n  </item>\r\n  <item x=\"186\" y=\"132\" w=\"67\" h=\"35\">\r\n    <t w=\"85\">live</t>\r\n    <t w=\"51\">jive</t>\r\n  </item>\r\n  <item x=\"263\" y=\"140\" w=\"36\" h=\"27\">\r\n    <t w=\"82\">In</t>\r\n    <t w=\"56\">m</t>\r\n    <t w=\"53\">1n</t>\r\n  </item>\r\n  <item x=\"309\" y=\"140\" w=\"53\" h=\"27\">\r\n    <t w=\"82\">an</t>\r\n  </item>\r\n  <item x=\"372\" y=\"141\" w=\"90\" h=\"26\">\r\n    <t w=\"94\">area</t>\r\n  </item>\r\n  <item x=\"472\" y=\"132\" w=\"138\" h=\"35\">\r\n    <t w=\"85\">devoid</t>\r\n  </item>\r\n  <item x=\"620\" y=\"132\" w=\"43\" h=\"35\">\r\n    <t w=\"82\">of</t>\r\n  </item>\r\n  <item x=\"68\" y=\"190\" w=\"137\" h=\"35\">\r\n    <t w=\"87\">natural</t>\r\n  </item>\r\n  <item x=\"215\" y=\"190\" w=\"187\" h=\"39\">\r\n    <t w=\"57\">wonders,</t>\r\n    <t w=\"55\">wanders,</t>\r\n    <t w=\"52\">wonders ?</t>\r\n    <t w=\"45\">wonders</t>\r\n    <t w=\"42\">wonders.</t>\r\n  </item>\r\n  <item x=\"410\" y=\"198\" w=\"98\" h=\"36\">\r\n    <t w=\"88\">your</t>\r\n  </item>\r\n  <item x=\"518\" y=\"190\" w=\"102\" h=\"35\">\r\n    <t w=\"86\">mind</t>\r\n  </item>\r\n  <item x=\"630\" y=\"190\" w=\"69\" h=\"34\">\r\n    <t w=\"87\">will</t>\r\n  </item>\r\n  <item x=\"709\" y=\"190\" w=\"55\" h=\"35\">\r\n    <t w=\"82\">be</t>\r\n  </item>\r\n  <item x=\"774\" y=\"190\" w=\"137\" h=\"34\">\r\n    <t w=\"56\">blown,</t>\r\n    <t w=\"55\">blown'</t>\r\n    <t w=\"48\">blown.</t>\r\n    <t w=\"48\">blown ?</t>\r\n  </item>\r\n  <item x=\"166\" y=\"736\" w=\"8\" h=\"6\">\r\n    <t w=\"66\">-e</t>\r\n  </item>\r\n  <item x=\"273\" y=\"966\" w=\"29\" h=\"21\">\r\n    <t w=\"11\">i ?</t>\r\n  </item>\r\n  <item x=\"281\" y=\"1004\" w=\"28\" h=\"11\">\r\n    <t w=\"11\">,1</t>\r\n  </item>\r\n  <item x=\"512\" y=\"1083\" w=\"10\" h=\"7\">\r\n    <t w=\"10\">IL</t>\r\n  </item>\r\n  <item x=\"29\" y=\"1447\" w=\"7\" h=\"23\">\r\n    <t w=\"17\">it ?</t>\r\n    <t w=\"15\">at ?</t>\r\n    <t w=\"13\">KY ?</t>\r\n  </item>\r\n  <item x=\"414\" y=\"841\" w=\"8\" h=\"16\">\r\n    <t w=\"22\">fl</t>\r\n    <t w=\"20\">ft</t>\r\n    <t w=\"20\">bat</t>\r\n    <t w=\"19\">at</t>\r\n  </item>\r\n</recoIndex>\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718949182, "label": "Better handling of OCR data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785201", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6", "id": 706785201, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc4NTIwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T23:29:39Z", "updated_at": "2020-10-11T23:29:39Z", "author_association": "MEMBER", "body": "It looks to me like each of those `<item>` blocks has a number of guesses in order of confidence:\r\n```xml\r\n  <item x=\"215\" y=\"190\" w=\"187\" h=\"39\">\r\n    <t w=\"57\">wonders,</t>\r\n    <t w=\"55\">wanders,</t>\r\n    <t w=\"52\">wonders ?</t>\r\n    <t w=\"45\">wonders</t>\r\n    <t w=\"42\">wonders.</t>\r\n  </item>\r\n```\r\nSo maybe the best approach here is to just take the first `t` element within each `item`.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718949182, "label": "Better handling of OCR data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/7#issuecomment-777827396", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7", "id": 777827396, "node_id": "MDEyOklzc3VlQ29tbWVudDc3NzgyNzM5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-11T22:13:14Z", "updated_at": "2021-02-11T22:13:14Z", "author_association": "MEMBER", "body": "My best guess is that you have an older version of `sqlite-utils` installed here - the `replace=True` argument was added in version 2.0. I've bumped the dependency in `setup.py`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 743297582, "label": "evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace'"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/9#issuecomment-777821383", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9", "id": 777821383, "node_id": "MDEyOklzc3VlQ29tbWVudDc3NzgyMTM4Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-11T22:01:28Z", "updated_at": "2021-02-11T22:01:28Z", "author_association": "MEMBER", "body": "Aha! I think I've figured out what's going on here.\r\n\r\nThe CData blocks containing the notes look like this:\r\n\r\n`<![CDATA[<!DOCTYPE en-note SYSTEM \"http://xml.evernote.com/pub/enml2.dtd\"><en-note><div>This note includes two images.</div><div><br /></div>...`\r\n\r\nThe DTD at http://xml.evernote.com/pub/enml2.dtd includes some entities:\r\n\r\n```\r\n<!--=========== External character mnemonic entities ===================-->\r\n\r\n<!ENTITY % HTMLlat1 PUBLIC\r\n   \"-//W3C//ENTITIES Latin 1 for XHTML//EN\"\r\n   \"http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent\">\r\n%HTMLlat1;\r\n\r\n<!ENTITY % HTMLsymbol PUBLIC\r\n   \"-//W3C//ENTITIES Symbols for XHTML//EN\"\r\n   \"http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent\">\r\n%HTMLsymbol;\r\n\r\n<!ENTITY % HTMLspecial PUBLIC\r\n   \"-//W3C//ENTITIES Special for XHTML//EN\"\r\n   \"http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent\">\r\n%HTMLspecial;\r\n```\r\nSo I need to be able to handle all of those different entities. I think I can do that using `html.entities.entitydefs` from the Python standard library, which looks a bit like this:\r\n\r\n```python\r\n{'Aacute': '\u00c1',\r\n 'aacute': '\u00e1',\r\n 'Aacute;': '\u00c1',\r\n 'aacute;': '\u00e1',\r\n 'Abreve;': '\u0102',\r\n 'abreve;': '\u0103',\r\n 'ac;': '\u223e',\r\n 'acd;': '\u223f',\r\n# ...\r\n}\r\n```\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 748372469, "label": "ParseError: undefined entity &scaron;"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544646516", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 544646516, "node_id": "MDEyOklzc3VlQ29tbWVudDU0NDY0NjUxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-21T18:30:14Z", "updated_at": "2019-10-21T18:30:14Z", "author_association": "MEMBER", "body": "Thanks to help from Dr. Laura Cantino at Science Hack Day San Francisco I've been able to pull together this query:\r\n\r\n```sql\r\nselect rsid, genotype, case genotype\r\n  when 'AA' then 'brown eye color, 80% of the time'\r\n  when 'AG' then 'brown eye color'\r\n  when 'GG' then 'blue eye color, 99% of the time'\r\nend as interpretation from genome where rsid = 'rs12913832'\r\n```\r\n\r\nSee also https://www.snpedia.com/index.php/Rs12913832 - in particular this table:\r\n\r\n<img width=\"321\" alt=\"rs12913832_-_SNPedia\" src=\"https://user-images.githubusercontent.com/9599/67232392-216ff300-f3f6-11e9-8e14-b5f50c0c0d16.png\">\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544648863", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 544648863, "node_id": "MDEyOklzc3VlQ29tbWVudDU0NDY0ODg2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-21T18:36:03Z", "updated_at": "2019-10-21T18:36:03Z", "author_association": "MEMBER", "body": "<img width=\"1418\" alt=\"natalie__select_rsid__genotype__case_genotype_when__AA__then__brown_eye_color__80__of_the_time__when__AG__then__brown_eye_color__when__GG__then__blue_eye_color__99__of_the_time__end_as_interpretation_from_genome_where_rsid____rs12913832__an\" src=\"https://user-images.githubusercontent.com/9599/67232810-f4701000-f3f6-11e9-90e2-8fe2cca1d98d.png\">\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-549230337", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10", "id": 549230337, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTIzMDMzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-04T05:47:18Z", "updated_at": "2019-11-04T05:47:18Z", "author_association": "MEMBER", "body": "This definition isn't quite right - it's not pulling the identity of the user who starred the repo (`users.login` ends up being the owner login instead).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 516967682, "label": "Add this repos_starred view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622461122", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10", "id": 622461122, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTEyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:34:39Z", "updated_at": "2020-05-01T16:34:39Z", "author_association": "MEMBER", "body": "Blocked on #37", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 516967682, "label": "Add this repos_starred view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622980203", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10", "id": 622980203, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk4MDIwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T16:34:29Z", "updated_at": "2020-05-02T16:34:29Z", "author_association": "MEMBER", "body": "Fixed definition:\r\n```sql\r\nselect\r\n  stars.starred_at,\r\n  starring_user.login as starred_by,\r\n  repos.*\r\nfrom\r\n  repos\r\n  join stars on repos.id = stars.repo\r\n  join users as starring_user on stars.user = starring_user.id\r\n  join users on repos.owner = users.id\r\norder by\r\n  starred_at desc;\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 516967682, "label": "Add this repos_starred view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594151327", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 594151327, "node_id": "MDEyOklzc3VlQ29tbWVudDU5NDE1MTMyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-03T20:26:15Z", "updated_at": "2020-03-03T20:32:23Z", "author_association": "MEMBER", "body": "Better version (since this also includes JSON array of repository topics):\r\n```sql\r\nCREATE VIEW recent_releases AS select\r\n  repos.rowid as rowid,\r\n  json_object(\"label\", repos.full_name, \"href\", repos.html_url) as repo,\r\n  json_object(\r\n    \"href\",\r\n    releases.html_url,\r\n    \"label\",\r\n    releases.name\r\n  ) as release,\r\n  substr(releases.published_at, 0, 11) as date,\r\n  releases.body as body_markdown,\r\n  releases.published_at,\r\n  coalesce(repos.topics, '[]') as topics\r\nfrom\r\n  releases\r\n  join repos on repos.id = releases.repo\r\norder by\r\n  releases.published_at desc\r\n```\r\nThat `repos.rowid as rowid` bit is necessary because otherwise clicking on a link in facet-by-topic doesn't return any results.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594155249", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 594155249, "node_id": "MDEyOklzc3VlQ29tbWVudDU5NDE1NTI0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-03T20:35:17Z", "updated_at": "2020-03-03T20:35:17Z", "author_association": "MEMBER", "body": "`swarm-to-sqlite` has an example of adding views here: https://github.com/dogsheep/swarm-to-sqlite/blob/f2c89dd613fb8a7f14e5267ccc2145463b996190/swarm_to_sqlite/utils.py#L141\r\n\r\nI think that approach can be approved by first checking if the view exists, then dropping it, then recreating it. Could even try to see if the view exists and matches what we were going to set it to and do nothing if that is the case.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 622461223, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:34:52Z", "updated_at": "2020-05-01T16:34:52Z", "author_association": "MEMBER", "body": "Blocked on #37", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 622989874, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T17:46:14Z", "updated_at": "2020-05-02T17:46:14Z", "author_association": "MEMBER", "body": "Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 622998813, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T18:58:17Z", "updated_at": "2020-05-02T18:58:17Z", "author_association": "MEMBER", "body": "Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602861730", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602861730, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2MTczMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:19:16Z", "updated_at": "2020-03-23T21:19:42Z", "author_association": "MEMBER", "body": "I'm going to do this with a scheduled GitHub Action in this repo.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862236", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602862236, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2MjIzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:20:26Z", "updated_at": "2020-03-23T21:20:26Z", "author_association": "MEMBER", "body": "I'll run the `commits` and `issues` and `issue-comments` commands in addition to the `releases` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862967", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602862967, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2Mjk2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:22:04Z", "updated_at": "2020-03-23T21:22:04Z", "author_association": "MEMBER", "body": "Following these instructions: https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602895896", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602895896, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg5NTg5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T22:42:25Z", "updated_at": "2020-03-23T22:42:25Z", "author_association": "MEMBER", "body": "Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602916947", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602916947, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxNjk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:38:06Z", "updated_at": "2020-03-23T23:38:06Z", "author_association": "MEMBER", "body": "Woohoo! https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602917713", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602917713, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxNzcxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:40:29Z", "updated_at": "2020-03-23T23:40:29Z", "author_association": "MEMBER", "body": "Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602918689", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602918689, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxODY4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:43:39Z", "updated_at": "2020-03-23T23:47:50Z", "author_association": "MEMBER", "body": "I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602919058", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602919058, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxOTA1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:44:48Z", "updated_at": "2020-03-23T23:44:48Z", "author_association": "MEMBER", "body": "Next step: use a `metadata.json` file to add some extras. And add the `datasette-render-markdown` plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602924714", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602924714, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyNDcxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-24T00:03:25Z", "updated_at": "2020-03-24T00:03:25Z", "author_association": "MEMBER", "body": "This is good enough for the 1.0 release.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559883311", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14", "id": 559883311, "node_id": "MDEyOklzc3VlQ29tbWVudDU1OTg4MzMxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-29T21:30:37Z", "updated_at": "2019-11-29T21:30:37Z", "author_association": "MEMBER", "body": "I should build the command to persist ETags and obey their polling guidelines:\r\n\r\n> Events are optimized for polling with the \"ETag\" header. If no new events have been triggered, you will see a \"304 Not Modified\" response, and your current rate limit will be untouched. There is also an \"X-Poll-Interval\" header that specifies how often (in seconds) you are allowed to poll. In times of high server load, the time may increase. Please obey the header.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 530491074, "label": "Command for importing events"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559902818", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14", "id": 559902818, "node_id": "MDEyOklzc3VlQ29tbWVudDU1OTkwMjgxOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-30T01:32:38Z", "updated_at": "2019-11-30T01:32:38Z", "author_association": "MEMBER", "body": "Prototype:\r\n```\r\npip install sqlite-utils paginate-json\r\npaginate-json \"https://api.github.com/users/simonw/events\" | sqlite-utils insert /tmp/events.db events - --pk=id\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 530491074, "label": "Command for importing events"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-613641947", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14", "id": 613641947, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMzY0MTk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-14T19:38:24Z", "updated_at": "2020-04-14T19:38:34Z", "author_association": "MEMBER", "body": "Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 530491074, "label": "Command for importing events"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602309825", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602309825, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMwOTgyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T00:52:38Z", "updated_at": "2020-03-23T00:52:38Z", "author_association": "MEMBER", "body": "Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the `releases` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602311701", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602311701, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMxMTcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T01:05:15Z", "updated_at": "2020-03-23T01:06:14Z", "author_association": "MEMBER", "body": "None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0\r\n\r\n```\r\ngithub-to-sqlite releases releases.db instrumenta/conftest\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null}