{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/6#issuecomment-1493442956", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6", "id": 1493442956, "node_id": "IC_kwDOJHON9s5ZBCGM", "user": {"value": 14314871, "label": "amlestin"}, "created_at": "2023-04-02T21:20:43Z", "updated_at": "2023-04-02T21:25:37Z", "author_association": "NONE", "body": "I'm experiencing something similar. My apostrophes (') turn into (\u201a\u00c4\u00f4) and the output is truncated. Hoping to debug next weekend \r\n\"Screenshot\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617602868, "label": "Character encoding problem"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/6#issuecomment-1508784533", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6", "id": 1508784533, "node_id": "IC_kwDOJHON9s5Z7jmV", "user": {"value": 579727, "label": "sirnacnud"}, "created_at": "2023-04-14T15:22:09Z", "updated_at": "2023-04-14T15:22:09Z", "author_association": "NONE", "body": "Just changing the encoding in `extract_notes` to `utf8` seems to fix it for my titles that were messed up.\r\n\r\n![Screen Shot 2023-04-14 at 5 14 18 PM](https://user-images.githubusercontent.com/579727/232086062-e7edc4d1-0880-417a-925b-fd6c65b05155.png)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617602868, "label": "Character encoding problem"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/8#issuecomment-1468898285", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/8", "id": 1468898285, "node_id": "IC_kwDOJHON9s5XjZvt", "user": {"value": 41546558, "label": "RhetTbull"}, "created_at": "2023-03-14T22:00:21Z", "updated_at": "2023-03-14T22:00:21Z", "author_association": "NONE", "body": "Well that's embarrassing. I made a fork using macnotesapp and it's actually slower. This is because the Scripting Bridge sometimes fails to return the folder and thus macnotesapp resorts to AppleScript in this situation. The repeated AppleScript calls on a large library are slower than your \"slurp it all in\" approach. I've got some ideas about how to improve this--will make another attempt if I can fix the issues.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617823309, "label": "Increase performance using macnotesapp"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436115", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15", "id": 748436115, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQzNjExNQ==", "user": {"value": 8573886, "label": "nickvazz"}, "created_at": "2020-12-19T07:43:38Z", "updated_at": "2020-12-19T07:47:36Z", "author_association": "NONE", "body": "Hey Simon! I really enjoy datasette so far, just started trying it out today following your iPhone photos [example](https://simonwillison.net/2020/May/21/dogsheep-photos/). \r\n\r\nI am not sure if you had run into this or not, but it seems like they might have changed one of the column names from\r\n`ZGENERICASSET` to `ZASSET`. Should I open a PR? \r\n\r\nWould change:\r\n- [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L209-L213)\r\n- [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L238)\r\n- [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L240)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612151767, "label": "Expose scores from ZCOMPUTEDASSETATTRIBUTES"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633234781", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 633234781, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMzIzNDc4MQ==", "user": {"value": 41439, "label": "dmd"}, "created_at": "2020-05-24T13:56:13Z", "updated_at": "2020-05-24T13:56:13Z", "author_association": "NONE", "body": "As that seems to be closed, can you give a hint on how to make this work?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-748436195", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21", "id": 748436195, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQzNjE5NQ==", "user": {"value": 8573886, "label": "nickvazz"}, "created_at": "2020-12-19T07:44:32Z", "updated_at": "2020-12-19T07:44:49Z", "author_association": "NONE", "body": "I have also run into this a bit, would it be possible to post your `requirements.txt` so I can try and reproduce your [blog post](https://simonwillison.net/2020/May/21/dogsheep-photos/)?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 615474990, "label": "bpylist.archiver.CircularReference: archive has a cycle with uid(13)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/28#issuecomment-751125270", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/28", "id": 751125270, "node_id": "MDEyOklzc3VlQ29tbWVudDc1MTEyNTI3MA==", "user": {"value": 129786, "label": "jmelloy"}, "created_at": "2020-12-24T22:26:22Z", "updated_at": "2020-12-24T22:26:22Z", "author_association": "NONE", "body": "This comes around if you\u2019ve run the photo export without running an s3 upload. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 624490929, "label": "Invalid SQL no such table: main.uploads"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/3#issuecomment-934207940", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3", "id": 934207940, "node_id": "IC_kwDOD079W843ruHE", "user": {"value": 1751612, "label": "jratike80"}, "created_at": "2021-10-05T08:57:41Z", "updated_at": "2021-10-05T08:57:41Z", "author_association": "NONE", "body": "Maybe the exif-loader from the SpatiaLite project could be useful as a reference even it is written in C and it also saves images as blobs https://www.gaia-gis.it/fossil/spatialite-tools/file?name=exif_loader.c&ci=tip. The tool is also integrated into the spatialite-gui application.\r\n\r\nI found some user documentation from the web archive http://web.archive.org/web/20180629041238/https://www.gaia-gis.it/spatialite-2.3.1/spatialite-exif-2.3.1.html.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533481, "label": "Import EXIF data into SQLite - lens used, ISO, aperture etc"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-811362316", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31", "id": 811362316, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMTM2MjMxNg==", "user": {"value": 871250, "label": "PabloLerma"}, "created_at": "2021-03-31T19:14:39Z", "updated_at": "2021-03-31T19:14:39Z", "author_association": "NONE", "body": "\ud83d\udc4b could I help somehow for this to be merged? As Big Sur is going to be more used as the time goes I think it would be nice to merge and publish a new version. Nice work!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771511344, "label": "Update for Big Sur"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1035717429", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31", "id": 1035717429, "node_id": "IC_kwDOD079W849u8s1", "user": {"value": 18504, "label": "harperreed"}, "created_at": "2022-02-11T01:55:38Z", "updated_at": "2022-02-11T01:55:38Z", "author_association": "NONE", "body": "I would love this merged! ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771511344, "label": "Update for Big Sur"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1190995982", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31", "id": 1190995982, "node_id": "IC_kwDOD079W85G_SgO", "user": {"value": 19231792, "label": "jakewilkins"}, "created_at": "2022-07-21T03:26:38Z", "updated_at": "2023-04-14T22:41:31Z", "author_association": "NONE", "body": "\ud83d\udc4b Any update on getting this merged?\r\n\r\nAlternatively, is there a work around for this issue to unblock myself?\r\n\r\nedit to add: huge fan of both this project and `osxphotos`, thanks so much for your work here \ud83d\ude4f If I had any experience with Python I would offer to help but somehow I've managed to not write any Python in 10+ years of programming \ud83d\ude05 \r\n\r\nEdit again to add:\r\n\r\n> Alternatively, is there a work around for this issue to unblock myself?\r\n\r\nYes, there is. I was able to apply the patch of this PR and it applies (mostly) cleanly and works.\r\n\r\n- verified I have a high enough version of `osxphotos`\r\n- downloaded the .patch of this (by appending `.patch` to the URL)\r\n- edited the patch to remove the `setup.py` changes\r\n- `cd` to the directory containing `dogsheep-photos` and `git apply 31.patch`\r\n", "reactions": "{\"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771511344, "label": "Update for Big Sur"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1382655354", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31", "id": 1382655354, "node_id": "IC_kwDOD079W85SaaV6", "user": {"value": 2704860, "label": "fidiego"}, "created_at": "2023-01-14T04:08:36Z", "updated_at": "2023-01-14T04:08:36Z", "author_association": "NONE", "body": "I just tried this branch and saw some errors. \r\n\r\nI installed this PR locally with:\r\n\r\n```bash\r\npip install https://github.com/RhetTbull/dogsheep-photos/archive/update_for_bigsur.zip\r\n```\r\n\r\n
\r\nSystem Details\r\n\r\n**OS:** MacOS Monterey\r\n**Python Version:** Python 3.10.8\r\n\r\n
\r\n\r\n
\r\nStacktrace\r\n\r\n```python\r\nTraceback (most recent call last):\r\n File \"/Users/df/.venvs/photo-experiments/bin/dogsheep-photos\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py\", line 1657, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/dogsheep_photos/cli.py\", line 254, in apple_photos\r\n sha256 = calculate_hash(pathlib.Path(photo.path))\r\n File \"/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py\", line 960, in __new__\r\n self = cls._from_parts(args)\r\n File \"/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py\", line 594, in _from_parts\r\n drv, root, parts = self._parse_args(args)\r\n File \"/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py\", line 578, in _parse_args\r\n a = os.fspath(a)\r\nTypeError: expected str, bytes or os.PathLike object, not NoneType\r\n```\r\n
", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771511344, "label": "Update for Big Sur"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-791053721", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32", "id": 791053721, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MTA1MzcyMQ==", "user": {"value": 6213, "label": "dsisnero"}, "created_at": "2021-03-05T00:31:27Z", "updated_at": "2021-03-05T00:31:27Z", "author_association": "NONE", "body": "I am getting the same thing for US West (N. California) us-west-1", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 803333769, "label": "KeyError: 'Contents' on running upload"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-882091516", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32", "id": 882091516, "node_id": "IC_kwDOD079W840k6X8", "user": {"value": 10793464, "label": "aaronyih1"}, "created_at": "2021-07-18T17:29:39Z", "updated_at": "2021-07-18T17:33:02Z", "author_association": "NONE", "body": "Same here for US West (N. California) us-west-1. Running on Catalina.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 803333769, "label": "KeyError: 'Contents' on running upload"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-884688833", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32", "id": 884688833, "node_id": "IC_kwDOD079W840u0fB", "user": {"value": 10793464, "label": "aaronyih1"}, "created_at": "2021-07-22T06:40:25Z", "updated_at": "2021-07-22T06:40:25Z", "author_association": "NONE", "body": "The solution here is to upload an image to the bucket first. It is caused because it does not properly handle the case when there are no images in the bucket.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 803333769, "label": "KeyError: 'Contents' on running upload"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-777951854", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33", "id": 777951854, "node_id": "MDEyOklzc3VlQ29tbWVudDc3Nzk1MTg1NA==", "user": {"value": 675335, "label": "leafgarland"}, "created_at": "2021-02-12T03:54:39Z", "updated_at": "2021-02-12T03:54:39Z", "author_association": "NONE", "body": "I think that is a typo in the docs, you can use\r\n\r\n > dogsheep-photos apple-photos photos.db", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 803338729, "label": "photo-to-sqlite: command not found"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778002092", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33", "id": 778002092, "node_id": "MDEyOklzc3VlQ29tbWVudDc3ODAwMjA5Mg==", "user": {"value": 11855322, "label": "robmarkcole"}, "created_at": "2021-02-12T06:19:32Z", "updated_at": "2021-02-12T06:19:32Z", "author_association": "NONE", "body": "hi @leafgarland that results in a new error:\r\n```\r\n(venv) (base) Robins-MacBook:datasette robin$ dogsheep-photos apple-photos photos.db\r\nTraceback (most recent call last):\r\n File \"/Users/robin/datasette/venv/bin/dogsheep-photos\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/cli.py\", line 206, in apple_photos\r\n db.conn.execute(\r\nsqlite3.OperationalError: no such table: attached.ZGENERICASSET\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 803338729, "label": "photo-to-sqlite: command not found"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778014990", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33", "id": 778014990, "node_id": "MDEyOklzc3VlQ29tbWVudDc3ODAxNDk5MA==", "user": {"value": 675335, "label": "leafgarland"}, "created_at": "2021-02-12T06:54:14Z", "updated_at": "2021-02-12T06:54:14Z", "author_association": "NONE", "body": "Ahh, that might be because macOS Big Sur has changed the structure of the photos db. Might need to wait for a later release, there is a PR which adds support for Big Sur. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 803338729, "label": "photo-to-sqlite: command not found"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/35#issuecomment-813249000", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/35", "id": 813249000, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzI0OTAwMA==", "user": {"value": 1151557, "label": "ligurio"}, "created_at": "2021-04-05T07:37:57Z", "updated_at": "2021-04-05T07:37:57Z", "author_association": "NONE", "body": "There are trained ML models used in Photoprism:\r\n- https://dl.photoprism.org/tensorflow/nasnet.zip\r\n- https://dl.photoprism.org/tensorflow/nsfw.zip", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 842695374, "label": "Support to annotate photos on other than macOS OSes"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/pull/36#issuecomment-1006708046", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/36", "id": 1006708046, "node_id": "IC_kwDOD079W848ASVO", "user": {"value": 71983, "label": "scoates"}, "created_at": "2022-01-06T16:04:46Z", "updated_at": "2022-01-06T16:04:46Z", "author_association": "NONE", "body": "This one got me, today, too. \ud83d\udc4d", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 988493790, "label": "Correct naming of tool in readme"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/7#issuecomment-906015471", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7", "id": 906015471, "node_id": "IC_kwDOD079W842ALLv", "user": {"value": 18232, "label": "dkam"}, "created_at": "2021-08-26T02:01:01Z", "updated_at": "2021-08-26T02:01:01Z", "author_association": "NONE", "body": "Perceptual hashes might be what you're after : http://phash.org", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602585497, "label": "Integrate image content hashing"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep.github.io/pull/6#issuecomment-1021264135", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/6", "id": 1021264135, "node_id": "IC_kwDODMzF1s4830EH", "user": {"value": 1151557, "label": "ligurio"}, "created_at": "2022-01-25T14:52:40Z", "updated_at": "2022-01-25T14:52:40Z", "author_association": "NONE", "body": "@simonw, could you review?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 842765105, "label": "Add testres-db tool"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777690332", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11", "id": 777690332, "node_id": "MDEyOklzc3VlQ29tbWVudDc3NzY5MDMzMg==", "user": {"value": 3613583, "label": "dskrad"}, "created_at": "2021-02-11T18:16:01Z", "updated_at": "2021-02-11T18:16:01Z", "author_association": "NONE", "body": "I solved this issue by modifying line 31 of utils.py\r\n\r\n content = ET.tostring(ET.fromstring(content_xml.strip())).decode(\"utf-8\")", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 792851444, "label": "XML parse error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/14#issuecomment-911772943", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/14", "id": 911772943, "node_id": "IC_kwDOEhK-wc42WI0P", "user": {"value": 46968, "label": "step21"}, "created_at": "2021-09-02T14:53:11Z", "updated_at": "2021-09-02T14:53:11Z", "author_association": "NONE", "body": "Additionally, assuming the line numbers match up with the provided enenx file, the mentioned line plus one before and after is as follows:\r\n```\r\n]]>\r\n\r\n

\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 986829194, "label": "xml.etree.ElementTree.Parse Error - mismatched tag"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765495861", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 765495861, "node_id": "MDEyOklzc3VlQ29tbWVudDc2NTQ5NTg2MQ==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-01-22T15:44:00Z", "updated_at": "2021-01-22T15:44:00Z", "author_association": "NONE", "body": "Risk of autoimmune disorders: https://www.snpedia.com/index.php/Genotype\r\n```\r\nselect rsid, genotype, case genotype\r\n when 'AA' then '2x risk of rheumatoid arthritis and other autoimmune diseases'\r\n when 'GG' then 'Normal risk for autoimmune disorders'\r\nend as interpretation from genome where rsid = 'rs2476601'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765498984", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 765498984, "node_id": "MDEyOklzc3VlQ29tbWVudDc2NTQ5ODk4NA==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-01-22T15:48:25Z", "updated_at": "2021-01-22T15:49:33Z", "author_association": "NONE", "body": "The \"Warrior Gene\" https://www.snpedia.com/index.php/Rs4680\r\n\r\n```\r\nselect rsid, genotype, case genotype\r\n when 'AA' then '(worrier) advantage in memory and attention tasks'\r\n when 'AG' then 'Intermediate dopamine levels, other effects'\r\n when 'GG' then '(warrior) multiple associations, see details'\r\nend as interpretation from genome where rsid = 'rs4680'\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765502845", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 765502845, "node_id": "MDEyOklzc3VlQ29tbWVudDc2NTUwMjg0NQ==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-01-22T15:53:19Z", "updated_at": "2021-01-22T15:53:19Z", "author_association": "NONE", "body": "rs7903146 Influences risk of Type-2 diabetes\r\nhttps://www.snpedia.com/index.php/Rs7903146\r\n```\r\nselect rsid, genotype, case genotype\r\n when 'CC' then 'Normal (lower) risk of Type 2 Diabetes and Gestational Diabetes.'\r\n when 'CT' then '1.4x increased risk for diabetes (and perhaps colon cancer).'\r\n when 'TT' then '2x increased risk for Type-2 diabetes'\r\nend as interpretation from genome where rsid = 'rs7903146'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765506901", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 765506901, "node_id": "MDEyOklzc3VlQ29tbWVudDc2NTUwNjkwMQ==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-01-22T15:58:41Z", "updated_at": "2021-01-22T15:58:58Z", "author_association": "NONE", "body": "Both rs10757274 and rs2383206 can both indicate higher risks of heart disease\r\nhttps://www.snpedia.com/index.php/Rs2383206\r\n\r\n```\r\nselect rsid, genotype, case genotype\r\n when 'AA' then 'Normal'\r\n when 'AG' then '~1.2x increased risk for heart disease'\r\n when 'GG' then '~1.3x increased risk for heart disease'\r\nend as interpretation from genome where rsid = 'rs10757274'\r\n```\r\n\r\n```\r\nselect rsid, genotype, case genotype\r\n when 'AA' then 'Normal'\r\n when 'AG' then '1.4x increased risk for heart disease'\r\n when 'GG' then '1.7x increased risk for heart disease'\r\nend as interpretation from genome where rsid = 'rs2383206'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765523517", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 765523517, "node_id": "MDEyOklzc3VlQ29tbWVudDc2NTUyMzUxNw==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-01-22T16:20:25Z", "updated_at": "2021-01-22T16:20:25Z", "author_association": "NONE", "body": "rs53576: the oxytocin receptor (OXTR) gene\r\n\r\n```\r\nselect rsid, genotype, case genotype\r\n when 'AA' then 'Lack of empathy?'\r\n when 'AG' then 'Lack of empathy?'\r\n when 'GG' then 'Optimistic and empathetic; handle stress well'\r\nend as interpretation from genome where rsid = 'rs53576'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765525338", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 765525338, "node_id": "MDEyOklzc3VlQ29tbWVudDc2NTUyNTMzOA==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-01-22T16:22:44Z", "updated_at": "2021-01-22T16:22:44Z", "author_association": "NONE", "body": "rs1333049 associated with coronary artery disease\r\nhttps://www.snpedia.com/index.php/Rs1333049\r\n```\r\n\r\nselect rsid, genotype, case genotype\r\n when 'CC' then '1.9x increased risk for coronary artery disease'\r\n when 'CG' then '1.5x increased risk for CAD'\r\n when 'GG' then 'normal'\r\nend as interpretation from genome where rsid = 'rs1333049'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-831004775", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 831004775, "node_id": "MDEyOklzc3VlQ29tbWVudDgzMTAwNDc3NQ==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-05-03T03:46:23Z", "updated_at": "2021-05-03T03:46:23Z", "author_association": "NONE", "body": "RS1800955 is related to novelty seeking and ADHD\r\nhttps://www.snpedia.com/index.php/Rs1800955\r\n\r\n`select rsid, genotype, case genotype\r\n when 'CC' then 'increased susceptibility to novelty seeking'\r\n when 'CT' then 'increased susceptibility to novelty seeking'\r\n when 'TT' then 'normal'\r\nend as interpretation from genome where rsid = 'rs1800955'`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-605439685", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 605439685, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTQzOTY4NQ==", "user": {"value": 2029, "label": "garethr"}, "created_at": "2020-03-28T12:17:01Z", "updated_at": "2020-03-28T12:17:01Z", "author_association": "NONE", "body": "That looks great, thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-571412923", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16", "id": 571412923, "node_id": "MDEyOklzc3VlQ29tbWVudDU3MTQxMjkyMw==", "user": {"value": 15092, "label": "jayvdb"}, "created_at": "2020-01-07T03:06:46Z", "updated_at": "2020-01-07T03:06:46Z", "author_association": "NONE", "body": "I re-tried after doing `auth`, and I get the same result.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546051181, "label": "Exception running first command: IndexError: list index out of range"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-602136481", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16", "id": 602136481, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjEzNjQ4MQ==", "user": {"value": 15092, "label": "jayvdb"}, "created_at": "2020-03-22T02:08:57Z", "updated_at": "2020-03-22T02:08:57Z", "author_association": "NONE", "body": "I'd love to be using your library as a better cached gh layer for a new library I have built, replacing large parts of the very ugly https://github.com/jayvdb/pypidb/blob/master/pypidb/_github.py , and then probably being able to rebuild the setuppy chunk as a feature here at a later stage.\r\n\r\nI would also need tokenless and netrc support, but I would be happy to add those bits.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546051181, "label": "Exception running first command: IndexError: list index out of range"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622279374", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33", "id": 622279374, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjI3OTM3NA==", "user": {"value": 2029, "label": "garethr"}, "created_at": "2020-05-01T07:12:47Z", "updated_at": "2020-05-01T07:12:47Z", "author_association": "NONE", "body": "I also go it working with:\r\n\r\n```yaml\r\nrun: echo ${{ secrets.github_token }} | github-to-sqlite auth\r\n```", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 609950090, "label": "Fall back to authentication via ENV"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038148", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38", "id": 623038148, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzAzODE0OA==", "user": {"value": 5779832, "label": "zzeleznick"}, "created_at": "2020-05-03T01:18:57Z", "updated_at": "2020-05-03T01:18:57Z", "author_association": "NONE", "body": "Thanks, @simonw! \r\n\r\nI feel a little foolish in hindsight, but I'm on the same page now and am glad to have discovered first-hand a motivation for this `repos_starred` use case.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 611284481, "label": "[Feature Request] Support Repo Name in Search \ud83e\udd7a"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623044643", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38", "id": 623044643, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzA0NDY0Mw==", "user": {"value": 5779832, "label": "zzeleznick"}, "created_at": "2020-05-03T02:34:32Z", "updated_at": "2020-05-03T02:34:32Z", "author_association": "NONE", "body": "1. More than glad to share feedback from the sidelines as a [starrer](https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++starred_at%2C%0D%0A++starred_by%2C%0D%0A++full_name+as+repo_name%0D%0Afrom%0D%0A++repos_starred%0D%0Awhere%0D%0A++starred_by+%3D+%22zzeleznick%22%0D%0Aorder+by%0D%0A++starred_at+desc). \r\n\r\n```\r\n-- Motivation:\r\n-- Datasette is a data hammer and I'm looking for nails\r\n-- e.g. Find which repos a user has starred => trigger a TBD downstream action\r\nselect\r\n starred_at,\r\n starred_by,\r\n full_name as repo_name\r\nfrom\r\n repos_starred\r\nwhere\r\n starred_by = \"zzeleznick\"\r\norder by\r\n starred_at desc\r\n``` \r\n\r\n| starred_at | starred_by | repo_name |\r\n| --- | --- | --- |\r\n| 2020-02-11T01:08:59Z | zzeleznick | dogsheep/twitter-to-sqlite |\r\n| 2020-01-11T21:57:34Z | zzeleznick | simonw/datasette |\r\n\r\n2. In my day job, I use [airflow](https://github.com/apache/airflow), and that's the mental model I'm bringing to [datasette](https://github.com/simonw/datasette). \r\n\r\n3. I see your project like [twitter-to-sqlite](https://github.com/dogsheep/twitter-to-sqlite) akin to [Operators](https://airflow.apache.org/docs/stable/_api/index.html#pythonapi-operators) in Airflow world.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 611284481, "label": "[Feature Request] Support Repo Name in Search \ud83e\udd7a"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-1359468823", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46", "id": 1359468823, "node_id": "IC_kwDODFdgUs5RB9kX", "user": {"value": 1839645, "label": "choldgraf"}, "created_at": "2022-12-20T14:39:39Z", "updated_at": "2022-12-20T14:40:15Z", "author_association": "NONE", "body": "Just a quick +1 to this one from me - I would like to do a better job of tracking who is reviewing one another's pull requests in repositories, since this is a specific kind of maintenance work that I think often goes unrewarded. I can't seem to figure this out just by looking at the `pull_request` or `issue_comments` tables, so I think it would be helpful to support PR reviews natively (even if just for summary statistics). Alternatively if there is a way in the API to tell if an issue comment is part of a review, then perhaps you could quickly calculate the number of unique reviews that an author performed. But that was beyond my SQL-foo :-) ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 664485022, "label": "Feature: pull request reviews and comments"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-1208757153", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51", "id": 1208757153, "node_id": "IC_kwDODFdgUs5IDCuh", "user": {"value": 9020979, "label": "hydrosquall"}, "created_at": "2022-08-09T00:29:44Z", "updated_at": "2022-08-09T00:29:44Z", "author_association": "NONE", "body": "I've been looking into how to to get this data out of Github (especially now there are \"secondary rate limits\" without an advertised allowance separate from the regular rate limits. \r\n\r\nI've had decent success with the Airbyte github extractor (aside from one data quality issue https://github.com/airbytehq/airbyte/pull/15420 ). Airbyte splits data extraction between the GraphQL and REST endpoints depending on the resource type, but they're very comprehensive. \r\n\r\nhttps://github.com/airbytehq/airbyte/blob/306a75ef5370728e0912cf52a1a898a530db0c90/airbyte-integrations/connectors/source-github/source_github/streams.py#L22-L122\r\n\r\nBefore this, I tried a few solutions in my own custom wrapper mentioned in this thread + its children https://github.com/PyGithub/PyGithub/issues/1989 , but they weren't working as expected.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 703246031, "label": "github-to-sqlite should handle rate limits better"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-1279224780", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51", "id": 1279224780, "node_id": "IC_kwDODFdgUs5MP2vM", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-10-14T16:34:07Z", "updated_at": "2022-10-14T16:34:07Z", "author_association": "NONE", "body": "also, it says that authenticated requests have a much higher \"rate limit\". Unauthenticated requests only get 60 req/hour ?? seems more like a quota than a \"rate limit\" (although I guess that is semantic equivalence)\r\n\r\nYou would want to use `x-ratelimit-reset`\r\n\r\n```\r\ntime.sleep(r['x-ratelimit-reset'] + 1 - time.time())\r\n```\r\n\r\nBut a more complete solution would bring authenticated requests to the other subcommands. I'm surprised only `github-to-sqlite get` is using the `--auth=` CLI flag", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 703246031, "label": "github-to-sqlite should handle rate limits better"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-860895838", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64", "id": 860895838, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MDg5NTgzOA==", "user": {"value": 231498, "label": "khimaros"}, "created_at": "2021-06-14T18:23:21Z", "updated_at": "2021-06-14T21:37:35Z", "author_association": "NONE", "body": "i have a basic working version at https://github.com/khimaros/github-to-sqlite\r\n\r\nthis can be tested with `github-to-sqlite events.db khimaros/events`\r\n\r\ncaveat: the GitHub API doesn't seem to provide a complete history of events.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 920636216, "label": "feature: support \"events\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861035862", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64", "id": 861035862, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MTAzNTg2Mg==", "user": {"value": 231498, "label": "khimaros"}, "created_at": "2021-06-14T22:29:20Z", "updated_at": "2021-06-14T22:29:20Z", "author_association": "NONE", "body": "it looks like the v4 GraphQL API is the only way to get data beyond 90 days from GitHub.\r\n\r\nthis is significant change, but may be worth considering in the future.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 920636216, "label": "feature: support \"events\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861087651", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64", "id": 861087651, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MTA4NzY1MQ==", "user": {"value": 231498, "label": "khimaros"}, "created_at": "2021-06-15T00:48:37Z", "updated_at": "2021-06-15T00:48:37Z", "author_association": "NONE", "body": "@simonw -- i've created an omega-query that fetched most of what was interesting to me for a single user.\r\n\r\nfound by poking around in the \"Explorer\" tab in https://docs.github.com/en/graphql/overview/explorer\r\n\r\nnote: pagination is still required via `first` and `last` but it seems to allow unlimited history.\r\n\r\n```\r\nquery MyQuery {\r\n __typename\r\n user(login: \"\") {\r\n id\r\n pinnedItems(first: 100) {\r\n edges {\r\n node\r\n }\r\n }\r\n pullRequests(first: 100) {\r\n nodes {\r\n body\r\n title\r\n state\r\n createdAt\r\n }\r\n }\r\n createdAt\r\n issues(first: 100) {\r\n pageInfo {\r\n endCursor\r\n startCursor\r\n }\r\n nodes {\r\n title\r\n url\r\n createdAt\r\n body\r\n }\r\n }\r\n issueComments(first: 100) {\r\n edges {\r\n node {\r\n id\r\n updatedAt\r\n url\r\n body\r\n }\r\n }\r\n }\r\n repositories(first: 100) {\r\n nodes {\r\n createdAt\r\n description\r\n parent {\r\n name\r\n }\r\n pinnedIssues(first: 100) {\r\n edges {\r\n node {\r\n id\r\n }\r\n }\r\n }\r\n pinnedDiscussions(first: 100) {\r\n edges {\r\n node {\r\n id\r\n }\r\n }\r\n }\r\n }\r\n }\r\n starredRepositories(first: 100) {\r\n edges {\r\n node {\r\n id\r\n }\r\n }\r\n }\r\n }\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 920636216, "label": "feature: support \"events\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/pull/65#issuecomment-885964242", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65", "id": 885964242, "node_id": "IC_kwDODFdgUs40zr3S", "user": {"value": 231498, "label": "khimaros"}, "created_at": "2021-07-23T23:45:35Z", "updated_at": "2021-07-23T23:45:35Z", "author_association": "NONE", "body": "@simonw is this PR of interest to you?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 923270900, "label": "basic support for events"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/pull/65#issuecomment-1266141699", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65", "id": 1266141699, "node_id": "IC_kwDODFdgUs5Ld8oD", "user": {"value": 231498, "label": "khimaros"}, "created_at": "2022-10-03T22:35:03Z", "updated_at": "2022-10-03T22:35:03Z", "author_association": "NONE", "body": "@simonw rebased against latest, please let me know if i should drop this PR.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 923270900, "label": "basic support for events"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/pull/66#issuecomment-929651819", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/66", "id": 929651819, "node_id": "IC_kwDODFdgUs43aVxr", "user": {"value": 30531572, "label": "sarcasticadmin"}, "created_at": "2021-09-28T21:50:31Z", "updated_at": "2021-09-28T21:50:31Z", "author_association": "NONE", "body": "@simonw any feedback/thoughts? ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 975161924, "label": "Add --merged-by flag to pull-requests sub command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/pull/76#issuecomment-1238190601", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/76", "id": 1238190601, "node_id": "IC_kwDODFdgUs5JzUoJ", "user": {"value": 2757699, "label": "OverkillGuy"}, "created_at": "2022-09-06T13:58:20Z", "updated_at": "2022-09-06T13:59:08Z", "author_association": "NONE", "body": "Tested PR just now in private org, fetched >2k repos infos flawlessly!\r\n\r\n poetry run github-to-sqlite repos --organization github.db MYORG", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1363280254, "label": "Add organization support to repos command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/issues/10#issuecomment-1073152522", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/10", "id": 1073152522, "node_id": "IC_kwDODFE5qs4_9wIK", "user": {"value": 9290214, "label": "csusanu"}, "created_at": "2022-03-20T02:38:07Z", "updated_at": "2022-03-20T02:38:07Z", "author_association": "NONE", "body": "[This line](https://github.com/dogsheep/google-takeout-to-sqlite/blob/e54e544427f1cc3ea8189f0e95f54046301a8645/google_takeout_to_sqlite/utils.py) needs to say `\"MyActivity.json\"` instead of `\"My Activity.json\"`. Google must have changed the file name.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1123393829, "label": "sqlite3.OperationalError: no such table: main.my_activity"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/issues/2#issuecomment-747130908", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2", "id": 747130908, "node_id": "MDEyOklzc3VlQ29tbWVudDc0NzEzMDkwOA==", "user": {"value": 231498, "label": "khimaros"}, "created_at": "2020-12-17T00:47:04Z", "updated_at": "2020-12-17T00:47:43Z", "author_association": "NONE", "body": "it looks like almost all of the memory consumption is coming from `json.load()`.\r\n\r\nanother direction here may be to use the new \"Semantic Location History\" data which is already broken down by year and month.\r\n\r\nit also provides much more interesting data, such as estimated address, form of travel, etc.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 769376447, "label": "killed by oomkiller on large location-history"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-780817596", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4", "id": 780817596, "node_id": "MDEyOklzc3VlQ29tbWVudDc4MDgxNzU5Ng==", "user": {"value": 306240, "label": "UtahDave"}, "created_at": "2021-02-17T20:01:35Z", "updated_at": "2021-02-17T20:01:35Z", "author_association": "NONE", "body": "I've got this almost working. Just needs some polish", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 778380836, "label": "Feature Request: Gmail"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-781451701", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4", "id": 781451701, "node_id": "MDEyOklzc3VlQ29tbWVudDc4MTQ1MTcwMQ==", "user": {"value": 203343, "label": "Btibert3"}, "created_at": "2021-02-18T16:06:21Z", "updated_at": "2021-02-18T16:06:21Z", "author_association": "NONE", "body": "Awesome!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 778380836, "label": "Feature Request: Gmail"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-783688547", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4", "id": 783688547, "node_id": "MDEyOklzc3VlQ29tbWVudDc4MzY4ODU0Nw==", "user": {"value": 306240, "label": "UtahDave"}, "created_at": "2021-02-22T21:31:28Z", "updated_at": "2021-02-22T21:31:28Z", "author_association": "NONE", "body": "@Btibert3 I've opened a PR with my initial attempt at this. Would you be willing to give this a try?\r\n\r\nhttps://github.com/dogsheep/google-takeout-to-sqlite/pull/5", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 778380836, "label": "Feature Request: Gmail"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790198930", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4", "id": 790198930, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDE5ODkzMA==", "user": {"value": 203343, "label": "Btibert3"}, "created_at": "2021-03-04T00:58:40Z", "updated_at": "2021-03-04T00:58:40Z", "author_association": "NONE", "body": "I am just seeing this sorry, yes! I will kick the tires later on tonight. My apologies for the delay.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 778380836, "label": "Feature Request: Gmail"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790934616", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4", "id": 790934616, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDkzNDYxNg==", "user": {"value": 203343, "label": "Btibert3"}, "created_at": "2021-03-04T20:54:44Z", "updated_at": "2021-03-04T20:54:44Z", "author_association": "NONE", "body": "Sorry for the delay, I got sidetracked after class last night. I am getting the following error:\r\n\r\n```\r\n/content# google-takeout-to-sqlite mbox takeout.db Takeout/Mail/gmail.mbox \r\nUsage: google-takeout-to-sqlite [OPTIONS] COMMAND [ARGS]...Try 'google-takeout-to-sqlite --help' for help.\r\n\r\nError: No such command 'mbox'.\r\n```\r\n\r\nOn the box, I installed with pip after cloning: https://github.com/UtahDave/google-takeout-to-sqlite.git", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 778380836, "label": "Feature Request: Gmail"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-783794520", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 783794520, "node_id": "MDEyOklzc3VlQ29tbWVudDc4Mzc5NDUyMA==", "user": {"value": 306240, "label": "UtahDave"}, "created_at": "2021-02-23T01:13:54Z", "updated_at": "2021-02-23T01:13:54Z", "author_association": "NONE", "body": "Also, @simonw I created a test based off the existing tests. I think it's working correctly", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-784638394", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 784638394, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NDYzODM5NA==", "user": {"value": 306240, "label": "UtahDave"}, "created_at": "2021-02-24T00:36:18Z", "updated_at": "2021-02-24T00:36:18Z", "author_association": "NONE", "body": "I noticed that @simonw is using black for formatting. I ran black on my additions in this PR.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790389335", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790389335, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM4OTMzNQ==", "user": {"value": 306240, "label": "UtahDave"}, "created_at": "2021-03-04T07:32:04Z", "updated_at": "2021-03-04T07:32:04Z", "author_association": "NONE", "body": "> The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count:\r\n> \r\n> https://github.com/dogsheep/google-takeout-to-sqlite/blob/a3de045eba0fae4b309da21aa3119102b0efc576/google_takeout_to_sqlite/utils.py#L66-L67\r\n> \r\n> I'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long.\r\n\r\nThe wait is from python loading the mbox file. This happens regardless if you're getting the length of the mbox. The mbox module is on the slow side. It is possible to do one's own parsing of the mbox, but I kind of wanted to avoid doing that.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790391711", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790391711, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM5MTcxMQ==", "user": {"value": 306240, "label": "UtahDave"}, "created_at": "2021-03-04T07:36:24Z", "updated_at": "2021-03-04T07:36:24Z", "author_association": "NONE", "body": "> Looks like you're doing this:\r\n> \r\n> ```python\r\n> elif message.get_content_type() == \"text/plain\":\r\n> body = message.get_payload(decode=True)\r\n> ```\r\n> \r\n> So presumably that decodes to a unicode string?\r\n> \r\n> I imagine the reason the column is a `BLOB` for me is that `sqlite-utils` determines the column type based on the first batch of items - https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1927-L1928 - and I got unlucky and had something in my first batch that wasn't a unicode string.\r\n\r\nAh, that's good to know. I think explicitly creating the tables will be a great improvement. I'll add that.\r\n\r\nAlso, I noticed after I opened this PR that the `message.get_payload()` is being deprecated in favor of `message.get_content()` or something like that. I'll see if that handles the decoding better, too.\r\n\r\nThanks for the feedback. I should have time tomorrow to put together some improvements.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791089881", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 791089881, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MTA4OTg4MQ==", "user": {"value": 28565, "label": "maxhawkins"}, "created_at": "2021-03-05T02:03:19Z", "updated_at": "2021-03-05T02:03:19Z", "author_association": "NONE", "body": "I just tried to run this on a small VPS instance with 2GB of memory and it crashed out of memory while processing a 12GB mbox from Takeout.\r\n\r\nIs it possible to stream the emails to sqlite instead of loading it all into memory and upserting at once?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791530093", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 791530093, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MTUzMDA5Mw==", "user": {"value": 306240, "label": "UtahDave"}, "created_at": "2021-03-05T16:28:07Z", "updated_at": "2021-03-05T16:28:07Z", "author_association": "NONE", "body": "> I just tried to run this on a small VPS instance with 2GB of memory and it crashed out of memory while processing a 12GB mbox from Takeout.\r\n> \r\n> Is it possible to stream the emails to sqlite instead of loading it all into memory and upserting at once?\r\n\r\n@maxhawkins a limitation of the python mbox module is it loads the entire mbox into memory. I did find another approach to this problem that didn't use the builtin python mbox module and created a generator so that it didn't have to load the whole mbox into memory. I was hoping to use standard library modules, but this might be a good reason to investigate that approach a bit more. My worry is making sure a custom processor handles all the ins and outs of the mbox format correctly.\r\n\r\nHm. As I'm writing this, I thought of something. I think I can parse each message one at a time, and then use an mbox function to load each message using the python mbox module. That way the mbox module can still deal with the specifics of the mbox format, but I can use a generator.\r\n\r\nI'll give that a try. Thanks for the feedback @maxhawkins and @simonw. I'll give that a try.\r\n\r\n@simonw can we hold off on merging this until I can test this new approach?", "reactions": "{\"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-849708617", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 849708617, "node_id": "MDEyOklzc3VlQ29tbWVudDg0OTcwODYxNw==", "user": {"value": 28565, "label": "maxhawkins"}, "created_at": "2021-05-27T15:01:42Z", "updated_at": "2021-05-27T15:01:42Z", "author_association": "NONE", "body": "Any updates?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-884672647", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 884672647, "node_id": "IC_kwDODFE5qs40uwiH", "user": {"value": 28565, "label": "maxhawkins"}, "created_at": "2021-07-22T05:56:31Z", "updated_at": "2021-07-22T14:03:08Z", "author_association": "NONE", "body": "How does this commit look? https://github.com/maxhawkins/google-takeout-to-sqlite/commit/72802a83fee282eb5d02d388567731ba4301050d\r\n\r\nIt seems that Takeout's mbox format is pretty simple, so we can get away with just splitting the file on lines begining with `From `. My commit just splits the file every time a line starts with `From ` and uses `email.message_from_bytes` to parse each chunk.\r\n\r\nI was able to load a 12GB takeout mbox without the program using more than a couple hundred MB of memory during the import process. It does make us lose the progress bar, but maybe I can add that back in a later commit.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885022230", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 885022230, "node_id": "IC_kwDODFE5qs40wF4W", "user": {"value": 28565, "label": "maxhawkins"}, "created_at": "2021-07-22T15:51:46Z", "updated_at": "2021-07-22T15:51:46Z", "author_association": "NONE", "body": "One thing I noticed is this importer doesn't save attachments along with the body of the emails. It would be nice if those got stored as blobs in a separate attachments table so attachments can be included while fetching search results.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885094284", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 885094284, "node_id": "IC_kwDODFE5qs40wXeM", "user": {"value": 28565, "label": "maxhawkins"}, "created_at": "2021-07-22T17:41:32Z", "updated_at": "2021-07-22T17:41:32Z", "author_association": "NONE", "body": "I added a follow-up commit that deals with emails that don't have a `Date` header: https://github.com/maxhawkins/google-takeout-to-sqlite/commit/4bc70103582c10802c85a523ef1e99a8a2154aa9", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885098025", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 885098025, "node_id": "IC_kwDODFE5qs40wYYp", "user": {"value": 306240, "label": "UtahDave"}, "created_at": "2021-07-22T17:47:50Z", "updated_at": "2021-07-22T17:47:50Z", "author_association": "NONE", "body": "Hi @maxhawkins , I'm sorry, I haven't had any time to work on this. I'll have some time tomorrow to test your commits. I think they look great. I'm great with your commits superseding my initial attempt here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-888075098", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 888075098, "node_id": "IC_kwDODFE5qs407vNa", "user": {"value": 28565, "label": "maxhawkins"}, "created_at": "2021-07-28T07:18:56Z", "updated_at": "2021-07-28T07:18:56Z", "author_association": "NONE", "body": "> I'm not sure why but my most recent import, when displayed in Datasette, looks like this:\r\n> \r\n> \"mbox__mbox_emails__753_446_rows\"\r\n\r\nI did some investigation into this issue and made a fix [here](https://github.com/dogsheep/google-takeout-to-sqlite/pull/8/commits/8ee555c2889a38ff42b95664ee074b4a01a82f06). The problem was that some messages (like gchat logs) don't have a `Message-Id` and we need to use `X-GM-THRID` as the pkey instead.\r\n\r\n@simonw While looking into this I found something unexpected about how sqlite_utils handles upserts if the pkey column is `None`. When the pkey is NULL I'd expect the function to either use rowid or throw an exception. Instead, it seems upsert_all creates a row where all columns are NULL instead of using the values provided as parameters.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-894581223", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8", "id": 894581223, "node_id": "IC_kwDODFE5qs41Ujnn", "user": {"value": 28565, "label": "maxhawkins"}, "created_at": "2021-08-07T00:57:48Z", "updated_at": "2021-08-07T00:57:48Z", "author_association": "NONE", "body": "Just added two more fixes:\r\n\r\n* Added parsing for rfc 2047 encoded unicode headers\r\n* Body is now stored as TEXT rather than a BLOB regardless of what order the messages are parsed in.\r\n\r\nI was able to run this on my Takeout export and everything seems to work fine. @simonw let me know if this looks good to merge.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 954546309, "label": "Add Gmail takeout mbox import (v2)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-896378525", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8", "id": 896378525, "node_id": "IC_kwDODFE5qs41baad", "user": {"value": 28565, "label": "maxhawkins"}, "created_at": "2021-08-10T23:28:45Z", "updated_at": "2021-08-10T23:28:45Z", "author_association": "NONE", "body": "I added parsing of text/html emails using BeautifulSoup.\r\n\r\nAround half of the emails in my archive don't include a text/plain payload so adding html parsing makes a good chunk of them searchable.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 954546309, "label": "Add Gmail takeout mbox import (v2)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1002735370", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8", "id": 1002735370, "node_id": "IC_kwDODFE5qs47xIcK", "user": {"value": 203343, "label": "Btibert3"}, "created_at": "2021-12-29T18:58:23Z", "updated_at": "2021-12-29T18:58:23Z", "author_association": "NONE", "body": "@maxhawkins how hard would it be to add an entry to the table that includes the HTML version of the email, if it exists? I just attempted your the PR branch on a very small mbox file, and it worked great. My use case is a research project and I need to access more than just the body plain text.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 954546309, "label": "Add Gmail takeout mbox import (v2)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1003437288", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8", "id": 1003437288, "node_id": "IC_kwDODFE5qs47zzzo", "user": {"value": 28565, "label": "maxhawkins"}, "created_at": "2021-12-31T19:06:20Z", "updated_at": "2021-12-31T19:06:20Z", "author_association": "NONE", "body": "> @maxhawkins how hard would it be to add an entry to the table that includes the HTML version of the email, if it exists? I just attempted your the PR branch on a very small mbox file, and it worked great. My use case is a research project and I need to access more than just the body plain text.\r\n\r\nShouldn't be hard. The easiest way is probably to remove the `if body.content_type == \"text/html\"` clause from [utils.py:254](https://github.com/dogsheep/google-takeout-to-sqlite/pull/8/commits/8e6d487b697ce2e8ad885acf613a157bfba84c59#diff-25ad9dd1ced1b8bfc37fda8444819c803232c08891e4af3d4064aa205d8174eaR254) and just return content directly without parsing.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 954546309, "label": "Add Gmail takeout mbox import (v2)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/pull/6#issuecomment-1489110168", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/6", "id": 1489110168, "node_id": "IC_kwDODtX3eM5YwgSY", "user": {"value": 1231935, "label": "xavdid"}, "created_at": "2023-03-29T18:36:16Z", "updated_at": "2023-03-29T18:36:16Z", "author_association": "NONE", "body": "@simonw can you take a look when you have a chance?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1641117021, "label": "Add permalink virtual field to items table"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711083698", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11", "id": 711083698, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMTA4MzY5OA==", "user": {"value": 572, "label": "jarib"}, "created_at": "2020-10-17T21:39:15Z", "updated_at": "2020-10-17T21:39:15Z", "author_association": "NONE", "body": "Nice! Works perfectly. Thanks for the quick response and great tooling in general.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 723838331, "label": "export.xml file name varies with different language settings"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-877805513", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12", "id": 877805513, "node_id": "MDEyOklzc3VlQ29tbWVudDg3NzgwNTUxMw==", "user": {"value": 956433, "label": "Mjboothaus"}, "created_at": "2021-07-11T14:03:01Z", "updated_at": "2021-07-11T14:03:01Z", "author_association": "NONE", "body": "Hi Simon -- just experimenting with your excellent software!\r\n\r\nUp to this point in time I have been using the (paid) [HealthFit App](https://apps.apple.com/au/app/healthfit/id1202650514) to export my workouts from my Apple Watch, one walk at the time into either .GPX or .FIT format and then using another library to suck it into Python and eventually here to my \"Emmaus Walking\" app:\r\n\r\nhttps://share.streamlit.io/mjboothaus/emmaus_walking/emmaus_walking/app.py\r\n\r\nI just used `healthkit-to-sqlite` to convert my export.zip file and it all \"just worked\".\r\n\r\nI did notice the issue with various numeric fields being stored in the SQLite db as TEXT for now and just thought I'd flag it - but you're already self-reported this issue.\r\n\r\nKeep up the great work!\r\n\r\nI was curious if you have any thoughts about periodically exporting \"export.zip\" and how to just update the SQLite file instead of re-creating it each time. Hopefully Apple will give some thought to managing this data in a more sensible fashion as it grows over time. Ideally one could pull it from iCloud (where it is allegedly being backed up).\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 727848625, "label": "Some workout columns should be float, not text"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-877874117", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12", "id": 877874117, "node_id": "MDEyOklzc3VlQ29tbWVudDg3Nzg3NDExNw==", "user": {"value": 956433, "label": "Mjboothaus"}, "created_at": "2021-07-11T23:03:37Z", "updated_at": "2021-07-11T23:03:37Z", "author_association": "NONE", "body": "P.s. wondering if you have explored using the spatialite functionality with the location data in workouts?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 727848625, "label": "Some workout columns should be float, not text"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-1163917719", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12", "id": 1163917719, "node_id": "IC_kwDOC8tyDs5FX_mX", "user": {"value": 956433, "label": "Mjboothaus"}, "created_at": "2022-06-23T04:35:02Z", "updated_at": "2022-06-23T04:35:02Z", "author_association": "NONE", "body": "In terms of unique identifiers - could you use values stored in `HKMetadataKeySyncIdentifier`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 727848625, "label": "Some workout columns should be float, not text"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/pull/13#issuecomment-904642396", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/13", "id": 904642396, "node_id": "IC_kwDOC8tyDs41679c", "user": {"value": 32016596, "label": "FabianHertwig"}, "created_at": "2021-08-24T13:27:40Z", "updated_at": "2021-08-24T13:28:26Z", "author_association": "NONE", "body": "This would fix #21 and make #22 obsolete.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 743071410, "label": "SQLite does not have case sensitive columns"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798436026", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14", "id": 798436026, "node_id": "MDEyOklzc3VlQ29tbWVudDc5ODQzNjAyNg==", "user": {"value": 1234956, "label": "n8henrie"}, "created_at": "2021-03-13T14:23:16Z", "updated_at": "2021-03-13T14:23:16Z", "author_association": "NONE", "body": "This PR allows my import to succeed.\r\n\r\nIt looks like some events don't have an `id`, but do have `HKExternalUUID` (which gets turned into `metadata_HKExternalUUID`), so I use this as a fallback.\r\n\r\nIf a record has neither of these, I changed it to just print the record (for debugging) and `return`.\r\n\r\nFor some odd reason this ran fine at first, and now (after removing the generated db and trying again) I'm getting a different error (duplicate column name).\r\n\r\nLooks like it may have run when I had two successive runs without remembering to delete the db in between. Will try to refactor.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771608692, "label": "UNIQUE constraint failed: workouts.id"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798468572", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14", "id": 798468572, "node_id": "MDEyOklzc3VlQ29tbWVudDc5ODQ2ODU3Mg==", "user": {"value": 1234956, "label": "n8henrie"}, "created_at": "2021-03-13T14:47:31Z", "updated_at": "2021-03-13T14:47:31Z", "author_association": "NONE", "body": "Ok, new PR works. I'm not `git` enough so I just force-pushed over the old one.\r\n\r\nI still end up with a lot of activities that are missing an `id` and therefore skipped (since this is used as the primary key). For example:\r\n\r\n```\r\n{'workoutActivityType': 'HKWorkoutActivityTypeRunning', 'duration': '35.31666666666667', 'durationUnit': 'min', 'totalDistance': '4.010870267636999', 'totalDistanceUnit': 'mi', 'totalEnergyBurned': '660.3516235351562', 'totalEnergyBurnedUnit': 'Cal', 'sourceName': 'Strava', 'sourceVersion': '22810', 'creationDate': '2020-07-16 13:38:26 -0700', 'startDate': '2020-07-16 06:38:26 -0700', 'endDate': '2020-07-16 07:13:45 -0700'}\r\n```\r\n\r\nI also end up with some unhappy characters (in the skipped events), such as: `'sourceName': 'Nathan\u2019s Apple\\xa0Watch',`.\r\n\r\nBut it's successfully making it through the file, and the resulting db opens in datasette, so I'd call that progress.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771608692, "label": "UNIQUE constraint failed: workouts.id"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1073123231", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14", "id": 1073123231, "node_id": "IC_kwDOC8tyDs4_9o-f", "user": {"value": 343884, "label": "lchski"}, "created_at": "2022-03-19T22:39:29Z", "updated_at": "2022-03-19T22:39:29Z", "author_association": "NONE", "body": "I have this issue, too, with a fresh export. None of my `Workout` entries in `export.xml` have an `id` key, though [the sample `export.xml` in the tests folder doesn\u2019t either](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/tests/zip_contents/apple_health_export/export.xml#L14-L21), so I don\u2019t think this is the culprit. Indeed, it seems @simonw is using the [`hash_id` function from `sqlite_utils`](https://sqlite-utils.datasette.io/en/stable/python-api.html#setting-an-id-based-on-the-hash-of-the-row-contents), which creates a column (`id`, in this case) based on a hash of the row\u2019s contents.\r\n\r\nWhen I run the script, a `workouts` table is created, with one entry: my first workout. No `workout_points` table is created, as [I\u2019d expect from `utils.py`](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/healthkit_to_sqlite/utils.py#L89-L90). I then get essentially the same error as noted in this thread:\r\n\r\n```Importing from HealthKit [###################################-] 98% 00:00:01\r\nTraceback (most recent call last):\r\n File \"/Users/lchski/.pyenv/versions/3.10.3/bin/healthkit-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/cli.py\", line 57, in cli\r\n convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)\r\n File \"/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/utils.py\", line 34, in convert_xml_to_sqlite\r\n workout_to_db(el, db, zipfile)\r\n File \"/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/utils.py\", line 57, in workout_to_db\r\n pk = db[\"workouts\"].insert(record, alter=True, hash_id=\"id\").last_pk\r\n File \"/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py\", line 2822, in insert\r\n return self.insert_all(\r\n File \"/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py\", line 2950, in insert_all\r\n self.insert_chunk(\r\n File \"/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py\", line 2715, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py\", line 458, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.IntegrityError: UNIQUE constraint failed: workouts.id\r\n```\r\n\r\nAre there maybe duplicate workouts in the data, which\u2019d cause multiple rows to share the same `id`? It\u2019s strange, though, that no `workout_points` is created at all. Export created from iOS 15.3.1.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771608692, "label": "UNIQUE constraint failed: workouts.id"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1073139067", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14", "id": 1073139067, "node_id": "IC_kwDOC8tyDs4_9s17", "user": {"value": 343884, "label": "lchski"}, "created_at": "2022-03-20T00:54:18Z", "updated_at": "2022-03-20T00:54:18Z", "author_association": "NONE", "body": "Update: this appears to be because of running the command twice without clearing the DB in between. Tries to insert a Workout that already exists, causing a collision on the (auto-generated) `id` column. Had a different error with a clean DB, likely due to the workout points format; will make a new issue for that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771608692, "label": "UNIQUE constraint failed: workouts.id"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1629123734", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14", "id": 1629123734, "node_id": "IC_kwDOC8tyDs5hGnSW", "user": {"value": 44622670, "label": "philipp-heinrich"}, "created_at": "2023-07-10T14:46:52Z", "updated_at": "2023-07-10T14:46:52Z", "author_association": "NONE", "body": "@simonw any chance to get this fixed soon? ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771608692, "label": "UNIQUE constraint failed: workouts.id"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/21#issuecomment-903950096", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/21", "id": 903950096, "node_id": "IC_kwDOC8tyDs414S8Q", "user": {"value": 32016596, "label": "FabianHertwig"}, "created_at": "2021-08-23T17:00:59Z", "updated_at": "2021-08-23T17:00:59Z", "author_association": "NONE", "body": "I think the issue is that I have records like these:\r\n\r\n```xml\r\n \r\n \r\n \r\n \r\n```\r\n\r\nAnd if sqlite is case insensitive, then `metadata_meal` and `metadata_Meal` result in the same column.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 977128935, "label": "Duplicate Column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/pull/22#issuecomment-904641261", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/22", "id": 904641261, "node_id": "IC_kwDOC8tyDs4167rt", "user": {"value": 32016596, "label": "FabianHertwig"}, "created_at": "2021-08-24T13:26:20Z", "updated_at": "2021-08-24T13:26:20Z", "author_association": "NONE", "body": "Did not see that #13 fixes the same issue in a similar way. You can decide which one to merge ;)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978086284, "label": "Make sure that case-insensitive column names are unique"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/24#issuecomment-1464786643", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24", "id": 1464786643, "node_id": "IC_kwDOC8tyDs5XTt7T", "user": {"value": 956433, "label": "Mjboothaus"}, "created_at": "2023-03-11T02:01:27Z", "updated_at": "2023-03-11T02:01:27Z", "author_association": "NONE", "body": "Thanks for reporting this and providing a solution -- I was puzzled by this error when I revisited my walking data and experienced this issues. I haven't tried the fix yet.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1515883470, "label": "DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 "}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/24#issuecomment-1464796494", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24", "id": 1464796494, "node_id": "IC_kwDOC8tyDs5XTwVO", "user": {"value": 956433, "label": "Mjboothaus"}, "created_at": "2023-03-11T02:23:42Z", "updated_at": "2023-03-11T02:23:42Z", "author_association": "NONE", "body": "@simonw - maybe put in some error handling to trap for poorly formed XML (from Apple engineers) so that it suggests that there are problems with export.zip rather than odd looking Python errors :)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1515883470, "label": "DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 "}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-514745798", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9", "id": 514745798, "node_id": "MDEyOklzc3VlQ29tbWVudDUxNDc0NTc5OA==", "user": {"value": 166463, "label": "tholo"}, "created_at": "2019-07-24T18:25:36Z", "updated_at": "2019-07-24T18:25:36Z", "author_association": "NONE", "body": "This is on macOS 10.14.6, with Python 3.7.4, packages in the virtual environment:\r\n\r\n```\r\nPackage Version\r\n------------------- -------\r\naiofiles 0.4.0\r\nClick 7.0\r\nclick-default-group 1.2.1\r\ndatasette 0.29.2\r\nh11 0.8.1\r\nhealthkit-to-sqlite 0.3.1\r\nhttptools 0.0.13\r\nhupper 1.8.1\r\nimportlib-metadata 0.18\r\nJinja2 2.10.1\r\nMarkupSafe 1.1.1\r\nPint 0.8.1\r\npip 19.2.1\r\npluggy 0.12.0\r\nsetuptools 41.0.1\r\nsqlite-utils 1.7\r\ntabulate 0.8.3\r\nuvicorn 0.8.4\r\nuvloop 0.12.2\r\nwebsockets 7.0\r\nzipp 0.5.2\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 472429048, "label": "Too many SQL variables"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515370687", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9", "id": 515370687, "node_id": "MDEyOklzc3VlQ29tbWVudDUxNTM3MDY4Nw==", "user": {"value": 166463, "label": "tholo"}, "created_at": "2019-07-26T09:01:19Z", "updated_at": "2019-07-26T09:01:19Z", "author_association": "NONE", "body": "Yes, that did fix the issue I was seeing \u2014 it will now import my complete HealthKit data.\n\nThorsten\n\n> On Jul 25, 2019, at 23:07, Simon Willison wrote:\n> \n> @tholo this should be fixed in just-released version 0.3.2 - could you run a pip install -U healthkit-to-sqlite and let me know if it works for you now?\n> \n> \u2014\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub , or mute the thread .\n> \n\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 472429048, "label": "Too many SQL variables"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1239516561", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10", "id": 1239516561, "node_id": "IC_kwDODLZ_YM5J4YWR", "user": {"value": 11887, "label": "ashanan"}, "created_at": "2022-09-07T15:07:38Z", "updated_at": "2022-09-07T15:07:38Z", "author_association": "NONE", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1246826792, "label": "When running `auth` command, don't overwrite an existing auth.json file"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221521377", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11", "id": 1221521377, "node_id": "IC_kwDODLZ_YM5Izu_h", "user": {"value": 2467, "label": "fernand0"}, "created_at": "2022-08-21T10:51:37Z", "updated_at": "2022-08-21T10:51:37Z", "author_association": "NONE", "body": "I didn't see there is a PR about this: https://github.com/dogsheep/pocket-to-sqlite/pull/7", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1345452427, "label": "-a option is used for \"--auth\" and for \"--all\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/9#issuecomment-774726123", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9", "id": 774726123, "node_id": "MDEyOklzc3VlQ29tbWVudDc3NDcyNjEyMw==", "user": {"value": 12669260, "label": "jfeiwell"}, "created_at": "2021-02-07T18:21:08Z", "updated_at": "2021-02-07T18:21:08Z", "author_association": "NONE", "body": "@simonw any ideas here?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 801780625, "label": "SSL Error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/9#issuecomment-774730656", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9", "id": 774730656, "node_id": "MDEyOklzc3VlQ29tbWVudDc3NDczMDY1Ng==", "user": {"value": 635179, "label": "merwok"}, "created_at": "2021-02-07T18:45:04Z", "updated_at": "2021-02-07T18:45:04Z", "author_association": "NONE", "body": "That URL uses TLS 1.3, but maybe only if the client supports it.\r\nIt could be your Python version or your SSL library that\u2019s not recent enough.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 801780625, "label": "SSL Error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/issues/12#issuecomment-941274088", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/12", "id": 941274088, "node_id": "IC_kwDODD6af844GrPo", "user": {"value": 33631, "label": "fs111"}, "created_at": "2021-10-12T18:31:57Z", "updated_at": "2021-10-12T18:31:57Z", "author_association": "NONE", "body": "I am running into the same problem. Is there any workaround?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 951817328, "label": "403 when getting token"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/31#issuecomment-1251845216", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31", "id": 1251845216, "node_id": "IC_kwDODEm0Qs5KnaRg", "user": {"value": 150986, "label": "dckc"}, "created_at": "2022-09-20T05:05:03Z", "updated_at": "2022-09-20T05:05:03Z", "author_association": "NONE", "body": "yay! Thanks a bunch for the `twitter-to-sqlite friends` command!\r\n\r\nThe twitter \"Download an archive of your data\" feature doesn't include who I follow, so this is particularly handy.\r\n\r\nThe whole Dogsheep thing is great :) I've written about similar things under [cloud-services](https://www.madmode.com/search/label/cloud-services/):\r\n - 2021: [Closet Librarian Approach to Cloud Services](https://www.madmode.com/2021/closet-librarian-approach-cloud-services.html)\r\n - 2015: [jukekb \\- Browse iTunes libraries and upload playlists to Google Music](https://www.madmode.com/2015/jukekb)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520508502, "label": "\"friends\" command (similar to \"followers\")"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645515103", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47", "id": 645515103, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NTUxNTEwMw==", "user": {"value": 73579, "label": "hpk42"}, "created_at": "2020-06-17T17:30:01Z", "updated_at": "2020-06-17T17:30:01Z", "author_association": "NONE", "body": "It's the one with python3.7::\n\n >>> sqlite3.sqlite_version\n '3.11.0'\n\n \nOn Wed, Jun 17, 2020 at 10:24 -0700, Simon Willison wrote:\n\n> That means your version of SQLite is old enough that it doesn't support the FTS5 extension.\n> \n> Could you share what operating system you're running, and what the output is that you get from running this?\n> \n> python -c 'import sqlite3; print(sqlite3.connect(\":memory:\").execute(\"select sqlite_version()\").fetchone()[0])'\n> \n> I can teach this tool to fall back on FTS4 if FTS5 isn't available.\n> \n> -- \n> You are receiving this because you authored the thread.\n> Reply to this email directly or view it on GitHub:\n> https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 639542974, "label": "Fall back to FTS4 if FTS5 is not available"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/50#issuecomment-691501132", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50", "id": 691501132, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MTUwMTEzMg==", "user": {"value": 706257, "label": "bcongdon"}, "created_at": "2020-09-12T14:48:10Z", "updated_at": "2020-09-12T14:48:10Z", "author_association": "NONE", "body": "This seems to be an issue even with larger values of `--stop_after`:\r\n\r\n```\r\n$ twitter-to-sqlite favorites twitter.db --stop_after=2000\r\nImporting favorites [####################################] 198\r\n$\r\n```", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 698791218, "label": "favorites --stop_after=N stops after min(N, 200)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/52#issuecomment-729484478", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/52", "id": 729484478, "node_id": "MDEyOklzc3VlQ29tbWVudDcyOTQ4NDQ3OA==", "user": {"value": 4169772, "label": "fatihky"}, "created_at": "2020-11-18T07:12:45Z", "updated_at": "2020-11-18T07:12:45Z", "author_association": "NONE", "body": "I'm so sorry that you already have `--since_id` option and that's enough for the case I've mentioned. Thank you for this excellent tool!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 745393298, "label": "Discussion: Adding support for fetching only fresh tweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/53#issuecomment-748436453", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/53", "id": 748436453, "node_id": "MDEyOklzc3VlQ29tbWVudDc0ODQzNjQ1Mw==", "user": {"value": 27, "label": "anotherjesse"}, "created_at": "2020-12-19T07:47:01Z", "updated_at": "2020-12-19T07:47:01Z", "author_association": "NONE", "body": "I think this should probably be closed as won't fix.\r\n\r\nAttempting to make a patch for this I realized that the since_id would limit to tweets posted since that since_id, not when it was favorited. So favoriting something in the older would be missed if you used `--since` with a cron script\r\n\r\nBetter to just use `--stop_after` set to a couple hundred", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771324837, "label": "--since support for favorites"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-767888743", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54", "id": 767888743, "node_id": "MDEyOklzc3VlQ29tbWVudDc2Nzg4ODc0Mw==", "user": {"value": 19328961, "label": "henry501"}, "created_at": "2021-01-26T23:07:41Z", "updated_at": "2021-01-26T23:07:41Z", "author_association": "NONE", "body": "My import got much further with the applied fixes than 0.21.3, but not 100%. I do appear to have all of the tweets imported at least. \r\nNot sure when I'll have a chance to look further to try to fix or see what didn't make it into the import.\r\n\r\nHere's my output:\r\n\r\n```\r\ndirect-messages-group: not yet implemented\r\nbranch-links: not yet implemented\r\nperiscope-expired-broadcasts: not yet implemented\r\ndirect-messages: not yet implemented\r\nmute: not yet implemented\r\nperiscope-comments-made-by-user: not yet implemented\r\nperiscope-ban-information: not yet implemented\r\nperiscope-profile-description: not yet implemented\r\nscreen-name-change: not yet implemented\r\nmanifest: not yet implemented\r\nfleet: not yet implemented\r\nuser-link-clicks: not yet implemented\r\nperiscope-broadcast-metadata: not yet implemented\r\ncontact: not yet implemented\r\nfleet-mute: not yet implemented\r\ndevice-token: not yet implemented\r\nprotected-history: not yet implemented\r\ndirect-message-mute: not yet implemented\r\nTraceback (most recent call last):\r\n File \"/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/bin/twitter-to-sqlite\", line 33, in \r\n sys.exit(load_entry_point('twitter-to-sqlite==0.21.3', 'console_scripts', 'twitter-to-sqlite')())\r\n File \"/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/cli.py\", line 772, in import_\r\n archive.import_from_file(db, filepath.name, open(filepath, \"rb\").read())\r\n File \"/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py\", line 233, in import_from_file\r\n to_insert = transformer(data)\r\n File \"/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py\", line 21, in callback\r\n return {filename: [fn(item) for item in data]}\r\n File \"/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py\", line 21, in \r\n return {filename: [fn(item) for item in data]}\r\n File \"/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py\", line 88, in ageinfo\r\n return item[\"ageMeta\"][\"ageInfo\"]\r\nKeyError: 'ageInfo'\r\n\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 779088071, "label": "Archive import appears to be broken on recent exports"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-927312650", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54", "id": 927312650, "node_id": "IC_kwDODEm0Qs43RasK", "user": {"value": 2182, "label": "danp"}, "created_at": "2021-09-26T14:09:51Z", "updated_at": "2021-09-26T14:09:51Z", "author_association": "NONE", "body": "Similar trouble with ageinfo using 0.22. Here's what my ageinfo.js file looks like:\r\n\r\n```\r\nwindow.YTD.ageinfo.part0 = [\r\n {\r\n \"ageMeta\" : { }\r\n }\r\n]\r\n```\r\n\r\nCommenting out the registration for ageinfo in archive.py gets my archive to import.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 779088071, "label": "Archive import appears to be broken on recent exports"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-1370786026", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54", "id": 1370786026, "node_id": "IC_kwDODEm0Qs5RtIjq", "user": {"value": 6764957, "label": "swyxio"}, "created_at": "2023-01-04T11:06:44Z", "updated_at": "2023-01-04T11:06:44Z", "author_association": "NONE", "body": "as of 2023 it appears that `tweets: not yet implemented` is happening.. pretty important for a twitter export functionality!\r\n\r\nthis seems an impossible task with twitter liable to switch this around every other day", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 779088071, "label": "Archive import appears to be broken on recent exports"}, "performed_via_github_app": null}