{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129686", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29", "id": 552129686, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEyOTY4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T19:27:39Z", "updated_at": "2019-11-09T19:27:39Z", "author_association": "MEMBER", "body": "I think this is fixed by the latest version of `sqlite-utils` - https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-12-1 - I'll bump the dependency.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 518725064, "label": "`import` command fails on empty files"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129921", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29", "id": 552129921, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEyOTkyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T19:30:42Z", "updated_at": "2019-11-09T19:30:42Z", "author_association": "MEMBER", "body": "Confirmed, that seems to fix it:\r\n```\r\n(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js \r\nTraceback (most recent call last):\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite\", line 11, in \r\n load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')()\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 764, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 717, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 1137, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 956, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 555, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py\", line 633, in import_\r\n archive.import_from_file(db, path.name, open(path, \"rb\").read())\r\n File \"/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/archive.py\", line 224, in import_from_file\r\n db[table_name].upsert_all(rows, hash_id=\"pk\")\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1094, in upsert_all\r\n extracts=extracts,\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py\", line 961, in insert_all\r\n first_record = next(records)\r\nStopIteration\r\n(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ pip install -U sqlite-utils\r\nCollecting sqlite-utils\r\n Using cached https://files.pythonhosted.org/packages/ee/a2/1b135010c7ac8e2d7545f659e9e6c6ede0f406f20b52e08d5817e1e31a9a/sqlite_utils-1.12.1-py3-none-any.whl\r\nRequirement already satisfied, skipping upgrade: click in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (7.0)\r\nRequirement already satisfied, skipping upgrade: tabulate in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (0.8.5)\r\nRequirement already satisfied, skipping upgrade: click-default-group in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (1.2.2)\r\nInstalling collected packages: sqlite-utils\r\n Found existing installation: sqlite-utils 1.11\r\n Uninstalling sqlite-utils-1.11:\r\n Successfully uninstalled sqlite-utils-1.11\r\nSuccessfully installed sqlite-utils-1.12.1\r\n(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js \r\n(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ \r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 518725064, "label": "`import` command fails on empty files"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552133449", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29", "id": 552133449, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T20:15:15Z", "updated_at": "2019-11-09T20:15:15Z", "author_association": "MEMBER", "body": "Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 518725064, "label": "`import` command fails on empty files"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552131798", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30", "id": 552131798, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEzMTc5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T19:54:45Z", "updated_at": "2019-11-09T19:54:45Z", "author_association": "MEMBER", "body": "Good catch - not sure how that bug crept in. Removing line 116 looks like the right fix to me.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 518739697, "label": "`followers` fails because `transform_user` is called twice"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552133468", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30", "id": 552133468, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T20:15:27Z", "updated_at": "2019-11-09T20:15:27Z", "author_association": "MEMBER", "body": "Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 518739697, "label": "`followers` fails because `transform_user` is called twice"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550783316", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10", "id": 550783316, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MDc4MzMxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-07T05:16:56Z", "updated_at": "2019-11-07T05:34:29Z", "author_association": "MEMBER", "body": "It looks like Apple changed the location of these in iOS 13 - they are now in separate `.gpx` files:\r\n\r\n![2FF70E95-CDEE-4241-A5C5-EE95A862E519](https://user-images.githubusercontent.com/9599/68362042-be12e000-00da-11ea-8925-7397410332d8.png)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 519038979, "label": "Failed to import workout points"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550806302", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10", "id": 550806302, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MDgwNjMwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-07T05:33:31Z", "updated_at": "2019-11-07T05:33:31Z", "author_association": "MEMBER", "body": "The XML now includes references to these new files:\r\n\r\n![CBBA54FC-51FB-4BB3-927C-C2CA99237B04](https://user-images.githubusercontent.com/9599/68362716-121ec400-00dd-11ea-9846-387c7cd64c8b.jpeg)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 519038979, "label": "Failed to import workout points"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550824838", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10", "id": 550824838, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MDgyNDgzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-07T05:47:07Z", "updated_at": "2019-11-07T05:47:07Z", "author_association": "MEMBER", "body": "Relevant code:\r\n\r\nhttps://github.com/dogsheep/healthkit-to-sqlite/blob/d16f45f06fbae6ec8a78cc9ca7b5b7db0413f139/healthkit_to_sqlite/utils.py#L58-L64", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 519038979, "label": "Failed to import workout points"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550828084", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10", "id": 550828084, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MDgyODA4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-07T05:49:24Z", "updated_at": "2019-11-07T05:49:24Z", "author_association": "MEMBER", "body": "So the fix there is going to be to detect the new `FileReference` element and load the corresponding points data from it.\r\n\r\nThis will be a little tricky because that function will need access to the zip file.\r\n\r\nIt probably won't work at all for the mode where the `export.xml` file is passed directly using the `--xml` option.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 519038979, "label": "Failed to import workout points"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/31#issuecomment-552135263", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31", "id": 552135263, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEzNTI2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T20:38:35Z", "updated_at": "2019-11-09T20:38:35Z", "author_association": "MEMBER", "body": "Command still needs documentation and a bit more testing.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520508502, "label": "\"friends\" command (similar to \"followers\")"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594151327", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 594151327, "node_id": "MDEyOklzc3VlQ29tbWVudDU5NDE1MTMyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-03T20:26:15Z", "updated_at": "2020-03-03T20:32:23Z", "author_association": "MEMBER", "body": "Better version (since this also includes JSON array of repository topics):\r\n```sql\r\nCREATE VIEW recent_releases AS select\r\n repos.rowid as rowid,\r\n json_object(\"label\", repos.full_name, \"href\", repos.html_url) as repo,\r\n json_object(\r\n \"href\",\r\n releases.html_url,\r\n \"label\",\r\n releases.name\r\n ) as release,\r\n substr(releases.published_at, 0, 11) as date,\r\n releases.body as body_markdown,\r\n releases.published_at,\r\n coalesce(repos.topics, '[]') as topics\r\nfrom\r\n releases\r\n join repos on repos.id = releases.repo\r\norder by\r\n releases.published_at desc\r\n```\r\nThat `repos.rowid as rowid` bit is necessary because otherwise clicking on a link in facet-by-topic doesn't return any results.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594155249", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 594155249, "node_id": "MDEyOklzc3VlQ29tbWVudDU5NDE1NTI0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-03T20:35:17Z", "updated_at": "2020-03-03T20:35:17Z", "author_association": "MEMBER", "body": "`swarm-to-sqlite` has an example of adding views here: https://github.com/dogsheep/swarm-to-sqlite/blob/f2c89dd613fb8a7f14e5267ccc2145463b996190/swarm_to_sqlite/utils.py#L141\r\n\r\nI think that approach can be approved by first checking if the view exists, then dropping it, then recreating it. Could even try to see if the view exists and matches what we were going to set it to and do nothing if that is the case.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 622461223, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:34:52Z", "updated_at": "2020-05-01T16:34:52Z", "author_association": "MEMBER", "body": "Blocked on #37", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 622989874, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T17:46:14Z", "updated_at": "2020-05-02T17:46:14Z", "author_association": "MEMBER", "body": "Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 622998813, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T18:58:17Z", "updated_at": "2020-05-02T18:58:17Z", "author_association": "MEMBER", "body": "Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602861730", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602861730, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2MTczMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:19:16Z", "updated_at": "2020-03-23T21:19:42Z", "author_association": "MEMBER", "body": "I'm going to do this with a scheduled GitHub Action in this repo.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862236", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602862236, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2MjIzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:20:26Z", "updated_at": "2020-03-23T21:20:26Z", "author_association": "MEMBER", "body": "I'll run the `commits` and `issues` and `issue-comments` commands in addition to the `releases` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862967", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602862967, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2Mjk2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:22:04Z", "updated_at": "2020-03-23T21:22:04Z", "author_association": "MEMBER", "body": "Following these instructions: https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602895896", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602895896, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg5NTg5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T22:42:25Z", "updated_at": "2020-03-23T22:42:25Z", "author_association": "MEMBER", "body": "Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602916947", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602916947, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxNjk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:38:06Z", "updated_at": "2020-03-23T23:38:06Z", "author_association": "MEMBER", "body": "Woohoo! https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602917713", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602917713, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxNzcxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:40:29Z", "updated_at": "2020-03-23T23:40:29Z", "author_association": "MEMBER", "body": "Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602918689", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602918689, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxODY4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:43:39Z", "updated_at": "2020-03-23T23:47:50Z", "author_association": "MEMBER", "body": "I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602919058", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602919058, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxOTA1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:44:48Z", "updated_at": "2020-03-23T23:44:48Z", "author_association": "MEMBER", "body": "Next step: use a `metadata.json` file to add some extras. And add the `datasette-render-markdown` plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602924714", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602924714, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyNDcxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-24T00:03:25Z", "updated_at": "2020-03-24T00:03:25Z", "author_association": "MEMBER", "body": "This is good enough for the 1.0 release.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559883311", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14", "id": 559883311, "node_id": "MDEyOklzc3VlQ29tbWVudDU1OTg4MzMxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-29T21:30:37Z", "updated_at": "2019-11-29T21:30:37Z", "author_association": "MEMBER", "body": "I should build the command to persist ETags and obey their polling guidelines:\r\n\r\n> Events are optimized for polling with the \"ETag\" header. If no new events have been triggered, you will see a \"304 Not Modified\" response, and your current rate limit will be untouched. There is also an \"X-Poll-Interval\" header that specifies how often (in seconds) you are allowed to poll. In times of high server load, the time may increase. Please obey the header.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 530491074, "label": "Command for importing events"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559902818", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14", "id": 559902818, "node_id": "MDEyOklzc3VlQ29tbWVudDU1OTkwMjgxOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-30T01:32:38Z", "updated_at": "2019-11-30T01:32:38Z", "author_association": "MEMBER", "body": "Prototype:\r\n```\r\npip install sqlite-utils paginate-json\r\npaginate-json \"https://api.github.com/users/simonw/events\" | sqlite-utils insert /tmp/events.db events - --pk=id\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 530491074, "label": "Command for importing events"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-613641947", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14", "id": 613641947, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMzY0MTk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-14T19:38:24Z", "updated_at": "2020-04-14T19:38:34Z", "author_association": "MEMBER", "body": "Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 530491074, "label": "Command for importing events"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/pull/6#issuecomment-605382373", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/6", "id": 605382373, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTM4MjM3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-28T02:27:32Z", "updated_at": "2020-03-28T02:27:32Z", "author_association": "MEMBER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 543355051, "label": "don't break if source is missing"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602309825", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602309825, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMwOTgyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T00:52:38Z", "updated_at": "2020-03-23T00:52:38Z", "author_association": "MEMBER", "body": "Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the `releases` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602311701", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602311701, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMxMTcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T01:05:15Z", "updated_at": "2020-03-23T01:06:14Z", "author_association": "MEMBER", "body": "None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0\r\n\r\n```\r\ngithub-to-sqlite releases releases.db instrumenta/conftest\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602312054", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602312054, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMxMjA1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T01:07:10Z", "updated_at": "2020-03-23T01:07:10Z", "author_association": "MEMBER", "body": "Each asset looks like this:\r\n```json\r\n {\r\n \"url\": \"https://api.github.com/repos/instrumenta/conftest/releases/assets/11811946\",\r\n \"id\": 11811946,\r\n \"node_id\": \"MDEyOlJlbGVhc2VBc3NldDExODExOTQ2\",\r\n \"name\": \"checksums.txt\",\r\n \"label\": \"\",\r\n \"uploader\": {\r\n \"login\": \"garethr\",\r\n \"id\": 2029,\r\n \"node_id\": \"MDQ6VXNlcjIwMjk=\",\r\n \"avatar_url\": \"https://avatars2.githubusercontent.com/u/2029?v=4\",\r\n \"gravatar_id\": \"\",\r\n \"url\": \"https://api.github.com/users/garethr\",\r\n \"html_url\": \"https://github.com/garethr\",\r\n \"followers_url\": \"https://api.github.com/users/garethr/followers\",\r\n \"following_url\": \"https://api.github.com/users/garethr/following{/other_user}\",\r\n \"gists_url\": \"https://api.github.com/users/garethr/gists{/gist_id}\",\r\n \"starred_url\": \"https://api.github.com/users/garethr/starred{/owner}{/repo}\",\r\n \"subscriptions_url\": \"https://api.github.com/users/garethr/subscriptions\",\r\n \"organizations_url\": \"https://api.github.com/users/garethr/orgs\",\r\n \"repos_url\": \"https://api.github.com/users/garethr/repos\",\r\n \"events_url\": \"https://api.github.com/users/garethr/events{/privacy}\",\r\n \"received_events_url\": \"https://api.github.com/users/garethr/received_events\",\r\n \"type\": \"User\",\r\n \"site_admin\": false\r\n },\r\n \"content_type\": \"text/plain; charset=utf-8\",\r\n \"state\": \"uploaded\",\r\n \"size\": 600,\r\n \"download_count\": 2,\r\n \"created_at\": \"2019-03-30T16:56:44Z\",\r\n \"updated_at\": \"2019-03-30T16:56:44Z\",\r\n \"browser_download_url\": \"https://github.com/instrumenta/conftest/releases/download/v0.1.0/checksums.txt\"\r\n }\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-601979709", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16", "id": 601979709, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTk3OTcwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-21T02:20:08Z", "updated_at": "2020-03-21T02:20:08Z", "author_association": "MEMBER", "body": "I'm pretty sure this means I need to upgrade the underlying `sqlite-utils` dependency. I saw the same bug in `twitter-to-sqlite`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546051181, "label": "Exception running first command: IndexError: list index out of range"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-613611455", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16", "id": 613611455, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMzYxMTQ1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-14T18:37:21Z", "updated_at": "2020-04-14T18:37:21Z", "author_association": "MEMBER", "body": "This should have been fixed by #20 and #23 \r\n\r\n@jayvdb I'm definitely interested in this tool working as a library - it's purely designed as a CLI tool at the moment, but cleaning it up to work better as a dependency is totally in-scope for the project. https://sqlite-utils.readthedocs.io/ is an example of a tool I've built that works for both.\r\n\r\nFeel free to open a new issue here with some notes on what you would need for this to work as a library for your project!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546051181, "label": "Exception running first command: IndexError: list index out of range"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/pull/33#issuecomment-601871930", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/33", "id": 601871930, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3MTkzMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:21:41Z", "updated_at": "2020-03-20T19:21:41Z", "author_association": "MEMBER", "body": "Made obsolete by https://github.com/dogsheep/twitter-to-sqlite/commit/d3887d1535c3edd59d3ebf6aea4495bacd82e59b", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 561469252, "label": "Upgrade to sqlite-utils 2.2.1"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597354514", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17", "id": 597354514, "node_id": "MDEyOklzc3VlQ29tbWVudDU5NzM1NDUxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-10T22:37:45Z", "updated_at": "2020-03-10T22:37:45Z", "author_association": "MEMBER", "body": "I should add an option to stop the moment you see a commit you have fetched before.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 578883725, "label": "Command for importing commits"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597358364", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17", "id": 597358364, "node_id": "MDEyOklzc3VlQ29tbWVudDU5NzM1ODM2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-10T22:50:20Z", "updated_at": "2020-03-11T01:18:36Z", "author_association": "MEMBER", "body": "By default it will stop when it sees a commit that has already been stored. You will be able to over-ride that behaviour using `--all`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 578883725, "label": "Command for importing commits"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601861908", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34", "id": 601861908, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg2MTkwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T18:56:44Z", "updated_at": "2020-03-20T18:56:44Z", "author_association": "MEMBER", "body": "Could this be a bug in `sqlite-utils`? This table has a primary key, so why is it running a query on `rowid = ?`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585266763, "label": "IndexError running user-timeline command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601871579", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34", "id": 601871579, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3MTU3OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:20:52Z", "updated_at": "2020-03-20T19:20:52Z", "author_association": "MEMBER", "body": "Cause was the redefining of how `upsert()` works in sqlite-utils 2.x.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585266763, "label": "IndexError running user-timeline command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875405", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601875405, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3NTQwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:30:09Z", "updated_at": "2020-03-20T19:32:33Z", "author_association": "MEMBER", "body": "I'm going to redesign the command a little. Right now it defaults to fetching the authenticated user's timeline and allows `--screen_name` to be used to fetch for one other user instead.\r\n\r\nI think the design of `users-lookup` is much better. I'm going to copy that:\r\n\r\n```\r\ntwitter-to-sqlite user-timeline users.db simonw cleopaws\r\ntwitter-to-sqlite user-timeline users.db 12497 3166449535 --ids\r\n```\r\nIt will also accept `--since` and `--sql` and `--alter` and `--since_id`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875524", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601875524, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3NTUyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:30:27Z", "updated_at": "2020-03-20T19:30:27Z", "author_association": "MEMBER", "body": "I can give it a snazzier progress bar to, as requested by #10.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601876051", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601876051, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3NjA1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:31:43Z", "updated_at": "2020-03-20T19:31:43Z", "author_association": "MEMBER", "body": "I'm going to keep `--screen_name` working for backwards compatibility. Using `hidden=True` on it should hide it from the help output.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601890795", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601890795, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg5MDc5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T20:13:08Z", "updated_at": "2020-03-20T20:13:08Z", "author_association": "MEMBER", "body": "Needs documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601892338", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601892338, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg5MjMzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T20:17:00Z", "updated_at": "2020-03-20T20:17:00Z", "author_association": "MEMBER", "body": "Released as 0.18.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509578", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37", "id": 615509578, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUwOTU3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:36:00Z", "updated_at": "2020-04-17T23:36:00Z", "author_association": "MEMBER", "body": "```\r\n$ twitter-to-sqlite user-timeline doggo.db doggoenthuonetuh\r\nTraceback (most recent call last):\r\n...\r\n File \"/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py\", line 272, in transform_user\r\n user[\"created_at\"] = parser.parse(user[\"created_at\"])\r\nKeyError: 'created_at'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585353598, "label": "Handle \"User not found\" error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509803", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37", "id": 615509803, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUwOTgwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:36:40Z", "updated_at": "2020-04-17T23:36:40Z", "author_association": "MEMBER", "body": "I'm going to print a warning to stderr, skip and continue - because if you have 100 screen names and only one of them is invalid you should still execute for the other 99.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585353598, "label": "Handle \"User not found\" error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615510361", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37", "id": 615510361, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxMDM2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:38:27Z", "updated_at": "2020-04-17T23:38:27Z", "author_association": "MEMBER", "body": "That's a bit tricky since I'd have to rewrite the internals of a bunch of other commands. For the moment I'll exit the script with an error but at least it will be a decent error!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585353598, "label": "Handle \"User not found\" error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/38#issuecomment-601939656", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/38", "id": 601939656, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTkzOTY1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T22:37:04Z", "updated_at": "2020-03-20T22:37:04Z", "author_association": "MEMBER", "body": "New formatting:\r\n```\r\n@Alaska_DHSS [####################################] 3205\r\n@BWhippleKS [####################################] 1\r\n@CAPublicHealth [####################################] 2\r\n@CDPHE [####################################] 67\r\n@CHFSKy [####################################] 3216\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585359363, "label": "Screen name display for user-timeline is uneven"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980724", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 601980724, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTk4MDcyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-21T02:28:05Z", "updated_at": "2020-03-21T02:28:05Z", "author_association": "MEMBER", "body": "This suggests that `commit[\"author\"]` can be `None` in some cases?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980957", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 601980957, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTk4MDk1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-21T02:30:03Z", "updated_at": "2020-03-21T02:30:03Z", "author_association": "MEMBER", "body": "https://github.community/t5/GitHub-API-Development-and/Request-for-commits-quot-author-quot-null-and-quot-committer/m-p/35842/highlight/true#M3372\r\n> Commits aren't always associated with a GitHub user. For example, perhaps a friend of mine and I were working on a project together. I have a GitHub account and my friend doesn't. If we both add commits to the repository using our own email addresses and names and then I push the repository to GitHub, my commits will be associated with my GitHub user account but my friends' commits will show up with `author` and `committer` as `null`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602807178", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602807178, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgwNzE3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:24:43Z", "updated_at": "2020-03-23T19:24:43Z", "author_association": "MEMBER", "body": "I need to find an example before I work on this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811468", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602811468, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxMTQ2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:33:14Z", "updated_at": "2020-03-23T19:33:14Z", "author_association": "MEMBER", "body": "Found one: https://api.github.com/repos/simonw/simonw.github.com/commits", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811785", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602811785, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxMTc4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:33:56Z", "updated_at": "2020-03-23T19:34:06Z", "author_association": "MEMBER", "body": "```json\r\n[\r\n {\r\n \"sha\": \"a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"node_id\": \"MDY6Q29tbWl0OTMyMDk6YThkYzkxNDA4OWQzOTlkOWI1MjJlYmI1MWI2N2Y5YWMyZThhYTZiMA==\",\r\n \"commit\": {\r\n \"author\": {\r\n \"name\": \"Simon Willison\",\r\n \"email\": \"simon@...\",\r\n \"date\": \"2008-12-18T23:17:12Z\"\r\n },\r\n \"committer\": {\r\n \"name\": \"Simon Willison\",\r\n \"email\": \"simon@...\",\r\n \"date\": \"2008-12-18T23:17:12Z\"\r\n },\r\n \"message\": \"First commit\",\r\n \"tree\": {\r\n \"sha\": \"ac2dfb75e2592c59165c2880f3f7a16dafd452a1\",\r\n \"url\": \"https://api.github.com/repos/simonw/simonw.github.com/git/trees/ac2dfb75e2592c59165c2880f3f7a16dafd452a1\"\r\n },\r\n \"url\": \"https://api.github.com/repos/simonw/simonw.github.com/git/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"comment_count\": 0,\r\n \"verification\": {\r\n \"verified\": false,\r\n \"reason\": \"unsigned\",\r\n \"signature\": null,\r\n \"payload\": null\r\n }\r\n },\r\n \"url\": \"https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"html_url\": \"https://github.com/simonw/simonw.github.com/commit/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"comments_url\": \"https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0/comments\",\r\n \"author\": null,\r\n \"committer\": null,\r\n \"parents\": []\r\n }\r\n]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602813731", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602813731, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxMzczMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:38:03Z", "updated_at": "2020-03-23T20:44:49Z", "author_association": "MEMBER", "body": "So it turns out `\"author\"` and `\"committer\"` on the commit are `null` if the email address in the nested `\"commit\"` doesn't match an existing GitHub user.\r\n\r\nMaybe I should be storing the nested data somewhere as well?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602815120", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602815120, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxNTEyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:40:55Z", "updated_at": "2020-03-23T19:43:19Z", "author_association": "MEMBER", "body": "I could pull a pk-hashed version of the name/email into separate `raw_author` and `raw_committer` columns perhaps - against a `commit_authors` table. Could be interesting.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602846293", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602846293, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg0NjI5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T20:44:40Z", "updated_at": "2020-03-23T20:44:40Z", "author_association": "MEMBER", "body": "I implemented the `raw_authors` idea.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/1#issuecomment-602094386", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/1", "id": 602094386, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjA5NDM4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-21T19:47:46Z", "updated_at": "2020-03-21T19:47:46Z", "author_association": "MEMBER", "body": "Released as 0.3.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585526292, "label": "Set up full text search"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602304531", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19", "id": 602304531, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMwNDUzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T00:20:28Z", "updated_at": "2020-03-23T00:20:28Z", "author_association": "MEMBER", "body": "Current FTS code:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/e35eec4343aa560c58c1634cc228d0d46c442304/github_to_sqlite/utils.py#L136-L143", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585850715, "label": "Enable full-text search for more stuff (like commits, issues and issue_comments)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602782437", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19", "id": 602782437, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjc4MjQzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T18:36:12Z", "updated_at": "2020-03-23T18:51:23Z", "author_association": "MEMBER", "body": "* `commits`: `message`\r\n* `issue_comments`: `body`\r\n* `issues`: `title`, `body`\r\n* `labels`: `name`, `description`\r\n* `licenses`: `name`\r\n* `milestones`: `title`, `description`\r\n* `releases`: `name`, `body`\r\n* `repos`: `name`, `description`\r\n* `users`: `login`, `name` (maybe `company`, `bio`)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585850715, "label": "Enable full-text search for more stuff (like commits, issues and issue_comments)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602896434", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21", "id": 602896434, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg5NjQzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T22:43:37Z", "updated_at": "2020-03-23T22:43:37Z", "author_association": "MEMBER", "body": "I'm going to do this now to help figure out the latest error in #13:\r\n```\r\nTraceback (most recent call last):\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite\", line 11, in \r\n load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')()\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 237, in commits\r\n utils.save_commits(db, commits, repo_full[\"id\"])\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 345, in save_commits\r\n for commit in commits:\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 207, in fetch_commits\r\n if stop_when(commit):\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 224, in stop_when\r\n db[\"commits\"].get(commit[\"sha\"])\r\nTypeError: string indices must be integers\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 586561727, "label": "Turn GitHub API errors into exceptions"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602920163", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21", "id": 602920163, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyMDE2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:48:22Z", "updated_at": "2020-03-23T23:48:22Z", "author_association": "MEMBER", "body": "I'm happy with this pattern:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 586561727, "label": "Turn GitHub API errors into exceptions"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/23#issuecomment-602928533", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23", "id": 602928533, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyODUzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-24T00:15:49Z", "updated_at": "2020-03-24T00:15:49Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/github-to-sqlite/releases/tag/1.0", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 586595839, "label": "Release 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606304837", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606304837, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNDgzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:27:50Z", "updated_at": "2020-03-30T23:29:31Z", "author_association": "MEMBER", "body": "One option would be something like this:\r\n```sql\r\nselect max(id) from tweets\r\nwhere user = ?\r\nand not exists (select id from tweets where retweeted_status = id)\r\nand not exists (select id from tweets where quoted_status = id)\r\nand not exists (select id from tweets where in_reply_to_status_id = id)\r\n```\r\nMight be a good idea to index those columns (after confirming that doing so would indeed speed up the query).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606305701", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606305701, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNTcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:30:27Z", "updated_at": "2020-03-30T23:30:27Z", "author_association": "MEMBER", "body": "A better alternative would be to maintain a separate table with the last seen since value for when we ran `user-timeline` for any specific user.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606309165", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606309165, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwOTE2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:41:31Z", "updated_at": "2020-03-30T23:41:31Z", "author_association": "MEMBER", "body": "I like the separate `user_timeline_since` table solution.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606824992", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606824992, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjgyNDk5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T19:24:23Z", "updated_at": "2020-03-31T19:24:23Z", "author_association": "MEMBER", "body": "The `--since` option is actually used by four commands:\r\n\r\n* `user-timeline`\r\n* `home-timeline`\r\n* `mentions-timeline`\r\n* `search`\r\n\r\nAll of them use the same `fetch_timeline()` utility function under the hood. I should move the logic that looks up the last `since_id` into that shared function.\r\n\r\nQuestion: should I have a table for each of those four methods or a single table that is used by them all? I'm leaning towards four separate tables.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606843224", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606843224, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg0MzIyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T19:59:11Z", "updated_at": "2020-03-31T20:06:32Z", "author_association": "MEMBER", "body": "Or... have a single `since_ids` table to track since values, and have its primary key be a string that looks something like this:\r\n\r\n`user:123145`\r\n`home:23441`\r\n`mentions:23425`\r\n`search:99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d`\r\n\r\nThat last example would use the hash generated here:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/cli.py#L792-L808", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606844521", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606844521, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg0NDUyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:01:39Z", "updated_at": "2020-03-31T20:01:39Z", "author_association": "MEMBER", "body": "I think `utils.fetch_timeline()` grows a new argument, `since_key`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850008", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606850008, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg1MDAwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:13:59Z", "updated_at": "2020-04-01T00:23:00Z", "author_association": "MEMBER", "body": "Table design for `since_ids` table:\r\n\r\ntype | key | since_id\r\n--- | --- | ---\r\n1 | 124324 | 2347239847293\r\n2 | 99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d | 2125947927344\r\n\r\nPrimary compound key on `(category, key)`\r\n\r\n`type` is also a foreign key to a `since_id_types` table with `id` and `name` columns (probably created using https://sqlite-utils.readthedocs.io/en/stable/python-api.html#working-with-lookup-tables )", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850453", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606850453, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg1MDQ1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:14:58Z", "updated_at": "2020-04-01T03:03:50Z", "author_association": "MEMBER", "body": "Actually I'll hard-code the population of `since_id_types` to get known ID constants.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606998669", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606998669, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjk5ODY2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T02:57:36Z", "updated_at": "2020-04-01T02:57:36Z", "author_association": "MEMBER", "body": "The tricky thing here is thinking about the interaction between the recorded since_id and a desire to run the initial import.\r\n\r\nThe first time you run `twitter-to-sqlite user-timeline db.db username` we want to fetch as many tweets from that user as possible - probably around 3,200 before the API limitations cut us off.\r\n\r\nWe need to record the maximum ID from those as the `since_id` - which we will see on the very first page we paginate through. That way next time we run the command with `--since` we will only fetch new tweets.\r\n\r\nBut what happens if our initial import is cancelled after only a few tweets? We risk never pulling in the rest of the tweets.\r\n\r\nNot sure if I need to solve this at all or if I should instead trust users to run the command a second time without `--since` if they think they didn't retrieve anything the first time.\r\n\r\nI had considered letting `--stop_after=` over-ride `--since` but that doesn't actually make sense - if you send a since_id to the Twitter API you'll never get back more tweets than exist after that ID, so the `--stop_after` would not make a meaningful difference.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607003655", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 607003655, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAwMzY1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:18:00Z", "updated_at": "2020-04-01T03:18:00Z", "author_association": "MEMBER", "body": "I've got this working for the `user-timeline` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607010634", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 607010634, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMDYzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:45:16Z", "updated_at": "2020-04-01T03:45:16Z", "author_association": "MEMBER", "body": "OK, fix is applied to everything now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307019", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 606307019, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNzAxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:34:27Z", "updated_at": "2020-03-30T23:34:27Z", "author_association": "MEMBER", "body": "The count properties available for a user are:\r\n\r\n* followers_count\r\n* friends_count\r\n* listed_count\r\n* favourites_count\r\n* statuses_count\r\n\r\nMay as well track history for all of them? Should be pretty cheap to store.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307376", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 606307376, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNzM3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:35:40Z", "updated_at": "2020-03-30T23:39:15Z", "author_association": "MEMBER", "body": "I think five separate tables:\r\n\r\n* followers_count_history\r\n* friends_count_history\r\n* listed_count_history\r\n* favourites_count_history\r\n* statuses_count_history\r\n\r\nEach with the following structure:\r\n\r\n* datetime (ISO UTC)\r\n* user (ID, foreign key to users)\r\n* count (integer)\r\n\r\nI'm tempted to have a compound primary key here - user, datetime\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011421", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 607011421, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMTQyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:47:37Z", "updated_at": "2020-04-01T03:55:08Z", "author_association": "MEMBER", "body": "Actually a single table with a `type` integer ID referencing a `count_history_types` table would better match the way I implemented the `since_ids` table:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/4b6c8d8c1cc6fefdb566ec8506157133f47c569a/twitter_to_sqlite/utils.py#L331-L341\r\n\r\nIn which case the compound primary key would be `type`, `user`, `datetime`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011972", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 607011972, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMTk3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:49:02Z", "updated_at": "2020-04-01T03:50:01Z", "author_association": "MEMBER", "body": "I want the datetime value to look like `2020-04-01T03:34:58+00:00` (the format returned by the Twitter API which I am storing in other tables at the moment).\r\n```\r\n>>> datetime.utcnow().isoformat().split('.')[0] + '+00:00'\r\n'2020-04-01T03:49:52+00:00'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607019151", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 607019151, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxOTE1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T04:11:10Z", "updated_at": "2020-04-01T04:11:10Z", "author_association": "MEMBER", "body": "In testing this collects a LOT of data. I'm going to skip tracking favourites_count and statuses_count and just track followers, friends and listed instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/25#issuecomment-614810417", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25", "id": 614810417, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDgxMDQxNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T18:07:11Z", "updated_at": "2020-04-16T18:07:11Z", "author_association": "MEMBER", "body": "Turns out the main problem was #26 - now fixed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601265023, "label": "Improvements to demo instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614794739", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26", "id": 614794739, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDc5NDczOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T17:38:28Z", "updated_at": "2020-04-16T17:38:28Z", "author_association": "MEMBER", "body": "I'm already doing this here: https://github.com/dogsheep/github-to-sqlite/blob/c4aaa50e167cfa9021c7c94260bc3e89e10947bf/github_to_sqlite/utils.py#L246-L250", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601271612, "label": "Topics are missing from repositories"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614795712", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26", "id": 614795712, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDc5NTcxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T17:40:27Z", "updated_at": "2020-04-16T17:40:27Z", "author_association": "MEMBER", "body": "Aha! it was missing from the `fetch_repo()` function.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601271612, "label": "Topics are missing from repositories"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831451", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 614831451, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDgzMTQ1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T18:47:25Z", "updated_at": "2020-04-16T18:47:25Z", "author_association": "MEMBER", "body": "Is it possible for a repo to have an `owner` that differs from its `organization`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831842", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 614831842, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDgzMTg0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T18:48:18Z", "updated_at": "2020-04-16T18:48:18Z", "author_association": "MEMBER", "body": "I'm going to make `organization` another foreign key to the `users` table just in case it IS possible (maybe with GitHub Enterprise or similar?)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 614843406, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDg0MzQwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T19:11:53Z", "updated_at": "2020-04-16T19:20:23Z", "author_association": "MEMBER", "body": "This didn't quite work: the column type is incorrect, so the foreign key relationship isn't sticking:\r\n\r\nhttps://github-to-sqlite.dogsheep.net/github/repos?organization=53015001\r\n\r\n`[organization] TEXT REFERENCES [users]([id])` - should be `INTEGER`.\r\n\r\nThe problem is that if the first repo inserted has no organization it's set to `null`, which `sqlite-utils` derives as a `TEXT` column.\r\n\r\nOne solution would be to create the column explicitly with a type, but this could get messy.\r\n\r\nI think I want a new sqlite-utils feature for this instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615518606", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 615518606, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxODYwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T00:14:32Z", "updated_at": "2020-04-18T00:14:32Z", "author_association": "MEMBER", "body": "https://github.com/simonw/sqlite-utils/issues/100 is done and released in sqlite-utils 2.7.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615519409", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 615519409, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxOTQwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T00:19:16Z", "updated_at": "2020-04-18T00:19:16Z", "author_association": "MEMBER", "body": "```\r\n$ github-to-sqlite repos b.db dogsheep\r\n$ sqlite3 b.db '.schema repos'\r\nCREATE TABLE [repos] (\r\n [id] INTEGER PRIMARY KEY,\r\n...\r\n [permissions] TEXT,\r\n [organization] INTEGER REFERENCES [users]([id]),\r\n FOREIGN KEY(license) REFERENCES licenses(key)\r\n);\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883040", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28", "id": 615883040, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTg4MzA0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T14:45:38Z", "updated_at": "2020-04-18T14:45:38Z", "author_association": "MEMBER", "body": "```\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 219, in contributors\r\n utils.save_contributors(db, contributors, repo_full[\"id\"])\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 354, in save_contributors\r\n for contributor in contributors:\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 228, in fetch_contributors\r\n for contributors in paginate(url, headers):\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 286, in paginate\r\n data = response.json()\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/requests/models.py\", line 898, in json\r\n return complexjson.loads(self.text, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/__init__.py\", line 357, in loads\r\n return _default_decoder.decode(s)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py\", line 337, in decode\r\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py\", line 355, in raw_decode\r\n raise JSONDecodeError(\"Expecting value\", s, err.value) from None\r\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601333634, "label": "Pull repository contributors"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883687", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28", "id": 615883687, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTg4MzY4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T14:49:58Z", "updated_at": "2020-04-18T14:49:58Z", "author_association": "MEMBER", "body": "That happened trying to pull contributors for `dogsheep/beta` - an empty repository. Turns out it was returning a `204 no content`:\r\n```\r\n~ $ curl -i 'https://api.github.com/repos/dogsheep/beta/contributors'\r\nHTTP/1.1 204 No Content\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601333634, "label": "Pull repository contributors"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615886206", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28", "id": 615886206, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTg4NjIwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T15:04:59Z", "updated_at": "2020-04-18T15:04:59Z", "author_association": "MEMBER", "body": "Demo: https://github-to-sqlite.dogsheep.net/github/contributors\r\n\r\nDocumentation: https://github.com/dogsheep/github-to-sqlite/blob/13f8868fb5efa01c263b24f6dd91c617e6e938e1/README.md#fetching-contributors-to-a-repository ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601333634, "label": "Pull repository contributors"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/43#issuecomment-615513491", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/43", "id": 615513491, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxMzQ5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:48:28Z", "updated_at": "2020-04-17T23:48:28Z", "author_association": "MEMBER", "body": "Released in 0.21.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602176870, "label": "\"twitter-to-sqlite lists\" command for retrieving a user's owned lists"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623193947", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623193947, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5Mzk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T22:36:17Z", "updated_at": "2020-05-03T22:36:17Z", "author_association": "MEMBER", "body": "I'm going to use [osxphotos](https://github.com/RhetTbull/osxphotos) for this.\r\n\r\nSince I've already got code to upload photos and insert them into a table based on their `sha256` hash, my first go at this will be to import data using the tool and foreign-key it to the `sha256` hash in the existing table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623195197", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623195197, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5NTE5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T22:44:33Z", "updated_at": "2020-05-03T22:44:33Z", "author_association": "MEMBER", "body": "Command will be this:\r\n\r\n $ photos-to-sqlite apple-photos photos.db\r\n\r\nThis will populate a `apple_photos` table with the data imported by the `osxphotos` library, plus the calculated sha256.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198653", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623198653, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5ODY1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T23:09:57Z", "updated_at": "2020-05-03T23:09:57Z", "author_association": "MEMBER", "body": "For locations: I'll add `place_x` columns for all of these:\r\n```\r\n(Pdb) photo.place.address._asdict()\r\n{'street': None, 'sub_locality': None, 'city': 'Loreto', 'sub_administrative_area': 'Loreto', 'state_province': 'BCS', 'postal_code': None, 'country': 'Mexico', 'iso_country_code': 'MX'}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198986", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623198986, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5ODk4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T23:12:31Z", "updated_at": "2020-05-03T23:12:46Z", "author_association": "MEMBER", "body": "To get the taken date in UTC:\r\n```\r\nfrom datetime import timezone\r\n(Pdb) photo.date.astimezone(timezone.utc).isoformat()\r\n'2018-02-13T20:21:31.620000+00:00'\r\n(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(\".\")\r\n['2018-02-13T20:21:31', '620000+00:00']\r\n(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(\".\")[0]\r\n'2018-02-13T20:21:31'\r\n(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(\".\")[0] + \"+00:00\"\r\n'2018-02-13T20:21:31+00:00'\r\n\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199214", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623199214, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5OTIxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T23:14:08Z", "updated_at": "2020-05-03T23:14:08Z", "author_association": "MEMBER", "body": "Albums have UUIDs:\r\n```\r\n(Pdb) photo.album_info[0].__dict__\r\n{'_uuid': '17816791-ABF3-447B-942C-9FA8065EEBBA', '_db': osxphotos.PhotosDB(dbfile='/Users/simon/Pictures/Photos Library.photoslibrary/database/photos.db'), '_title': 'Geotaggable Photos geotagged'}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199701", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623199701, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5OTcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T23:17:38Z", "updated_at": "2020-05-03T23:17:38Z", "author_association": "MEMBER", "body": "Record burst_uuid as a column:\r\n```\r\n(Pdb) with_bursts[0]._info[\"burstUUID\"]\r\n'703FAA23-57BF-40B4-8A33-D9CEB143391B'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199750", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623199750, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5OTc1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T23:17:58Z", "updated_at": "2020-05-03T23:17:58Z", "author_association": "MEMBER", "body": "Reading this source code is really useful for figuring out how to store a photo in a DB table: https://github.com/RhetTbull/osxphotos/blob/7444b6d173918a3ad2a07aefce5ecf054786c787/osxphotos/photoinfo.py", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623232984", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623232984, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzIzMjk4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-04T02:41:32Z", "updated_at": "2020-05-04T02:41:32Z", "author_association": "MEMBER", "body": "Needs documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/2#issuecomment-615931488", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/2", "id": 615931488, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMTQ4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:24:02Z", "updated_at": "2020-04-18T19:24:02Z", "author_association": "MEMBER", "body": "I made a start on this last week with a https://github.com/simonw/heic-to-jpeg proxy.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533352, "label": "Ability to convert HEIC images to JPEG"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932007", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615932007, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMjAwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:27:55Z", "updated_at": "2020-04-18T19:27:55Z", "author_association": "MEMBER", "body": "Research thread: https://twitter.com/simonw/status/1249049694984011776\r\n> I want to build some software that lets people store their own data in their own S3 bucket, but if possible I'd like not to have to teach people the incantations needed to get their bucket setup and minimum-permission credentials figures out\r\n\r\nhttps://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/ looks useful", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932204", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615932204, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMjIwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:29:22Z", "updated_at": "2020-04-18T19:34:44Z", "author_association": "MEMBER", "body": "I'm going to call my bucket `dogsheep-photos-simon`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}