issue_comments
525 rows where author_association = "MEMBER" sorted by issue
This data as json, CSV (advanced)
id | html_url | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue ▼ | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
552129686 | https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129686 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29 | MDEyOklzc3VlQ29tbWVudDU1MjEyOTY4Ng== | simonw 9599 | 2019-11-09T19:27:39Z | 2019-11-09T19:27:39Z | MEMBER | I think this is fixed by the latest version of `sqlite-utils` - https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-12-1 - I'll bump the dependency. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `import` command fails on empty files 518725064 | |
552129921 | https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129921 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29 | MDEyOklzc3VlQ29tbWVudDU1MjEyOTkyMQ== | simonw 9599 | 2019-11-09T19:30:42Z | 2019-11-09T19:30:42Z | MEMBER | Confirmed, that seems to fix it: ``` (twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js Traceback (most recent call last): File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite", line 11, in <module> load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')() File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py", line 633, in import_ archive.import_from_file(db, path.name, open(path, "rb").read()) File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/archive.py", line 224, in import_from_file db[table_name].upsert_all(rows, hash_id="pk") File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py", line 1094, in upsert_all extracts=extracts, File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `import` command fails on empty files 518725064 | |
552133449 | https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552133449 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29 | MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ0OQ== | simonw 9599 | 2019-11-09T20:15:15Z | 2019-11-09T20:15:15Z | MEMBER | Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `import` command fails on empty files 518725064 | |
552131798 | https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552131798 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30 | MDEyOklzc3VlQ29tbWVudDU1MjEzMTc5OA== | simonw 9599 | 2019-11-09T19:54:45Z | 2019-11-09T19:54:45Z | MEMBER | Good catch - not sure how that bug crept in. Removing line 116 looks like the right fix to me. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `followers` fails because `transform_user` is called twice 518739697 | |
552133468 | https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552133468 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30 | MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ2OA== | simonw 9599 | 2019-11-09T20:15:27Z | 2019-11-09T20:15:27Z | MEMBER | Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `followers` fails because `transform_user` is called twice 518739697 | |
550783316 | https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550783316 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU1MDc4MzMxNg== | simonw 9599 | 2019-11-07T05:16:56Z | 2019-11-07T05:34:29Z | MEMBER | It looks like Apple changed the location of these in iOS 13 - they are now in separate `.gpx` files: ![2FF70E95-CDEE-4241-A5C5-EE95A862E519](https://user-images.githubusercontent.com/9599/68362042-be12e000-00da-11ea-8925-7397410332d8.png) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Failed to import workout points 519038979 | |
550806302 | https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550806302 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU1MDgwNjMwMg== | simonw 9599 | 2019-11-07T05:33:31Z | 2019-11-07T05:33:31Z | MEMBER | The XML now includes references to these new files: ![CBBA54FC-51FB-4BB3-927C-C2CA99237B04](https://user-images.githubusercontent.com/9599/68362716-121ec400-00dd-11ea-9846-387c7cd64c8b.jpeg) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Failed to import workout points 519038979 | |
550824838 | https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550824838 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU1MDgyNDgzOA== | simonw 9599 | 2019-11-07T05:47:07Z | 2019-11-07T05:47:07Z | MEMBER | Relevant code: https://github.com/dogsheep/healthkit-to-sqlite/blob/d16f45f06fbae6ec8a78cc9ca7b5b7db0413f139/healthkit_to_sqlite/utils.py#L58-L64 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Failed to import workout points 519038979 | |
550828084 | https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550828084 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU1MDgyODA4NA== | simonw 9599 | 2019-11-07T05:49:24Z | 2019-11-07T05:49:24Z | MEMBER | So the fix there is going to be to detect the new `FileReference` element and load the corresponding points data from it. This will be a little tricky because that function will need access to the zip file. It probably won't work at all for the mode where the `export.xml` file is passed directly using the `--xml` option. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Failed to import workout points 519038979 | |
552135263 | https://github.com/dogsheep/twitter-to-sqlite/issues/31#issuecomment-552135263 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31 | MDEyOklzc3VlQ29tbWVudDU1MjEzNTI2Mw== | simonw 9599 | 2019-11-09T20:38:35Z | 2019-11-09T20:38:35Z | MEMBER | Command still needs documentation and a bit more testing. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | "friends" command (similar to "followers") 520508502 | |
594151327 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594151327 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDU5NDE1MTMyNw== | simonw 9599 | 2020-03-03T20:26:15Z | 2020-03-03T20:32:23Z | MEMBER | Better version (since this also includes JSON array of repository topics): ```sql CREATE VIEW recent_releases AS select repos.rowid as rowid, json_object("label", repos.full_name, "href", repos.html_url) as repo, json_object( "href", releases.html_url, "label", releases.name ) as release, substr(releases.published_at, 0, 11) as date, releases.body as body_markdown, releases.published_at, coalesce(repos.topics, '[]') as topics from releases join repos on repos.id = releases.repo order by releases.published_at desc ``` That `repos.rowid as rowid` bit is necessary because otherwise clicking on a link in facet-by-topic doesn't return any results. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
594155249 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594155249 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDU5NDE1NTI0OQ== | simonw 9599 | 2020-03-03T20:35:17Z | 2020-03-03T20:35:17Z | MEMBER | `swarm-to-sqlite` has an example of adding views here: https://github.com/dogsheep/swarm-to-sqlite/blob/f2c89dd613fb8a7f14e5267ccc2145463b996190/swarm_to_sqlite/utils.py#L141 I think that approach can be approved by first checking if the view exists, then dropping it, then recreating it. Could even try to see if the view exists and matches what we were going to set it to and do nothing if that is the case. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
622461223 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw== | simonw 9599 | 2020-05-01T16:34:52Z | 2020-05-01T16:34:52Z | MEMBER | Blocked on #37 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
622989874 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA== | simonw 9599 | 2020-05-02T17:46:14Z | 2020-05-02T17:46:14Z | MEMBER | Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
622998813 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw== | simonw 9599 | 2020-05-02T18:58:17Z | 2020-05-02T18:58:17Z | MEMBER | Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
602861730 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602861730 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjg2MTczMA== | simonw 9599 | 2020-03-23T21:19:16Z | 2020-03-23T21:19:42Z | MEMBER | I'm going to do this with a scheduled GitHub Action in this repo. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602862236 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862236 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjg2MjIzNg== | simonw 9599 | 2020-03-23T21:20:26Z | 2020-03-23T21:20:26Z | MEMBER | I'll run the `commits` and `issues` and `issue-comments` commands in addition to the `releases` command. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602862967 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862967 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjg2Mjk2Nw== | simonw 9599 | 2020-03-23T21:22:04Z | 2020-03-23T21:22:04Z | MEMBER | Following these instructions: https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602895896 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602895896 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjg5NTg5Ng== | simonw 9599 | 2020-03-23T22:42:25Z | 2020-03-23T22:42:25Z | MEMBER | Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602916947 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602916947 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjkxNjk0Nw== | simonw 9599 | 2020-03-23T23:38:06Z | 2020-03-23T23:38:06Z | MEMBER | Woohoo! https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602917713 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602917713 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjkxNzcxMw== | simonw 9599 | 2020-03-23T23:40:29Z | 2020-03-23T23:40:29Z | MEMBER | Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602918689 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602918689 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjkxODY4OQ== | simonw 9599 | 2020-03-23T23:43:39Z | 2020-03-23T23:47:50Z | MEMBER | I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602919058 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602919058 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjkxOTA1OA== | simonw 9599 | 2020-03-23T23:44:48Z | 2020-03-23T23:44:48Z | MEMBER | Next step: use a `metadata.json` file to add some extras. And add the `datasette-render-markdown` plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602924714 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602924714 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjkyNDcxNA== | simonw 9599 | 2020-03-24T00:03:25Z | 2020-03-24T00:03:25Z | MEMBER | This is good enough for the 1.0 release. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
559883311 | https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559883311 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14 | MDEyOklzc3VlQ29tbWVudDU1OTg4MzMxMQ== | simonw 9599 | 2019-11-29T21:30:37Z | 2019-11-29T21:30:37Z | MEMBER | I should build the command to persist ETags and obey their polling guidelines: > Events are optimized for polling with the "ETag" header. If no new events have been triggered, you will see a "304 Not Modified" response, and your current rate limit will be untouched. There is also an "X-Poll-Interval" header that specifies how often (in seconds) you are allowed to poll. In times of high server load, the time may increase. Please obey the header. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing events 530491074 | |
559902818 | https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559902818 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14 | MDEyOklzc3VlQ29tbWVudDU1OTkwMjgxOA== | simonw 9599 | 2019-11-30T01:32:38Z | 2019-11-30T01:32:38Z | MEMBER | Prototype: ``` pip install sqlite-utils paginate-json paginate-json "https://api.github.com/users/simonw/events" | sqlite-utils insert /tmp/events.db events - --pk=id ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing events 530491074 | |
613641947 | https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-613641947 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14 | MDEyOklzc3VlQ29tbWVudDYxMzY0MTk0Nw== | simonw 9599 | 2020-04-14T19:38:24Z | 2020-04-14T19:38:34Z | MEMBER | Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing events 530491074 | |
605382373 | https://github.com/dogsheep/swarm-to-sqlite/pull/6#issuecomment-605382373 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDYwNTM4MjM3Mw== | simonw 9599 | 2020-03-28T02:27:32Z | 2020-03-28T02:27:32Z | MEMBER | Thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | don't break if source is missing 543355051 | |
602309825 | https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602309825 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15 | MDEyOklzc3VlQ29tbWVudDYwMjMwOTgyNQ== | simonw 9599 | 2020-03-23T00:52:38Z | 2020-03-23T00:52:38Z | MEMBER | Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the `releases` command. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Assets table with downloads 544571092 | |
602311701 | https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602311701 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15 | MDEyOklzc3VlQ29tbWVudDYwMjMxMTcwMQ== | simonw 9599 | 2020-03-23T01:05:15Z | 2020-03-23T01:06:14Z | MEMBER | None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0 ``` github-to-sqlite releases releases.db instrumenta/conftest ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Assets table with downloads 544571092 | |
602312054 | https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602312054 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15 | MDEyOklzc3VlQ29tbWVudDYwMjMxMjA1NA== | simonw 9599 | 2020-03-23T01:07:10Z | 2020-03-23T01:07:10Z | MEMBER | Each asset looks like this: ```json { "url": "https://api.github.com/repos/instrumenta/conftest/releases/assets/11811946", "id": 11811946, "node_id": "MDEyOlJlbGVhc2VBc3NldDExODExOTQ2", "name": "checksums.txt", "label": "", "uploader": { "login": "garethr", "id": 2029, "node_id": "MDQ6VXNlcjIwMjk=", "avatar_url": "https://avatars2.githubusercontent.com/u/2029?v=4", "gravatar_id": "", "url": "https://api.github.com/users/garethr", "html_url": "https://github.com/garethr", "followers_url": "https://api.github.com/users/garethr/followers", "following_url": "https://api.github.com/users/garethr/following{/other_user}", "gists_url": "https://api.github.com/users/garethr/gists{/gist_id}", "starred_url": "https://api.github.com/users/garethr/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/garethr/subscriptions", "organizations_url": "https://api.github.com/users/garethr/orgs", "repos_url": "https://api.github.com/users/garethr/repos", "events_url": "https://api.github.com/users/garethr/events{/privacy}", "received_events_url": "https://api.github.com/users/garethr/received_events", "type": "User", "site_admin": false }, "content_type": "text/plain; charset=utf-8", "state": "uploaded", "size": 600, "download_count": 2, "created_at": "2019-03-30T16:56:44Z", "updated_at": "2019-03-30T16:56:44Z", "browser_download_url": "https://github.com/instrumenta/conftest/releases/download/v0.1.0/checksums.txt" } ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Assets table with downloads 544571092 | |
601979709 | https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-601979709 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16 | MDEyOklzc3VlQ29tbWVudDYwMTk3OTcwOQ== | simonw 9599 | 2020-03-21T02:20:08Z | 2020-03-21T02:20:08Z | MEMBER | I'm pretty sure this means I need to upgrade the underlying `sqlite-utils` dependency. I saw the same bug in `twitter-to-sqlite`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Exception running first command: IndexError: list index out of range 546051181 | |
613611455 | https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-613611455 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16 | MDEyOklzc3VlQ29tbWVudDYxMzYxMTQ1NQ== | simonw 9599 | 2020-04-14T18:37:21Z | 2020-04-14T18:37:21Z | MEMBER | This should have been fixed by #20 and #23 @jayvdb I'm definitely interested in this tool working as a library - it's purely designed as a CLI tool at the moment, but cleaning it up to work better as a dependency is totally in-scope for the project. https://sqlite-utils.readthedocs.io/ is an example of a tool I've built that works for both. Feel free to open a new issue here with some notes on what you would need for this to work as a library for your project! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Exception running first command: IndexError: list index out of range 546051181 | |
601871930 | https://github.com/dogsheep/twitter-to-sqlite/pull/33#issuecomment-601871930 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/33 | MDEyOklzc3VlQ29tbWVudDYwMTg3MTkzMA== | simonw 9599 | 2020-03-20T19:21:41Z | 2020-03-20T19:21:41Z | MEMBER | Made obsolete by https://github.com/dogsheep/twitter-to-sqlite/commit/d3887d1535c3edd59d3ebf6aea4495bacd82e59b | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Upgrade to sqlite-utils 2.2.1 561469252 | |
597354514 | https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597354514 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17 | MDEyOklzc3VlQ29tbWVudDU5NzM1NDUxNA== | simonw 9599 | 2020-03-10T22:37:45Z | 2020-03-10T22:37:45Z | MEMBER | I should add an option to stop the moment you see a commit you have fetched before. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing commits 578883725 | |
597358364 | https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597358364 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17 | MDEyOklzc3VlQ29tbWVudDU5NzM1ODM2NA== | simonw 9599 | 2020-03-10T22:50:20Z | 2020-03-11T01:18:36Z | MEMBER | By default it will stop when it sees a commit that has already been stored. You will be able to over-ride that behaviour using `--all`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing commits 578883725 | |
601861908 | https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601861908 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34 | MDEyOklzc3VlQ29tbWVudDYwMTg2MTkwOA== | simonw 9599 | 2020-03-20T18:56:44Z | 2020-03-20T18:56:44Z | MEMBER | Could this be a bug in `sqlite-utils`? This table has a primary key, so why is it running a query on `rowid = ?`? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | IndexError running user-timeline command 585266763 | |
601871579 | https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601871579 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34 | MDEyOklzc3VlQ29tbWVudDYwMTg3MTU3OQ== | simonw 9599 | 2020-03-20T19:20:52Z | 2020-03-20T19:20:52Z | MEMBER | Cause was the redefining of how `upsert()` works in sqlite-utils 2.x. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | IndexError running user-timeline command 585266763 | |
601875405 | https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875405 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYwMTg3NTQwNQ== | simonw 9599 | 2020-03-20T19:30:09Z | 2020-03-20T19:32:33Z | MEMBER | I'm going to redesign the command a little. Right now it defaults to fetching the authenticated user's timeline and allows `--screen_name` to be used to fetch for one other user instead. I think the design of `users-lookup` is much better. I'm going to copy that: ``` twitter-to-sqlite user-timeline users.db simonw cleopaws twitter-to-sqlite user-timeline users.db 12497 3166449535 --ids ``` It will also accept `--since` and `--sql` and `--alter` and `--since_id`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | twitter-to-sqlite user-timeline [screen_names] --sql / --attach 585282212 | |
601875524 | https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875524 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYwMTg3NTUyNA== | simonw 9599 | 2020-03-20T19:30:27Z | 2020-03-20T19:30:27Z | MEMBER | I can give it a snazzier progress bar to, as requested by #10. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | twitter-to-sqlite user-timeline [screen_names] --sql / --attach 585282212 | |
601876051 | https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601876051 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYwMTg3NjA1MQ== | simonw 9599 | 2020-03-20T19:31:43Z | 2020-03-20T19:31:43Z | MEMBER | I'm going to keep `--screen_name` working for backwards compatibility. Using `hidden=True` on it should hide it from the help output. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | twitter-to-sqlite user-timeline [screen_names] --sql / --attach 585282212 | |
601890795 | https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601890795 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYwMTg5MDc5NQ== | simonw 9599 | 2020-03-20T20:13:08Z | 2020-03-20T20:13:08Z | MEMBER | Needs documentation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | twitter-to-sqlite user-timeline [screen_names] --sql / --attach 585282212 | |
601892338 | https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601892338 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYwMTg5MjMzOA== | simonw 9599 | 2020-03-20T20:17:00Z | 2020-03-20T20:17:00Z | MEMBER | Released as 0.18. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | twitter-to-sqlite user-timeline [screen_names] --sql / --attach 585282212 | |
615509578 | https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509578 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37 | MDEyOklzc3VlQ29tbWVudDYxNTUwOTU3OA== | simonw 9599 | 2020-04-17T23:36:00Z | 2020-04-17T23:36:00Z | MEMBER | ``` $ twitter-to-sqlite user-timeline doggo.db doggoenthuonetuh Traceback (most recent call last): ... File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py", line 272, in transform_user user["created_at"] = parser.parse(user["created_at"]) KeyError: 'created_at' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Handle "User not found" error 585353598 | |
615509803 | https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509803 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37 | MDEyOklzc3VlQ29tbWVudDYxNTUwOTgwMw== | simonw 9599 | 2020-04-17T23:36:40Z | 2020-04-17T23:36:40Z | MEMBER | I'm going to print a warning to stderr, skip and continue - because if you have 100 screen names and only one of them is invalid you should still execute for the other 99. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Handle "User not found" error 585353598 | |
615510361 | https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615510361 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37 | MDEyOklzc3VlQ29tbWVudDYxNTUxMDM2MQ== | simonw 9599 | 2020-04-17T23:38:27Z | 2020-04-17T23:38:27Z | MEMBER | That's a bit tricky since I'd have to rewrite the internals of a bunch of other commands. For the moment I'll exit the script with an error but at least it will be a decent error! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Handle "User not found" error 585353598 | |
601939656 | https://github.com/dogsheep/twitter-to-sqlite/issues/38#issuecomment-601939656 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/38 | MDEyOklzc3VlQ29tbWVudDYwMTkzOTY1Ng== | simonw 9599 | 2020-03-20T22:37:04Z | 2020-03-20T22:37:04Z | MEMBER | New formatting: ``` @Alaska_DHSS [####################################] 3205 @BWhippleKS [####################################] 1 @CAPublicHealth [####################################] 2 @CDPHE [####################################] 67 @CHFSKy [####################################] 3216 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Screen name display for user-timeline is uneven 585359363 | |
601980724 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980724 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMTk4MDcyNA== | simonw 9599 | 2020-03-21T02:28:05Z | 2020-03-21T02:28:05Z | MEMBER | This suggests that `commit["author"]` can be `None` in some cases? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
601980957 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980957 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMTk4MDk1Nw== | simonw 9599 | 2020-03-21T02:30:03Z | 2020-03-21T02:30:03Z | MEMBER | https://github.community/t5/GitHub-API-Development-and/Request-for-commits-quot-author-quot-null-and-quot-committer/m-p/35842/highlight/true#M3372 > Commits aren't always associated with a GitHub user. For example, perhaps a friend of mine and I were working on a project together. I have a GitHub account and my friend doesn't. If we both add commits to the repository using our own email addresses and names and then I push the repository to GitHub, my commits will be associated with my GitHub user account but my friends' commits will show up with `author` and `committer` as `null`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602807178 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602807178 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMjgwNzE3OA== | simonw 9599 | 2020-03-23T19:24:43Z | 2020-03-23T19:24:43Z | MEMBER | I need to find an example before I work on this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602811468 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811468 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMjgxMTQ2OA== | simonw 9599 | 2020-03-23T19:33:14Z | 2020-03-23T19:33:14Z | MEMBER | Found one: https://api.github.com/repos/simonw/simonw.github.com/commits | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602811785 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811785 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMjgxMTc4NQ== | simonw 9599 | 2020-03-23T19:33:56Z | 2020-03-23T19:34:06Z | MEMBER | ```json [ { "sha": "a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "node_id": "MDY6Q29tbWl0OTMyMDk6YThkYzkxNDA4OWQzOTlkOWI1MjJlYmI1MWI2N2Y5YWMyZThhYTZiMA==", "commit": { "author": { "name": "Simon Willison", "email": "simon@...", "date": "2008-12-18T23:17:12Z" }, "committer": { "name": "Simon Willison", "email": "simon@...", "date": "2008-12-18T23:17:12Z" }, "message": "First commit", "tree": { "sha": "ac2dfb75e2592c59165c2880f3f7a16dafd452a1", "url": "https://api.github.com/repos/simonw/simonw.github.com/git/trees/ac2dfb75e2592c59165c2880f3f7a16dafd452a1" }, "url": "https://api.github.com/repos/simonw/simonw.github.com/git/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "comment_count": 0, "verification": { "verified": false, "reason": "unsigned", "signature": null, "payload": null } }, "url": "https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "html_url": "https://github.com/simonw/simonw.github.com/commit/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "comments_url": "https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0/comments", "author": null, "committer": null, "parents": [] } ] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602813731 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602813731 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMjgxMzczMQ== | simonw 9599 | 2020-03-23T19:38:03Z | 2020-03-23T20:44:49Z | MEMBER | So it turns out `"author"` and `"committer"` on the commit are `null` if the email address in the nested `"commit"` doesn't match an existing GitHub user. Maybe I should be storing the nested data somewhere as well? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602815120 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602815120 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMjgxNTEyMA== | simonw 9599 | 2020-03-23T19:40:55Z | 2020-03-23T19:43:19Z | MEMBER | I could pull a pk-hashed version of the name/email into separate `raw_author` and `raw_committer` columns perhaps - against a `commit_authors` table. Could be interesting. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602846293 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602846293 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMjg0NjI5Mw== | simonw 9599 | 2020-03-23T20:44:40Z | 2020-03-23T20:44:40Z | MEMBER | I implemented the `raw_authors` idea. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602094386 | https://github.com/dogsheep/hacker-news-to-sqlite/issues/1#issuecomment-602094386 | https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDYwMjA5NDM4Ng== | simonw 9599 | 2020-03-21T19:47:46Z | 2020-03-21T19:47:46Z | MEMBER | Released as 0.3. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up full text search 585526292 | |
602304531 | https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602304531 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19 | MDEyOklzc3VlQ29tbWVudDYwMjMwNDUzMQ== | simonw 9599 | 2020-03-23T00:20:28Z | 2020-03-23T00:20:28Z | MEMBER | Current FTS code: https://github.com/dogsheep/github-to-sqlite/blob/e35eec4343aa560c58c1634cc228d0d46c442304/github_to_sqlite/utils.py#L136-L143 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Enable full-text search for more stuff (like commits, issues and issue_comments) 585850715 | |
602782437 | https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602782437 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19 | MDEyOklzc3VlQ29tbWVudDYwMjc4MjQzNw== | simonw 9599 | 2020-03-23T18:36:12Z | 2020-03-23T18:51:23Z | MEMBER | * `commits`: `message` * `issue_comments`: `body` * `issues`: `title`, `body` * `labels`: `name`, `description` * `licenses`: `name` * `milestones`: `title`, `description` * `releases`: `name`, `body` * `repos`: `name`, `description` * `users`: `login`, `name` (maybe `company`, `bio`) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Enable full-text search for more stuff (like commits, issues and issue_comments) 585850715 | |
602896434 | https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602896434 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21 | MDEyOklzc3VlQ29tbWVudDYwMjg5NjQzNA== | simonw 9599 | 2020-03-23T22:43:37Z | 2020-03-23T22:43:37Z | MEMBER | I'm going to do this now to help figure out the latest error in #13: ``` Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite", line 11, in <module> load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')() File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 237, in commits utils.save_commits(db, commits, repo_full["id"]) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 345, in save_commits for commit in commits: File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 207, in fetch_commits if stop_when(commit): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 224, in stop_when db["commits"].get(commit["sha"]) TypeError: string indices must be integers ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Turn GitHub API errors into exceptions 586561727 | |
602920163 | https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602920163 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21 | MDEyOklzc3VlQ29tbWVudDYwMjkyMDE2Mw== | simonw 9599 | 2020-03-23T23:48:22Z | 2020-03-23T23:48:22Z | MEMBER | I'm happy with this pattern: https://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Turn GitHub API errors into exceptions 586561727 | |
602928533 | https://github.com/dogsheep/github-to-sqlite/issues/23#issuecomment-602928533 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23 | MDEyOklzc3VlQ29tbWVudDYwMjkyODUzMw== | simonw 9599 | 2020-03-24T00:15:49Z | 2020-03-24T00:15:49Z | MEMBER | https://github.com/dogsheep/github-to-sqlite/releases/tag/1.0 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Release 1.0 586595839 | |
606304837 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606304837 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjMwNDgzNw== | simonw 9599 | 2020-03-30T23:27:50Z | 2020-03-30T23:29:31Z | MEMBER | One option would be something like this: ```sql select max(id) from tweets where user = ? and not exists (select id from tweets where retweeted_status = id) and not exists (select id from tweets where quoted_status = id) and not exists (select id from tweets where in_reply_to_status_id = id) ``` Might be a good idea to index those columns (after confirming that doing so would indeed speed up the query). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --since feature can be confused by retweets 590666760 | |
606305701 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606305701 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjMwNTcwMQ== | simonw 9599 | 2020-03-30T23:30:27Z | 2020-03-30T23:30:27Z | MEMBER | A better alternative would be to maintain a separate table with the last seen since value for when we ran `user-timeline` for any specific user. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --since feature can be confused by retweets 590666760 | |
606309165 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606309165 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjMwOTE2NQ== | simonw 9599 | 2020-03-30T23:41:31Z | 2020-03-30T23:41:31Z | MEMBER | I like the separate `user_timeline_since` table solution. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --since feature can be confused by retweets 590666760 | |
606824992 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606824992 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjgyNDk5Mg== | simonw 9599 | 2020-03-31T19:24:23Z | 2020-03-31T19:24:23Z | MEMBER | The `--since` option is actually used by four commands: * `user-timeline` * `home-timeline` * `mentions-timeline` * `search` All of them use the same `fetch_timeline()` utility function under the hood. I should move the logic that looks up the last `since_id` into that shared function. Question: should I have a table for each of those four methods or a single table that is used by them all? I'm leaning towards four separate tables. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --since feature can be confused by retweets 590666760 | |
606843224 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606843224 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjg0MzIyNA== | simonw 9599 | 2020-03-31T19:59:11Z | 2020-03-31T20:06:32Z | MEMBER | Or... have a single `since_ids` table to track since values, and have its primary key be a string that looks something like this: `user:123145` `home:23441` `mentions:23425` `search:99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d` That last example would use the hash generated here: https://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/cli.py#L792-L808 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --since feature can be confused by retweets 590666760 | |
606844521 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606844521 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjg0NDUyMQ== | simonw 9599 | 2020-03-31T20:01:39Z | 2020-03-31T20:01:39Z | MEMBER | I think `utils.fetch_timeline()` grows a new argument, `since_key`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --since feature can be confused by retweets 590666760 | |
606850008 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850008 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjg1MDAwOA== | simonw 9599 | 2020-03-31T20:13:59Z | 2020-04-01T00:23:00Z | MEMBER | Table design for `since_ids` table: type | key | since_id --- | --- | --- 1 | 124324 | 2347239847293 2 | 99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d | 2125947927344 Primary compound key on `(category, key)` `type` is also a foreign key to a `since_id_types` table with `id` and `name` columns (probably created using https://sqlite-utils.readthedocs.io/en/stable/python-api.html#working-with-lookup-tables ) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --since feature can be confused by retweets 590666760 | |
606850453 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850453 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjg1MDQ1Mw== | simonw 9599 | 2020-03-31T20:14:58Z | 2020-04-01T03:03:50Z | MEMBER | Actually I'll hard-code the population of `since_id_types` to get known ID constants. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --since feature can be confused by retweets 590666760 | |
606998669 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606998669 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjk5ODY2OQ== | simonw 9599 | 2020-04-01T02:57:36Z | 2020-04-01T02:57:36Z | MEMBER | The tricky thing here is thinking about the interaction between the recorded since_id and a desire to run the initial import. The first time you run `twitter-to-sqlite user-timeline db.db username` we want to fetch as many tweets from that user as possible - probably around 3,200 before the API limitations cut us off. We need to record the maximum ID from those as the `since_id` - which we will see on the very first page we paginate through. That way next time we run the command with `--since` we will only fetch new tweets. But what happens if our initial import is cancelled after only a few tweets? We risk never pulling in the rest of the tweets. Not sure if I need to solve this at all or if I should instead trust users to run the command a second time without `--since` if they think they didn't retrieve anything the first time. I had considered letting `--stop_after=` over-ride `--since` but that doesn't actually make sense - if you send a since_id to the Twitter API you'll never get back more tweets than exist after that ID, so the `--stop_after` would not make a meaningful difference. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --since feature can be confused by retweets 590666760 | |
607003655 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607003655 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNzAwMzY1NQ== | simonw 9599 | 2020-04-01T03:18:00Z | 2020-04-01T03:18:00Z | MEMBER | I've got this working for the `user-timeline` command. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --since feature can be confused by retweets 590666760 | |
607010634 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607010634 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNzAxMDYzNA== | simonw 9599 | 2020-04-01T03:45:16Z | 2020-04-01T03:45:16Z | MEMBER | OK, fix is applied to everything now. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --since feature can be confused by retweets 590666760 | |
606307019 | https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307019 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDYwNjMwNzAxOQ== | simonw 9599 | 2020-03-30T23:34:27Z | 2020-03-30T23:34:27Z | MEMBER | The count properties available for a user are: * followers_count * friends_count * listed_count * favourites_count * statuses_count May as well track history for all of them? Should be pretty cheap to store. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: record history of follower counts 590669793 | |
606307376 | https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307376 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDYwNjMwNzM3Ng== | simonw 9599 | 2020-03-30T23:35:40Z | 2020-03-30T23:39:15Z | MEMBER | I think five separate tables: * followers_count_history * friends_count_history * listed_count_history * favourites_count_history * statuses_count_history Each with the following structure: * datetime (ISO UTC) * user (ID, foreign key to users) * count (integer) I'm tempted to have a compound primary key here - user, datetime | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: record history of follower counts 590669793 | |
607011421 | https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011421 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDYwNzAxMTQyMQ== | simonw 9599 | 2020-04-01T03:47:37Z | 2020-04-01T03:55:08Z | MEMBER | Actually a single table with a `type` integer ID referencing a `count_history_types` table would better match the way I implemented the `since_ids` table: https://github.com/dogsheep/twitter-to-sqlite/blob/4b6c8d8c1cc6fefdb566ec8506157133f47c569a/twitter_to_sqlite/utils.py#L331-L341 In which case the compound primary key would be `type`, `user`, `datetime` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: record history of follower counts 590669793 | |
607011972 | https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011972 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDYwNzAxMTk3Mg== | simonw 9599 | 2020-04-01T03:49:02Z | 2020-04-01T03:50:01Z | MEMBER | I want the datetime value to look like `2020-04-01T03:34:58+00:00` (the format returned by the Twitter API which I am storing in other tables at the moment). ``` >>> datetime.utcnow().isoformat().split('.')[0] + '+00:00' '2020-04-01T03:49:52+00:00' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: record history of follower counts 590669793 | |
607019151 | https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607019151 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDYwNzAxOTE1MQ== | simonw 9599 | 2020-04-01T04:11:10Z | 2020-04-01T04:11:10Z | MEMBER | In testing this collects a LOT of data. I'm going to skip tracking favourites_count and statuses_count and just track followers, friends and listed instead. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: record history of follower counts 590669793 | |
614810417 | https://github.com/dogsheep/github-to-sqlite/issues/25#issuecomment-614810417 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25 | MDEyOklzc3VlQ29tbWVudDYxNDgxMDQxNw== | simonw 9599 | 2020-04-16T18:07:11Z | 2020-04-16T18:07:11Z | MEMBER | Turns out the main problem was #26 - now fixed. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Improvements to demo instance 601265023 | |
614794739 | https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614794739 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26 | MDEyOklzc3VlQ29tbWVudDYxNDc5NDczOQ== | simonw 9599 | 2020-04-16T17:38:28Z | 2020-04-16T17:38:28Z | MEMBER | I'm already doing this here: https://github.com/dogsheep/github-to-sqlite/blob/c4aaa50e167cfa9021c7c94260bc3e89e10947bf/github_to_sqlite/utils.py#L246-L250 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Topics are missing from repositories 601271612 | |
614795712 | https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614795712 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26 | MDEyOklzc3VlQ29tbWVudDYxNDc5NTcxMg== | simonw 9599 | 2020-04-16T17:40:27Z | 2020-04-16T17:40:27Z | MEMBER | Aha! it was missing from the `fetch_repo()` function. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Topics are missing from repositories 601271612 | |
614831451 | https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831451 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDYxNDgzMTQ1MQ== | simonw 9599 | 2020-04-16T18:47:25Z | 2020-04-16T18:47:25Z | MEMBER | Is it possible for a repo to have an `owner` that differs from its `organization`? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Repos have a big blob of JSON in the organization column 601330277 | |
614831842 | https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831842 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDYxNDgzMTg0Mg== | simonw 9599 | 2020-04-16T18:48:18Z | 2020-04-16T18:48:18Z | MEMBER | I'm going to make `organization` another foreign key to the `users` table just in case it IS possible (maybe with GitHub Enterprise or similar?) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Repos have a big blob of JSON in the organization column 601330277 | |
614843406 | https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDYxNDg0MzQwNg== | simonw 9599 | 2020-04-16T19:11:53Z | 2020-04-16T19:20:23Z | MEMBER | This didn't quite work: the column type is incorrect, so the foreign key relationship isn't sticking: https://github-to-sqlite.dogsheep.net/github/repos?organization=53015001 `[organization] TEXT REFERENCES [users]([id])` - should be `INTEGER`. The problem is that if the first repo inserted has no organization it's set to `null`, which `sqlite-utils` derives as a `TEXT` column. One solution would be to create the column explicitly with a type, but this could get messy. I think I want a new sqlite-utils feature for this instead. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Repos have a big blob of JSON in the organization column 601330277 | |
615518606 | https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615518606 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDYxNTUxODYwNg== | simonw 9599 | 2020-04-18T00:14:32Z | 2020-04-18T00:14:32Z | MEMBER | https://github.com/simonw/sqlite-utils/issues/100 is done and released in sqlite-utils 2.7. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Repos have a big blob of JSON in the organization column 601330277 | |
615519409 | https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615519409 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDYxNTUxOTQwOQ== | simonw 9599 | 2020-04-18T00:19:16Z | 2020-04-18T00:19:16Z | MEMBER | ``` $ github-to-sqlite repos b.db dogsheep $ sqlite3 b.db '.schema repos' CREATE TABLE [repos] ( [id] INTEGER PRIMARY KEY, ... [permissions] TEXT, [organization] INTEGER REFERENCES [users]([id]), FOREIGN KEY(license) REFERENCES licenses(key) ); ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Repos have a big blob of JSON in the organization column 601330277 | |
615883040 | https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883040 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28 | MDEyOklzc3VlQ29tbWVudDYxNTg4MzA0MA== | simonw 9599 | 2020-04-18T14:45:38Z | 2020-04-18T14:45:38Z | MEMBER | ``` File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 219, in contributors utils.save_contributors(db, contributors, repo_full["id"]) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 354, in save_contributors for contributor in contributors: File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 228, in fetch_contributors for contributors in paginate(url, headers): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 286, in paginate data = response.json() File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/requests/models.py", line 898, in json return complexjson.loads(self.text, **kwargs) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/__init__.py", line 357, in loads return _default_decoder.decode(s) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from N… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pull repository contributors 601333634 | |
615883687 | https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883687 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28 | MDEyOklzc3VlQ29tbWVudDYxNTg4MzY4Nw== | simonw 9599 | 2020-04-18T14:49:58Z | 2020-04-18T14:49:58Z | MEMBER | That happened trying to pull contributors for `dogsheep/beta` - an empty repository. Turns out it was returning a `204 no content`: ``` ~ $ curl -i 'https://api.github.com/repos/dogsheep/beta/contributors' HTTP/1.1 204 No Content ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pull repository contributors 601333634 | |
615886206 | https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615886206 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28 | MDEyOklzc3VlQ29tbWVudDYxNTg4NjIwNg== | simonw 9599 | 2020-04-18T15:04:59Z | 2020-04-18T15:04:59Z | MEMBER | Demo: https://github-to-sqlite.dogsheep.net/github/contributors Documentation: https://github.com/dogsheep/github-to-sqlite/blob/13f8868fb5efa01c263b24f6dd91c617e6e938e1/README.md#fetching-contributors-to-a-repository | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pull repository contributors 601333634 | |
615513491 | https://github.com/dogsheep/twitter-to-sqlite/issues/43#issuecomment-615513491 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDYxNTUxMzQ5MQ== | simonw 9599 | 2020-04-17T23:48:28Z | 2020-04-17T23:48:28Z | MEMBER | Released in 0.21. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | "twitter-to-sqlite lists" command for retrieving a user's owned lists 602176870 | |
623193947 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623193947 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5Mzk0Nw== | simonw 9599 | 2020-05-03T22:36:17Z | 2020-05-03T22:36:17Z | MEMBER | I'm going to use [osxphotos](https://github.com/RhetTbull/osxphotos) for this. Since I've already got code to upload photos and insert them into a table based on their `sha256` hash, my first go at this will be to import data using the tool and foreign-key it to the `sha256` hash in the existing table. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623195197 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623195197 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5NTE5Nw== | simonw 9599 | 2020-05-03T22:44:33Z | 2020-05-03T22:44:33Z | MEMBER | Command will be this: $ photos-to-sqlite apple-photos photos.db This will populate a `apple_photos` table with the data imported by the `osxphotos` library, plus the calculated sha256. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623198653 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198653 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5ODY1Mw== | simonw 9599 | 2020-05-03T23:09:57Z | 2020-05-03T23:09:57Z | MEMBER | For locations: I'll add `place_x` columns for all of these: ``` (Pdb) photo.place.address._asdict() {'street': None, 'sub_locality': None, 'city': 'Loreto', 'sub_administrative_area': 'Loreto', 'state_province': 'BCS', 'postal_code': None, 'country': 'Mexico', 'iso_country_code': 'MX'} ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623198986 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198986 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5ODk4Ng== | simonw 9599 | 2020-05-03T23:12:31Z | 2020-05-03T23:12:46Z | MEMBER | To get the taken date in UTC: ``` from datetime import timezone (Pdb) photo.date.astimezone(timezone.utc).isoformat() '2018-02-13T20:21:31.620000+00:00' (Pdb) photo.date.astimezone(timezone.utc).isoformat().split(".") ['2018-02-13T20:21:31', '620000+00:00'] (Pdb) photo.date.astimezone(timezone.utc).isoformat().split(".")[0] '2018-02-13T20:21:31' (Pdb) photo.date.astimezone(timezone.utc).isoformat().split(".")[0] + "+00:00" '2018-02-13T20:21:31+00:00' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623199214 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199214 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5OTIxNA== | simonw 9599 | 2020-05-03T23:14:08Z | 2020-05-03T23:14:08Z | MEMBER | Albums have UUIDs: ``` (Pdb) photo.album_info[0].__dict__ {'_uuid': '17816791-ABF3-447B-942C-9FA8065EEBBA', '_db': osxphotos.PhotosDB(dbfile='/Users/simon/Pictures/Photos Library.photoslibrary/database/photos.db'), '_title': 'Geotaggable Photos geotagged'} ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623199701 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199701 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5OTcwMQ== | simonw 9599 | 2020-05-03T23:17:38Z | 2020-05-03T23:17:38Z | MEMBER | Record burst_uuid as a column: ``` (Pdb) with_bursts[0]._info["burstUUID"] '703FAA23-57BF-40B4-8A33-D9CEB143391B' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623199750 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199750 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzE5OTc1MA== | simonw 9599 | 2020-05-03T23:17:58Z | 2020-05-03T23:17:58Z | MEMBER | Reading this source code is really useful for figuring out how to store a photo in a DB table: https://github.com/RhetTbull/osxphotos/blob/7444b6d173918a3ad2a07aefce5ecf054786c787/osxphotos/photoinfo.py | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
623232984 | https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623232984 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 | MDEyOklzc3VlQ29tbWVudDYyMzIzMjk4NA== | simonw 9599 | 2020-05-04T02:41:32Z | 2020-05-04T02:41:32Z | MEMBER | Needs documentation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import photo metadata from Apple Photos into SQLite 602533300 | |
615931488 | https://github.com/dogsheep/dogsheep-photos/issues/2#issuecomment-615931488 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/2 | MDEyOklzc3VlQ29tbWVudDYxNTkzMTQ4OA== | simonw 9599 | 2020-04-18T19:24:02Z | 2020-04-18T19:24:02Z | MEMBER | I made a start on this last week with a https://github.com/simonw/heic-to-jpeg proxy. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to convert HEIC images to JPEG 602533352 | |
615932007 | https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932007 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4 | MDEyOklzc3VlQ29tbWVudDYxNTkzMjAwNw== | simonw 9599 | 2020-04-18T19:27:55Z | 2020-04-18T19:27:55Z | MEMBER | Research thread: https://twitter.com/simonw/status/1249049694984011776 > I want to build some software that lets people store their own data in their own S3 bucket, but if possible I'd like not to have to teach people the incantations needed to get their bucket setup and minimum-permission credentials figures out https://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/ looks useful | {"total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Upload all my photos to a secure S3 bucket 602533539 | |
615932204 | https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932204 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4 | MDEyOklzc3VlQ29tbWVudDYxNTkzMjIwNA== | simonw 9599 | 2020-04-18T19:29:22Z | 2020-04-18T19:34:44Z | MEMBER | I'm going to call my bucket `dogsheep-photos-simon`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Upload all my photos to a secure S3 bucket 602533539 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
updated_at (date) >30 ✖