github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | 622461223 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw== | 9599 | 2020-05-01T16:34:52Z | 2020-05-01T16:34:52Z | MEMBER | Blocked on #37 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 520756546 | |
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | 622989874 | MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA== | 9599 | 2020-05-02T17:46:14Z | 2020-05-02T17:46:14Z | MEMBER | Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 520756546 | |
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | 622998813 | MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw== | 9599 | 2020-05-02T18:58:17Z | 2020-05-02T18:58:17Z | MEMBER | Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 520756546 | |
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602861730 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | 602861730 | MDEyOklzc3VlQ29tbWVudDYwMjg2MTczMA== | 9599 | 2020-03-23T21:19:16Z | 2020-03-23T21:19:42Z | MEMBER | I'm going to do this with a scheduled GitHub Action in this repo. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 521275281 | |
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862236 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | 602862236 | MDEyOklzc3VlQ29tbWVudDYwMjg2MjIzNg== | 9599 | 2020-03-23T21:20:26Z | 2020-03-23T21:20:26Z | MEMBER | I'll run the `commits` and `issues` and `issue-comments` commands in addition to the `releases` command. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 521275281 | |
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862967 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | 602862967 | MDEyOklzc3VlQ29tbWVudDYwMjg2Mjk2Nw== | 9599 | 2020-03-23T21:22:04Z | 2020-03-23T21:22:04Z | MEMBER | Following these instructions: https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 521275281 | |
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602895896 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | 602895896 | MDEyOklzc3VlQ29tbWVudDYwMjg5NTg5Ng== | 9599 | 2020-03-23T22:42:25Z | 2020-03-23T22:42:25Z | MEMBER | Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 521275281 | |
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602916947 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | 602916947 | MDEyOklzc3VlQ29tbWVudDYwMjkxNjk0Nw== | 9599 | 2020-03-23T23:38:06Z | 2020-03-23T23:38:06Z | MEMBER | Woohoo! https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 521275281 | |
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602917713 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | 602917713 | MDEyOklzc3VlQ29tbWVudDYwMjkxNzcxMw== | 9599 | 2020-03-23T23:40:29Z | 2020-03-23T23:40:29Z | MEMBER | Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 521275281 | |
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602918689 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | 602918689 | MDEyOklzc3VlQ29tbWVudDYwMjkxODY4OQ== | 9599 | 2020-03-23T23:43:39Z | 2020-03-23T23:47:50Z | MEMBER | I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 521275281 | |
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602919058 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | 602919058 | MDEyOklzc3VlQ29tbWVudDYwMjkxOTA1OA== | 9599 | 2020-03-23T23:44:48Z | 2020-03-23T23:44:48Z | MEMBER | Next step: use a `metadata.json` file to add some extras. And add the `datasette-render-markdown` plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 521275281 | |
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602924714 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | 602924714 | MDEyOklzc3VlQ29tbWVudDYwMjkyNDcxNA== | 9599 | 2020-03-24T00:03:25Z | 2020-03-24T00:03:25Z | MEMBER | This is good enough for the 1.0 release. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 521275281 | |
https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559883311 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14 | 559883311 | MDEyOklzc3VlQ29tbWVudDU1OTg4MzMxMQ== | 9599 | 2019-11-29T21:30:37Z | 2019-11-29T21:30:37Z | MEMBER | I should build the command to persist ETags and obey their polling guidelines: > Events are optimized for polling with the "ETag" header. If no new events have been triggered, you will see a "304 Not Modified" response, and your current rate limit will be untouched. There is also an "X-Poll-Interval" header that specifies how often (in seconds) you are allowed to poll. In times of high server load, the time may increase. Please obey the header. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 530491074 | |
https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559902818 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14 | 559902818 | MDEyOklzc3VlQ29tbWVudDU1OTkwMjgxOA== | 9599 | 2019-11-30T01:32:38Z | 2019-11-30T01:32:38Z | MEMBER | Prototype: ``` pip install sqlite-utils paginate-json paginate-json "https://api.github.com/users/simonw/events" | sqlite-utils insert /tmp/events.db events - --pk=id ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 530491074 | |
https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-613641947 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14 | 613641947 | MDEyOklzc3VlQ29tbWVudDYxMzY0MTk0Nw== | 9599 | 2020-04-14T19:38:24Z | 2020-04-14T19:38:34Z | MEMBER | Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 530491074 | |
https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602309825 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15 | 602309825 | MDEyOklzc3VlQ29tbWVudDYwMjMwOTgyNQ== | 9599 | 2020-03-23T00:52:38Z | 2020-03-23T00:52:38Z | MEMBER | Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the `releases` command. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 544571092 | |
https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602311701 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15 | 602311701 | MDEyOklzc3VlQ29tbWVudDYwMjMxMTcwMQ== | 9599 | 2020-03-23T01:05:15Z | 2020-03-23T01:06:14Z | MEMBER | None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0 ``` github-to-sqlite releases releases.db instrumenta/conftest ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 544571092 | |
https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602312054 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15 | 602312054 | MDEyOklzc3VlQ29tbWVudDYwMjMxMjA1NA== | 9599 | 2020-03-23T01:07:10Z | 2020-03-23T01:07:10Z | MEMBER | Each asset looks like this: ```json { "url": "https://api.github.com/repos/instrumenta/conftest/releases/assets/11811946", "id": 11811946, "node_id": "MDEyOlJlbGVhc2VBc3NldDExODExOTQ2", "name": "checksums.txt", "label": "", "uploader": { "login": "garethr", "id": 2029, "node_id": "MDQ6VXNlcjIwMjk=", "avatar_url": "https://avatars2.githubusercontent.com/u/2029?v=4", "gravatar_id": "", "url": "https://api.github.com/users/garethr", "html_url": "https://github.com/garethr", "followers_url": "https://api.github.com/users/garethr/followers", "following_url": "https://api.github.com/users/garethr/following{/other_user}", "gists_url": "https://api.github.com/users/garethr/gists{/gist_id}", "starred_url": "https://api.github.com/users/garethr/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/garethr/subscriptions", "organizations_url": "https://api.github.com/users/garethr/orgs", "repos_url": "https://api.github.com/users/garethr/repos", "events_url": "https://api.github.com/users/garethr/events{/privacy}", "received_events_url": "https://api.github.com/users/garethr/received_events", "type": "User", "site_admin": false }, "content_type": "text/plain; charset=utf-8", "state": "uploaded", "size": 600, "download_count": 2, "created_at": "2019-03-30T16:56:44Z", "updated_at": "2019-03-30T16:56:44Z", "browser_download_url": "https://github.com/instrumenta/conftest/releases/download/v0.1.0/checksums.txt" } ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 544571092 | |
https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-601979709 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16 | 601979709 | MDEyOklzc3VlQ29tbWVudDYwMTk3OTcwOQ== | 9599 | 2020-03-21T02:20:08Z | 2020-03-21T02:20:08Z | MEMBER | I'm pretty sure this means I need to upgrade the underlying `sqlite-utils` dependency. I saw the same bug in `twitter-to-sqlite`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 546051181 | |
https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-613611455 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16 | 613611455 | MDEyOklzc3VlQ29tbWVudDYxMzYxMTQ1NQ== | 9599 | 2020-04-14T18:37:21Z | 2020-04-14T18:37:21Z | MEMBER | This should have been fixed by #20 and #23 @jayvdb I'm definitely interested in this tool working as a library - it's purely designed as a CLI tool at the moment, but cleaning it up to work better as a dependency is totally in-scope for the project. https://sqlite-utils.readthedocs.io/ is an example of a tool I've built that works for both. Feel free to open a new issue here with some notes on what you would need for this to work as a library for your project! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 546051181 | |
https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597354514 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17 | 597354514 | MDEyOklzc3VlQ29tbWVudDU5NzM1NDUxNA== | 9599 | 2020-03-10T22:37:45Z | 2020-03-10T22:37:45Z | MEMBER | I should add an option to stop the moment you see a commit you have fetched before. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 578883725 | |
https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597358364 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17 | 597358364 | MDEyOklzc3VlQ29tbWVudDU5NzM1ODM2NA== | 9599 | 2020-03-10T22:50:20Z | 2020-03-11T01:18:36Z | MEMBER | By default it will stop when it sees a commit that has already been stored. You will be able to over-ride that behaviour using `--all`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 578883725 | |
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980724 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | 601980724 | MDEyOklzc3VlQ29tbWVudDYwMTk4MDcyNA== | 9599 | 2020-03-21T02:28:05Z | 2020-03-21T02:28:05Z | MEMBER | This suggests that `commit["author"]` can be `None` in some cases? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 585411547 | |
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980957 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | 601980957 | MDEyOklzc3VlQ29tbWVudDYwMTk4MDk1Nw== | 9599 | 2020-03-21T02:30:03Z | 2020-03-21T02:30:03Z | MEMBER | https://github.community/t5/GitHub-API-Development-and/Request-for-commits-quot-author-quot-null-and-quot-committer/m-p/35842/highlight/true#M3372 > Commits aren't always associated with a GitHub user. For example, perhaps a friend of mine and I were working on a project together. I have a GitHub account and my friend doesn't. If we both add commits to the repository using our own email addresses and names and then I push the repository to GitHub, my commits will be associated with my GitHub user account but my friends' commits will show up with `author` and `committer` as `null`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 585411547 | |
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602807178 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | 602807178 | MDEyOklzc3VlQ29tbWVudDYwMjgwNzE3OA== | 9599 | 2020-03-23T19:24:43Z | 2020-03-23T19:24:43Z | MEMBER | I need to find an example before I work on this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 585411547 | |
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811468 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | 602811468 | MDEyOklzc3VlQ29tbWVudDYwMjgxMTQ2OA== | 9599 | 2020-03-23T19:33:14Z | 2020-03-23T19:33:14Z | MEMBER | Found one: https://api.github.com/repos/simonw/simonw.github.com/commits | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 585411547 | |
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811785 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | 602811785 | MDEyOklzc3VlQ29tbWVudDYwMjgxMTc4NQ== | 9599 | 2020-03-23T19:33:56Z | 2020-03-23T19:34:06Z | MEMBER | ```json [ { "sha": "a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "node_id": "MDY6Q29tbWl0OTMyMDk6YThkYzkxNDA4OWQzOTlkOWI1MjJlYmI1MWI2N2Y5YWMyZThhYTZiMA==", "commit": { "author": { "name": "Simon Willison", "email": "simon@...", "date": "2008-12-18T23:17:12Z" }, "committer": { "name": "Simon Willison", "email": "simon@...", "date": "2008-12-18T23:17:12Z" }, "message": "First commit", "tree": { "sha": "ac2dfb75e2592c59165c2880f3f7a16dafd452a1", "url": "https://api.github.com/repos/simonw/simonw.github.com/git/trees/ac2dfb75e2592c59165c2880f3f7a16dafd452a1" }, "url": "https://api.github.com/repos/simonw/simonw.github.com/git/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "comment_count": 0, "verification": { "verified": false, "reason": "unsigned", "signature": null, "payload": null } }, "url": "https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "html_url": "https://github.com/simonw/simonw.github.com/commit/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "comments_url": "https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0/comments", "author": null, "committer": null, "parents": [] } ] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 585411547 | |
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602813731 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | 602813731 | MDEyOklzc3VlQ29tbWVudDYwMjgxMzczMQ== | 9599 | 2020-03-23T19:38:03Z | 2020-03-23T20:44:49Z | MEMBER | So it turns out `"author"` and `"committer"` on the commit are `null` if the email address in the nested `"commit"` doesn't match an existing GitHub user. Maybe I should be storing the nested data somewhere as well? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 585411547 | |
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602815120 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | 602815120 | MDEyOklzc3VlQ29tbWVudDYwMjgxNTEyMA== | 9599 | 2020-03-23T19:40:55Z | 2020-03-23T19:43:19Z | MEMBER | I could pull a pk-hashed version of the name/email into separate `raw_author` and `raw_committer` columns perhaps - against a `commit_authors` table. Could be interesting. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 585411547 | |
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602846293 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | 602846293 | MDEyOklzc3VlQ29tbWVudDYwMjg0NjI5Mw== | 9599 | 2020-03-23T20:44:40Z | 2020-03-23T20:44:40Z | MEMBER | I implemented the `raw_authors` idea. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 585411547 | |
https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602304531 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19 | 602304531 | MDEyOklzc3VlQ29tbWVudDYwMjMwNDUzMQ== | 9599 | 2020-03-23T00:20:28Z | 2020-03-23T00:20:28Z | MEMBER | Current FTS code: https://github.com/dogsheep/github-to-sqlite/blob/e35eec4343aa560c58c1634cc228d0d46c442304/github_to_sqlite/utils.py#L136-L143 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 585850715 | |
https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602782437 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19 | 602782437 | MDEyOklzc3VlQ29tbWVudDYwMjc4MjQzNw== | 9599 | 2020-03-23T18:36:12Z | 2020-03-23T18:51:23Z | MEMBER | * `commits`: `message` * `issue_comments`: `body` * `issues`: `title`, `body` * `labels`: `name`, `description` * `licenses`: `name` * `milestones`: `title`, `description` * `releases`: `name`, `body` * `repos`: `name`, `description` * `users`: `login`, `name` (maybe `company`, `bio`) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 585850715 | |
https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602896434 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21 | 602896434 | MDEyOklzc3VlQ29tbWVudDYwMjg5NjQzNA== | 9599 | 2020-03-23T22:43:37Z | 2020-03-23T22:43:37Z | MEMBER | I'm going to do this now to help figure out the latest error in #13: ``` Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite", line 11, in <module> load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')() File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 237, in commits utils.save_commits(db, commits, repo_full["id"]) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 345, in save_commits for commit in commits: File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 207, in fetch_commits if stop_when(commit): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 224, in stop_when db["commits"].get(commit["sha"]) TypeError: string indices must be integers ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 586561727 | |
https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602920163 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21 | 602920163 | MDEyOklzc3VlQ29tbWVudDYwMjkyMDE2Mw== | 9599 | 2020-03-23T23:48:22Z | 2020-03-23T23:48:22Z | MEMBER | I'm happy with this pattern: https://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 586561727 | |
https://github.com/dogsheep/github-to-sqlite/issues/23#issuecomment-602928533 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23 | 602928533 | MDEyOklzc3VlQ29tbWVudDYwMjkyODUzMw== | 9599 | 2020-03-24T00:15:49Z | 2020-03-24T00:15:49Z | MEMBER | https://github.com/dogsheep/github-to-sqlite/releases/tag/1.0 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 586595839 | |
https://github.com/dogsheep/github-to-sqlite/issues/25#issuecomment-614810417 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25 | 614810417 | MDEyOklzc3VlQ29tbWVudDYxNDgxMDQxNw== | 9599 | 2020-04-16T18:07:11Z | 2020-04-16T18:07:11Z | MEMBER | Turns out the main problem was #26 - now fixed. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 601265023 | |
https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614794739 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26 | 614794739 | MDEyOklzc3VlQ29tbWVudDYxNDc5NDczOQ== | 9599 | 2020-04-16T17:38:28Z | 2020-04-16T17:38:28Z | MEMBER | I'm already doing this here: https://github.com/dogsheep/github-to-sqlite/blob/c4aaa50e167cfa9021c7c94260bc3e89e10947bf/github_to_sqlite/utils.py#L246-L250 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 601271612 | |
https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614795712 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26 | 614795712 | MDEyOklzc3VlQ29tbWVudDYxNDc5NTcxMg== | 9599 | 2020-04-16T17:40:27Z | 2020-04-16T17:40:27Z | MEMBER | Aha! it was missing from the `fetch_repo()` function. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 601271612 | |
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831451 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | 614831451 | MDEyOklzc3VlQ29tbWVudDYxNDgzMTQ1MQ== | 9599 | 2020-04-16T18:47:25Z | 2020-04-16T18:47:25Z | MEMBER | Is it possible for a repo to have an `owner` that differs from its `organization`? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 601330277 | |
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831842 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | 614831842 | MDEyOklzc3VlQ29tbWVudDYxNDgzMTg0Mg== | 9599 | 2020-04-16T18:48:18Z | 2020-04-16T18:48:18Z | MEMBER | I'm going to make `organization` another foreign key to the `users` table just in case it IS possible (maybe with GitHub Enterprise or similar?) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 601330277 | |
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | 614843406 | MDEyOklzc3VlQ29tbWVudDYxNDg0MzQwNg== | 9599 | 2020-04-16T19:11:53Z | 2020-04-16T19:20:23Z | MEMBER | This didn't quite work: the column type is incorrect, so the foreign key relationship isn't sticking: https://github-to-sqlite.dogsheep.net/github/repos?organization=53015001 `[organization] TEXT REFERENCES [users]([id])` - should be `INTEGER`. The problem is that if the first repo inserted has no organization it's set to `null`, which `sqlite-utils` derives as a `TEXT` column. One solution would be to create the column explicitly with a type, but this could get messy. I think I want a new sqlite-utils feature for this instead. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 601330277 | |
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615518606 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | 615518606 | MDEyOklzc3VlQ29tbWVudDYxNTUxODYwNg== | 9599 | 2020-04-18T00:14:32Z | 2020-04-18T00:14:32Z | MEMBER | https://github.com/simonw/sqlite-utils/issues/100 is done and released in sqlite-utils 2.7. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 601330277 | |
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615519409 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | 615519409 | MDEyOklzc3VlQ29tbWVudDYxNTUxOTQwOQ== | 9599 | 2020-04-18T00:19:16Z | 2020-04-18T00:19:16Z | MEMBER | ``` $ github-to-sqlite repos b.db dogsheep $ sqlite3 b.db '.schema repos' CREATE TABLE [repos] ( [id] INTEGER PRIMARY KEY, ... [permissions] TEXT, [organization] INTEGER REFERENCES [users]([id]), FOREIGN KEY(license) REFERENCES licenses(key) ); ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 601330277 | |
https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883040 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28 | 615883040 | MDEyOklzc3VlQ29tbWVudDYxNTg4MzA0MA== | 9599 | 2020-04-18T14:45:38Z | 2020-04-18T14:45:38Z | MEMBER | ``` File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 219, in contributors utils.save_contributors(db, contributors, repo_full["id"]) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 354, in save_contributors for contributor in contributors: File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 228, in fetch_contributors for contributors in paginate(url, headers): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 286, in paginate data = response.json() File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/requests/models.py", line 898, in json return complexjson.loads(self.text, **kwargs) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/__init__.py", line 357, in loads return _default_decoder.decode(s) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from N… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 601333634 | |
https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883687 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28 | 615883687 | MDEyOklzc3VlQ29tbWVudDYxNTg4MzY4Nw== | 9599 | 2020-04-18T14:49:58Z | 2020-04-18T14:49:58Z | MEMBER | That happened trying to pull contributors for `dogsheep/beta` - an empty repository. Turns out it was returning a `204 no content`: ``` ~ $ curl -i 'https://api.github.com/repos/dogsheep/beta/contributors' HTTP/1.1 204 No Content ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 601333634 | |
https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615886206 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28 | 615886206 | MDEyOklzc3VlQ29tbWVudDYxNTg4NjIwNg== | 9599 | 2020-04-18T15:04:59Z | 2020-04-18T15:04:59Z | MEMBER | Demo: https://github-to-sqlite.dogsheep.net/github/contributors Documentation: https://github.com/dogsheep/github-to-sqlite/blob/13f8868fb5efa01c263b24f6dd91c617e6e938e1/README.md#fetching-contributors-to-a-repository | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 601333634 | |
https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29 | 616883275 | MDEyOklzc3VlQ29tbWVudDYxNjg4MzI3NQ== | 9599 | 2020-04-21T00:43:28Z | 2020-04-21T00:43:28Z | MEMBER | I'm copying repo from issue, which surprisingly is a string, not an integer ID. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 603617013 | |
https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531516956 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3 | 531516956 | MDEyOklzc3VlQ29tbWVudDUzMTUxNjk1Ng== | 9599 | 2019-09-14T21:56:31Z | 2019-09-14T21:56:31Z | MEMBER | https://api.github.com/users/simonw/repos It would be useful to be able to fetch stargazers, forks etc as well. Not sure if that should be a separate command or a `--stargazers` option to this command. Probably a separate command since `issues` is a separate command already. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 493670426 | |
https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531517083 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3 | 531517083 | MDEyOklzc3VlQ29tbWVudDUzMTUxNzA4Mw== | 9599 | 2019-09-14T21:58:42Z | 2019-09-14T21:58:42Z | MEMBER | Split stargazers into #4 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 493670426 | |
https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616879753 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30 | 616879753 | MDEyOklzc3VlQ29tbWVudDYxNjg3OTc1Mw== | 9599 | 2020-04-21T00:29:29Z | 2020-04-21T00:29:29Z | MEMBER | `assignee` looks like it's the incorrect type too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 603618244 | |
https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616883726 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30 | 616883726 | MDEyOklzc3VlQ29tbWVudDYxNjg4MzcyNg== | 9599 | 2020-04-21T00:45:23Z | 2020-04-21T00:45:23Z | MEMBER | Demo of fix: https://github-to-sqlite.dogsheep.net/github/issues?assignee__notblank=1&milestone__notblank=1 ![github__issues__4_rows_where_where_assignee_is_not_blank_and_milestone_is_not_blank_sorted_by_updated_at_descending](https://user-images.githubusercontent.com/9599/79812758-b49ace80-832e-11ea-81db-bdf993b872cc.png) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 603618244 | |
https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-616884647 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31 | 616884647 | MDEyOklzc3VlQ29tbWVudDYxNjg4NDY0Nw== | 9599 | 2020-04-21T00:49:16Z | 2020-04-21T00:50:20Z | MEMBER | The API just gives us the `repository_url`: https://api.github.com/repos/simonw/datasette/issues ![Mozilla_Firefox_and_Topic__Week_2__Discussion__Submit_your_six_story_points_here](https://user-images.githubusercontent.com/9599/79812950-283cdb80-832f-11ea-8759-9633087d1e7e.png) We currently turn that into a `simonw/datasette` string here: https://github.com/dogsheep/github-to-sqlite/blob/e0e8d8caa9657b04bfb8a2cf16c9b580f38b1805/github_to_sqlite/utils.py#L43-L46 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 603624862 | |
https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617348174 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31 | 617348174 | MDEyOklzc3VlQ29tbWVudDYxNzM0ODE3NA== | 9599 | 2020-04-21T18:50:29Z | 2020-04-21T18:50:29Z | MEMBER | Since this represents a breaking schema change for anyone running SQL queries against these tables, I'm going to do a major version bump to 2.0 when I release this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 603624862 | |
https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617491607 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31 | 617491607 | MDEyOklzc3VlQ29tbWVudDYxNzQ5MTYwNw== | 9599 | 2020-04-22T01:20:19Z | 2020-04-22T01:20:19Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/milestones now link to repo: <img width="1095" alt="Screen Shot 2020-04-21 at 6 19 03 PM" src="https://user-images.githubusercontent.com/9599/79930145-956a7280-83fc-11ea-9b47-b0c5674a0d4f.png"> And so do issues: https://github-to-sqlite.dogsheep.net/github/issues <img width="1095" alt="Screen Shot 2020-04-21 at 6 19 53 PM" src="https://user-images.githubusercontent.com/9599/79930200-bd59d600-83fc-11ea-8fb8-b40772140409.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 603624862 | |
https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617364956 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32 | 617364956 | MDEyOklzc3VlQ29tbWVudDYxNzM2NDk1Ng== | 9599 | 2020-04-21T19:24:45Z | 2020-04-21T19:24:45Z | MEMBER | That's because I just broke this code: https://github.com/dogsheep/github-to-sqlite/blob/2cf75a0a036719eb7e57fdc7c5c2ea0f4c26978a/github_to_sqlite/utils.py#L131-L139 It expects the `repo` column to be `simonw/datasette` but it's now an ID instead. I should add a test for this as part of the fix. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 604222295 | |
https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617369247 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32 | 617369247 | MDEyOklzc3VlQ29tbWVudDYxNzM2OTI0Nw== | 9599 | 2020-04-21T19:33:03Z | 2020-04-21T19:33:03Z | MEMBER | Caused by #31. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 604222295 | |
https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617490914 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32 | 617490914 | MDEyOklzc3VlQ29tbWVudDYxNzQ5MDkxNA== | 9599 | 2020-04-22T01:17:44Z | 2020-04-22T01:17:44Z | MEMBER | https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101 now shows issues. And https://github-to-sqlite.dogsheep.net/github/issue_comments links to them: <img width="395" alt="Screen Shot 2020-04-21 at 6 17 33 PM" src="https://user-images.githubusercontent.com/9599/79930066-6227e380-83fc-11ea-8416-8b9d55660cb5.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 604222295 | |
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622169728 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33 | 622169728 | MDEyOklzc3VlQ29tbWVudDYyMjE2OTcyOA== | 9599 | 2020-04-30T23:18:51Z | 2020-04-30T23:18:51Z | MEMBER | Sure, that sounds fine to me. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 609950090 | |
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622171097 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33 | 622171097 | MDEyOklzc3VlQ29tbWVudDYyMjE3MTA5Nw== | 9599 | 2020-04-30T23:22:45Z | 2020-04-30T23:23:57Z | MEMBER | The `auth.json` mechanism this uses is standard across all of the other Dogsheep tools - it's actually designed so you can have one `auth.json` with a bunch of different credentials for different tools: ```json { "goodreads_personal_token": "...", "goodreads_user_id": "...", "github_personal_token": "...", "pocket_consumer_key": "...", "pocket_username": "...", "pocket_access_token": "..." } ``` But... `github-to-sqlite` does feel like it deserves a special case here, since it's such a good fit for running inside of GitHub Actions - which even provide a `GITHUB_TOKEN` for you to use! So I don't think it will harm the family of tools too much if this has an environment variable alternative to the `-a` file. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 609950090 | |
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622997410 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33 | 622997410 | MDEyOklzc3VlQ29tbWVudDYyMjk5NzQxMA== | 9599 | 2020-05-02T18:46:10Z | 2020-05-02T18:46:10Z | MEMBER | Documented here: https://github.com/dogsheep/github-to-sqlite/blob/10fb34de41aaa35681f08b5991540d65bfcf2e2e/README.md#authentication | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 609950090 | |
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133298 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 | 622133298 | MDEyOklzc3VlQ29tbWVudDYyMjEzMzI5OA== | 9599 | 2020-04-30T21:48:24Z | 2020-04-30T21:48:24Z | MEMBER | Unfortunately it's not available through any GitHub API - I managed to figure out how to get dependencies, but I need dependents. https://github.com/simonw/til/blob/master/github/dependencies-graphql-api.md | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610408908 | |
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133422 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 | 622133422 | MDEyOklzc3VlQ29tbWVudDYyMjEzMzQyMg== | 9599 | 2020-04-30T21:48:39Z | 2020-04-30T21:48:39Z | MEMBER | It looks like the only option is to scrape them. I'll do that and then replace with an API as soon as one becomes available. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610408908 | |
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133775 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 | 622133775 | MDEyOklzc3VlQ29tbWVudDYyMjEzMzc3NQ== | 9599 | 2020-04-30T21:49:27Z | 2020-04-30T21:49:27Z | MEMBER | Proposed command: github-to-sqlite scrape-dependents github.db simonw/datasette I'll pull full details of the scraped repos from the regular API. I'll also record when they were "first seen" by the command. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610408908 | |
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622135654 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 | 622135654 | MDEyOklzc3VlQ29tbWVudDYyMjEzNTY1NA== | 9599 | 2020-04-30T21:53:44Z | 2020-04-30T21:56:06Z | MEMBER | I think this is the neatest scraping pattern: ```python [a["href"].lstrip("/") for a in soup.select("a[data-hovercard-type=repository]")] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610408908 | |
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622136585 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 | 622136585 | MDEyOklzc3VlQ29tbWVudDYyMjEzNjU4NQ== | 9599 | 2020-04-30T21:55:51Z | 2020-04-30T21:55:51Z | MEMBER | And to find the "Next" pagination link: ```python soup.select(".paginate-container")[0].find("a", text="Next") ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610408908 | |
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622162835 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 | 622162835 | MDEyOklzc3VlQ29tbWVudDYyMjE2MjgzNQ== | 9599 | 2020-04-30T22:59:26Z | 2020-04-30T22:59:26Z | MEMBER | Documentation: https://github.com/dogsheep/github-to-sqlite/blob/c9f48404481882e8b3af06f35e4801a80ac79ed6/README.md#scraping-dependents-for-a-repository | {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 2, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610408908 | |
https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622213950 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35 | 622213950 | MDEyOklzc3VlQ29tbWVudDYyMjIxMzk1MA== | 9599 | 2020-05-01T02:09:04Z | 2020-05-01T02:09:04Z | MEMBER | It sped up this query a lot - 2.5s down to 300ms: ```sql select repos.full_name, json_object( 'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number, 'label', '#' || issues.number ) as issue, issues.title, users.login, users.id, issues.state, issues.locked, issues.assignee, issues.milestone, issues.comments, issues.created_at, issues.updated_at, issues.closed_at, issues.author_association, issues.pull_request, issues.repo, issues.type from issues join repos on repos.id = issues.repo join users on issues.user = users.id where issues.state = 'open' and issues.user not in (9599, 27856297) and not exists ( select id from issue_comments where issue_comments.user = 9599 and issues.id = issue_comments.issue ) order by issues.updated_at desc; ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610511450 | |
https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622214262 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35 | 622214262 | MDEyOklzc3VlQ29tbWVudDYyMjIxNDI2Mg== | 9599 | 2020-05-01T02:10:32Z | 2020-05-01T02:11:19Z | MEMBER | This sped that query up even more - down to 4ms. ```sql create index issue_comments_issue on issue_comments(issue); ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610511450 | |
https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622982667 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35 | 622982667 | MDEyOklzc3VlQ29tbWVudDYyMjk4MjY2Nw== | 9599 | 2020-05-02T16:52:53Z | 2020-05-02T16:52:53Z | MEMBER | Easiest option: use `db.index_foreign_keys()`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#adding-indexes-for-all-foreign-keys | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610511450 | |
https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622461025 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36 | 622461025 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTAyNQ== | 9599 | 2020-05-01T16:34:24Z | 2020-05-01T16:34:24Z | MEMBER | Blocked on #37 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610842926 | |
https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622982346 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36 | 622982346 | MDEyOklzc3VlQ29tbWVudDYyMjk4MjM0Ng== | 9599 | 2020-05-02T16:50:31Z | 2020-05-02T16:50:31Z | MEMBER | Demo: https://github-to-sqlite.dogsheep.net/github/dependent_repos | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610842926 | |
https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461537 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37 | 622461537 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTUzNw== | 9599 | 2020-05-01T16:35:40Z | 2020-05-01T16:35:40Z | MEMBER | This will check if the view exists and has the exact same matching definition as the one we want. If it doesn't, we will drop it (if it exists) and recreate it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610843136 | |
https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461948 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37 | 622461948 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTk0OA== | 9599 | 2020-05-01T16:36:42Z | 2020-05-01T16:36:42Z | MEMBER | It should only create views if the underlying tables exist. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610843136 | |
https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622978173 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37 | 622978173 | MDEyOklzc3VlQ29tbWVudDYyMjk3ODE3Mw== | 9599 | 2020-05-02T16:19:31Z | 2020-05-02T16:19:47Z | MEMBER | I can use the new `.create_view(..., replace=True)` parameter in `sqlite-utils` 2.7.2 for this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 610843136 | |
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623027889 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38 | 623027889 | MDEyOklzc3VlQ29tbWVudDYyMzAyNzg4OQ== | 9599 | 2020-05-02T23:15:11Z | 2020-05-02T23:15:11Z | MEMBER | This is one of the use-cases for the `repos_starred` view: it allows you to easily run this kid of query without having to construct the SQL by hand. Here's a demo: https://github-to-sqlite.dogsheep.net/github/repos_starred?name__contains=twitter My philosophy here is to keep the raw tables (like `stars`) as normalized as possible, then use SQL views which expose the data in a form that's easier to query. | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 611284481 | |
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038378 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38 | 623038378 | MDEyOklzc3VlQ29tbWVudDYyMzAzODM3OA== | 9599 | 2020-05-03T01:21:13Z | 2020-05-03T01:21:13Z | MEMBER | No this is really useful feedback! I'm so close to this project that I miss what's not obvious to people dropping in for the first time. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 611284481 | |
https://github.com/dogsheep/github-to-sqlite/issues/39#issuecomment-675509550 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/39 | 675509550 | MDEyOklzc3VlQ29tbWVudDY3NTUwOTU1MA== | 9599 | 2020-08-18T14:23:56Z | 2020-08-18T14:23:56Z | MEMBER | I think this is fixed: https://github-to-sqlite.dogsheep.net/github/issues?_facet=repo | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 613777056 | |
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-531517138 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | 531517138 | MDEyOklzc3VlQ29tbWVudDUzMTUxNzEzOA== | 9599 | 2019-09-14T21:59:59Z | 2019-09-14T21:59:59Z | MEMBER | Paginate through https://api.github.com/repos/simonw/datasette/stargazers Send `Accept: application/vnd.github.v3.star+json` to get the `starred_at` dates. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 493670730 | |
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-550388354 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | 550388354 | MDEyOklzc3VlQ29tbWVudDU1MDM4ODM1NA== | 9599 | 2019-11-06T16:26:55Z | 2019-11-06T16:26:55Z | MEMBER | Here's a query I figured out using a window function that shows cumulative stargazers over time: ```sql select yyyymmdd, sum(n) over ( order by yyyymmdd rows unbounded preceding ) as cumulative_count from ( select substr(starred_at, 0, 11) as yyyymmdd, count(*) as n from stars group by yyyymmdd ) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 493670730 | |
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-622990947 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | 622990947 | MDEyOklzc3VlQ29tbWVudDYyMjk5MDk0Nw== | 9599 | 2020-05-02T17:54:16Z | 2020-05-02T17:54:16Z | MEMBER | I could add that window function query as a view, but only if the detected version of SQLite supports window functions. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 493670730 | |
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623000814 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | 623000814 | MDEyOklzc3VlQ29tbWVudDYyMzAwMDgxNA== | 9599 | 2020-05-02T19:15:23Z | 2020-05-02T19:15:23Z | MEMBER | I'm not going to do the `--sql` bit just yet. I have patterns for working around this for other commands which are working fine: https://github.com/dogsheep/github-to-sqlite/blob/d00a53061556dc403c166b443d141c4e1adbd64a/.github/workflows/deploy-demo.yml#L53-L70 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 493670730 | |
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623004836 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | 623004836 | MDEyOklzc3VlQ29tbWVudDYyMzAwNDgzNg== | 9599 | 2020-05-02T19:49:51Z | 2020-05-02T19:49:51Z | MEMBER | Alternative pattern: ``` sqlite-utils releases.db 'select full_name from repos' --csv --no-headers \ | tr -d '\r' \ | xargs github-to-sqlite stargazers stars.db ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 493670730 | |
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623006154 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | 623006154 | MDEyOklzc3VlQ29tbWVudDYyMzAwNjE1NA== | 9599 | 2020-05-02T20:01:39Z | 2020-05-02T20:01:54Z | MEMBER | Needs tests and documentation. I shipped it early to check that the live demo works. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 493670730 | |
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623007441 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | 623007441 | MDEyOklzc3VlQ29tbWVudDYyMzAwNzQ0MQ== | 9599 | 2020-05-02T20:13:37Z | 2020-05-02T20:13:37Z | MEMBER | Datasette cumulative stars over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 493670730 | |
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623010272 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | 623010272 | MDEyOklzc3VlQ29tbWVudDYyMzAxMDI3Mg== | 9599 | 2020-05-02T20:39:14Z | 2020-05-02T20:39:14Z | MEMBER | Graph of cumulative stars for Datasette over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29#g.mark=line&g.x_column=yyyymmdd&g.x_type=temporal&g.y_column=cumulative_count&g.y_type=quantitative <img width="291" alt="github__select_yyyymmdd__sum_n__over___order_by_yyyymmdd_rows_unbounded_preceding___as_cumulative_count_from___select_substr_starred_at__0__11__as_yyyymmdd__count____as_n_from_stars_where_repo___107914493_group_by_yyyymmdd__" src="https://user-images.githubusercontent.com/9599/80891642-31587180-8c7a-11ea-9169-60eb3cb6d3e0.png"> Stars per day (as a label bar chart, so very wide): https://github-to-sqlite.dogsheep.net/github?sql=%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++#g.mark=bar&g.x_column=yyyymmdd&g.x_type=ordinal&g.y_column=n&g.y_type=quantitative <img width="1424" alt="github__select_substr_starred_at__0__11__as_yyyymmdd__count____as_n_from_stars_where_repo___107914493_group_by_yyyymmdd" src="https://user-images.githubusercontent.com/9599/80891656-4e8d4000-8c7a-11ea-9ff2-02c9cf4cb5d5.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 493670730 | |
https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643393506 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40 | 643393506 | MDEyOklzc3VlQ29tbWVudDY0MzM5MzUwNg== | 9599 | 2020-06-12T17:21:14Z | 2020-06-12T17:21:14Z | MEMBER | I only install SQLite for this: https://github.com/dogsheep/github-to-sqlite/blob/c0d54e0260468be38152293df5abd775c068495d/.github/workflows/deploy-demo.yml#L77-L78 I'm going to remove the need to install sqlite3 by making this possible with sqlite-utils: https://github.com/simonw/sqlite-utils/issues/115 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 637899539 | |
https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643414646 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40 | 643414646 | MDEyOklzc3VlQ29tbWVudDY0MzQxNDY0Ng== | 9599 | 2020-06-12T18:06:48Z | 2020-06-12T18:06:48Z | MEMBER | That fixed it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 637899539 | |
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653947916 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | 653947916 | MDEyOklzc3VlQ29tbWVudDY1Mzk0NzkxNg== | 9599 | 2020-07-05T22:40:47Z | 2020-07-05T22:40:47Z | MEMBER | Might be that it's not got enough RAM. I'll try deploying to a larger instance. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 651159727 | |
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653960989 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | 653960989 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MDk4OQ== | 9599 | 2020-07-06T00:32:34Z | 2020-07-06T00:32:34Z | MEMBER | Same error. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 651159727 | |
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962418 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | 653962418 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjQxOA== | 9599 | 2020-07-06T00:41:38Z | 2020-07-06T00:41:38Z | MEMBER | https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/logs?project=datasette-222320 has some clues. <img width="818" alt="github-to-sqlite_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/86545887-92086280-bee6-11ea-8787-b3b85a7bb4b9.png"> <img width="1451" alt="github-to-sqlite_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/86545917-c11ed400-bee6-11ea-9cf8-957911ca9f6a.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 651159727 | |
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962530 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | 653962530 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjUzMA== | 9599 | 2020-07-06T00:42:13Z | 2020-07-06T00:42:13Z | MEMBER | So it looks like it's the ASGI lifespan change I made in https://github.com/simonw/datasette/commit/16f592247a2a0e140ada487e9972645406dcae69 - It must be incompatible with one of the plugins. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 651159727 | |
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962669 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | 653962669 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjY2OQ== | 9599 | 2020-07-06T00:42:57Z | 2020-07-06T00:42:57Z | MEMBER | https://github-to-sqlite.dogsheep.net/-/plugins ```json [ { "name": "datasette-json-html", "static": false, "templates": false, "version": "0.6", "hooks": [ "prepare_connection", "render_cell" ] }, { "name": "datasette-render-markdown", "static": false, "templates": false, "version": "1.1.2", "hooks": [ "extra_template_vars", "render_cell" ] }, { "name": "datasette-pretty-json", "static": false, "templates": false, "version": "0.2", "hooks": [ "render_cell" ] }, { "name": "datasette-search-all", "static": false, "templates": true, "version": "0.2.1", "hooks": [ "asgi_wrapper", "extra_template_vars" ] }, { "name": "datasette-vega", "static": true, "templates": false, "version": "0.6.2", "hooks": [ "extra_css_urls", "extra_js_urls" ] } ] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 651159727 | |
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962708 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | 653962708 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjcwOA== | 9599 | 2020-07-06T00:43:10Z | 2020-07-06T00:43:10Z | MEMBER | I bet it's datasette-search-all. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 651159727 | |
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653966670 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | 653966670 | MDEyOklzc3VlQ29tbWVudDY1Mzk2NjY3MA== | 9599 | 2020-07-06T01:07:02Z | 2020-07-06T01:07:02Z | MEMBER | OK that fix worked.https://github.com/dogsheep/github-to-sqlite/runs/839764768?check_suite_focus=true | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 651159727 | |
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660536265 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | 660536265 | MDEyOklzc3VlQ29tbWVudDY2MDUzNjI2NQ== | 9599 | 2020-07-18T20:15:12Z | 2020-07-18T20:15:12Z | MEMBER | I want to create a SQL query which shows me all of my repositories that have commits that are NOT in the most recent release. The releases table doesn't have enough information for this because it doesn't tell you the commit hash associated with each release, just the tag: https://github-to-sqlite.dogsheep.net/github/releases | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 660355904 | |
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660547502 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | 660547502 | MDEyOklzc3VlQ29tbWVudDY2MDU0NzUwMg== | 9599 | 2020-07-18T21:50:37Z | 2020-07-18T21:50:37Z | MEMBER | ``` $ github-to-sqlite tags tags.db simonw/datasette dogsheep/github-to-sqlite $ sqlite-utils tables tags.db --counts [{"table": "users", "count": 2}, {"table": "licenses", "count": 1}, {"table": "repos", "count": 2}, {"table": "tags", "count": 76}, {"table": "licenses_fts", "count": 1}, {"table": "licenses_fts_data", "count": 3}, {"table": "licenses_fts_idx", "count": 1}, {"table": "licenses_fts_docsize", "count": 1}, {"table": "licenses_fts_config", "count": 1}, {"table": "repos_fts", "count": 2}, {"table": "repos_fts_data", "count": 3}, {"table": "repos_fts_idx", "count": 1}, {"table": "repos_fts_docsize", "count": 2}, {"table": "repos_fts_config", "count": 1}, {"table": "users_fts", "count": 2}, {"table": "users_fts_data", "count": 3}, {"table": "users_fts_idx", "count": 1}, {"table": "users_fts_docsize", "count": 2}, {"table": "users_fts_config", "count": 1}] $ sqlite-utils rows tags.db tags [{"repo_id": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"}, {"repo_id": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"}, {"repo_id": 107914493, "name": "0.45a4", "sha": "265483173bc8341dc02c8b782b9b59d2ce8bbedb"}, {"repo_id": 107914493, "name": "0.45a3", "sha": "1f55a4a2b68fa65e56a28baeb7f44122fdeca7e7"}, {"repo_id": 107914493, "name": "0.45a2", "sha": "1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f"}, {"repo_id": 107914493, "name": "0.45a1", "sha": "b59b92b1b0517cf18fa748ff9d0a0bf86298dd43"}, {"repo_id": 107914493, "name": "0.45a0", "sha": "dda932d818b34ccab11730a76554f0a3748d8348"}, {"repo_id": 107914493, "name": "0.44", "sha": "b906030235efbdff536405d66078f4868ce0d3bd"}, {"repo_id": 107914493, "name": "0.43", "sha": "d56f402822df102f9cf1a9a056449d01a15e3aae"}, {"repo_id": 107914493, "name": "0.42", "sha": "af6c6c5d6f929f951c0e63bfd1c82e37a071b50f"}, {"repo_id": 107914493, "name": "0.41", "sha": "182e5c8745c94576718315f7596ccc81e5e2417b"}, {"repo_id": 107914493, "name": "0.40", "sha": "8da108193b08ab… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 660355904 | |
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660548780 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | 660548780 | MDEyOklzc3VlQ29tbWVudDY2MDU0ODc4MA== | 9599 | 2020-07-18T22:02:37Z | 2020-07-18T23:05:56Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/tags?_facet=repo | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 660355904 | |
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660551397 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | 660551397 | MDEyOklzc3VlQ29tbWVudDY2MDU1MTM5Nw== | 9599 | 2020-07-18T22:27:32Z | 2020-07-18T23:05:45Z | MEMBER | ```sql with most_recent_releases as ( with ranked as ( select repo, tag_name, published_at, row_number() OVER ( partition BY repo ORDER BY published_at DESC ) rank FROM releases ) select * from ranked where rank = 1 ) select repos.full_name as repo, most_recent_releases.tag_name as release, commits.committer_date as release_commit_date, ( select count(*) from commits c2 where c2.repo = repos.id and c2.committer_date > commits.committer_date ) as commits_since_release, 'https://github.com/' || repos.full_name || '/compare/' || most_recent_releases.tag_name || '...' || repos.default_branch as view_commits from most_recent_releases join repos on most_recent_releases.repo = repos.id join tags on tags.repo = repos.id and tags.name = most_recent_releases.tag_name join commits on tags.sha = commits.sha order by commits_since_release desc ``` repo | release | release_commit_date | commits_since_release | view_commits -- | -- | -- | -- | -- simonw/datasette | 0.45 | 2020-07-01T21:43:07Z | 9 | https://github.com/simonw/datasette/compare/0.45...master dogsheep/twitter-to-sqlite | 0.21.1 | 2020-04-30T18:20:43Z | 2 | https://github.com/dogsheep/twitter-to-sqlite/compare/0.21.1...master dogsheep/github-to-sqlite | 2.3 | 2020-07-09T23:26:34Z | 2 | https://github.com/dogsheep/github-to-sqlite/compare/2.3...master dogsheep/dogsheep-photos | 0.4.1 | 2020-05-25T20:11:20Z | 2 | https://github.com/dogsheep/dogsheep-photos/compare/0.4.1...master dogsheep/swarm-to-sqlite | 0.3.1 | 2020-03-28T02:29:41Z | 1 | https://github.com/dogsheep/swarm-to-sqlite/compare/0.3.1...master dogsheep/hacker-news-to-sqlite | 0.3.1 | 2020-03-21T22:39:34Z | 1 | https://github.com/dogsheep/hacker-news-to-sqlite/compare/0.3.1...master simonw/sqlite-utils | 2.11 | 2020-07-08T17:36:07Z | 0 | https://github.com/simonw/sqlite-utils/compare/2.11...master dog… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 660355904 | |
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553646 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | 660553646 | MDEyOklzc3VlQ29tbWVudDY2MDU1MzY0Ng== | 9599 | 2020-07-18T22:51:41Z | 2020-07-18T22:51:41Z | MEMBER | I could fix this by putting `REFRESH_DB` in a commit message: https://github.com/dogsheep/github-to-sqlite/blob/4ae4aa6f172344b19ff3513707195ee6d2654bd4/.github/workflows/deploy-demo.yml#L41-L46 But... doing so would lose the data I've collected in https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc concerning the first time each dependent repo was spotted. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 660429601 | |
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553711 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | 660553711 | MDEyOklzc3VlQ29tbWVudDY2MDU1MzcxMQ== | 9599 | 2020-07-18T22:52:16Z | 2020-07-18T22:52:16Z | MEMBER | I think the best fix is to download the `github.db` database, manually fix it and then manually deploy it to Cloud Run from my laptop. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 660429601 | |
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554162 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | 660554162 | MDEyOklzc3VlQ29tbWVudDY2MDU1NDE2Mg== | 9599 | 2020-07-18T22:56:58Z | 2020-07-18T22:56:58Z | MEMBER | Manually fixing the database: ``` $ wget 'https://github-to-sqlite.dogsheep.net/github.db' --2020-07-18 15:52:33-- https://github-to-sqlite.dogsheep.net/github.db Resolving github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)... 172.217.5.115 Connecting to github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)|172.217.5.115|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 14626816 (14M) [application/octet-stream] Saving to: ‘github.db’ github.db 100%[============================================================================>] 13.95M 1.22MB/s in 18s 2020-07-18 15:52:53 (773 KB/s) - ‘github.db’ saved [14626816/14626816] $ sqlite3 github.db SQLite version 3.28.0 2019-04-15 14:49:49 Enter ".help" for usage hints. sqlite> drop table tags; sqlite> ^D $ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils dogsheep/healthkit-to-sqlite dogsheep/swarm-to-sqlite dogsheep/twitter-to-sqlite dogsheep/inaturalist-to-sqlite dogsheep/google-takeout-to-sqlite dogsheep/github-to-sqlite dogsheep/genome-to-sqlite dogsheep/pocket-to-sqlite dogsheep/hacker-news-to-sqlite dogsheep/dogsheep-photos $ sqlite-utils tables github.db --counts [{"table": "users", "count": 4048}, {"table": "repos", "count": 210}, ... {"table": "stars", "count": 4140}, {"table": "tags", "count": 188}] $ sqlite-utils rows github.db tags [{"repo": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"}, {"repo": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"}, ... ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 660429601 |