\r\n```\r\nSo this is a bug in GitHub's API, but we need to work around it.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 771872303, "node_id": "MDExOlB1bGxSZXF1ZXN0NTQzMjQ2NTM1", "number": 59, "title": "Remove unneeded exists=True for -a/--auth flag.", "user": {"value": 631242, "label": "frosencrantz"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-12-21T06:03:55Z", "updated_at": "2021-05-22T14:06:19Z", "closed_at": "2021-05-19T16:08:12Z", "author_association": "CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/59", "body": "The file does not need to exist when using an environment variable.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 797097140, "node_id": "MDU6SXNzdWU3OTcwOTcxNDA=", "number": 60, "title": "Use Data from SQLite in other commands", "user": {"value": 22578954, "label": "daniel-butler"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-01-29T18:35:52Z", "updated_at": "2021-02-12T18:29:43Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "As a total beginner here how could you access data from the sqlite table to run other commands.\r\n\r\nWhat I am thinking is I want to get all the repos in an organization then using the repo list pull all the commit messages for each repo. \r\n\r\nI love this project by the way!", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 797108702, "node_id": "MDExOlB1bGxSZXF1ZXN0NTY0MTcyMTQw", "number": 61, "title": "fixing typo in get cli help text", "user": {"value": 22578954, "label": "daniel-butler"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-01-29T18:57:04Z", "updated_at": "2021-05-19T16:07:09Z", "closed_at": "2021-05-19T16:07:09Z", "author_association": "CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/61", "body": "", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/61/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 797784080, "node_id": "MDU6SXNzdWU3OTc3ODQwODA=", "number": 62, "title": "Stargazers and workflows commands always require an auth file when using GITHUB_TOKEN ", "user": {"value": 631242, "label": "frosencrantz"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-01-31T18:56:05Z", "updated_at": "2021-01-31T18:56:05Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Requested fix in https://github.com/dogsheep/github-to-sqlite/pull/59\r\n\r\nThe stargazers and workflows commands always require an auth file, even when using a `GITHUB_TOKEN`. Other commands don't require the auth file.\r\n\r\n", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/62/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 897212458, "node_id": "MDU6SXNzdWU4OTcyMTI0NTg=", "number": 63, "title": "Ability to fetch commits from branches other than the default", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-05-20T17:58:08Z", "updated_at": "2021-05-20T17:58:08Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "This tool is currently almost entirely ignorant of the concept of branches. One example: you can't retrieve commits from any branch other than the default (usually main).", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/63/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 920636216, "node_id": "MDU6SXNzdWU5MjA2MzYyMTY=", "number": 64, "title": "feature: support \"events\"", "user": {"value": 231498, "label": "khimaros"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-06-14T17:42:49Z", "updated_at": "2021-06-15T00:48:37Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "the GitHub API provides the ability to fetch all events for a given user, organization, or repository: https://docs.github.com/en/rest/reference/activity#list-events-for-the-authenticated-user\r\n\r\nthis would allow users to export all of the issue comments, new issues, etc. that they created. something which is currently missing from the GitHub takeout exports.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 923270900, "node_id": "MDExOlB1bGxSZXF1ZXN0NjcyMDUzODEx", "number": 65, "title": "basic support for events", "user": {"value": 231498, "label": "khimaros"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-06-17T00:51:30Z", "updated_at": "2022-10-03T22:35:03Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/65", "body": "a quick first pass at implementing the feature requested in https://github.com/dogsheep/github-to-sqlite/issues/64\r\n\r\ntesting instructions:\r\n\r\n```\r\n$ github-to-sqlite events events.db user/khimaros\r\n```\r\n\r\nif the specified user is the authenticated user, it will also include private events.\r\n\r\ncaveat: pagination appears to be broken (i don't see `next` in the response JSON from GitHub)", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 975161924, "node_id": "MDExOlB1bGxSZXF1ZXN0NzE2MzU3OTgy", "number": 66, "title": "Add --merged-by flag to pull-requests sub command", "user": {"value": 30531572, "label": "sarcasticadmin"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-08-20T00:57:55Z", "updated_at": "2021-09-28T21:50:31Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/66", "body": "## Description\r\n\r\nProposing a solution to the API limitation for `merged_by` in pull_requests. Specifically the following called out in the readme:\r\n\r\n```\r\nNote that the merged_by column on the pull_requests table will only be populated for pull requests that are loaded using the --pull-request option - the GitHub API does not return this field for pull requests that are loaded in bulk.\r\n```\r\n\r\nThis approach might cause larger repos to hit rate limits called out in https://github.com/dogsheep/github-to-sqlite/issues/51 but seems to work well in the repos I tested and included below.\r\n\r\n## Old Behavior\r\n- Had to list out the pull-requests individually via multiple `--pull-request` flags\r\n\r\n## New Behavior\r\n\r\n- `--merged-by` flag for getting 'merge_by' information out of pull-requests without having to specify individual PR numbers.\r\n\r\n# Testing\r\n\r\nPicking some repo that has more than one merger (datasette only has 1 \ud83d\ude09 )\r\n\r\n```\r\n$ github-to-sqlite pull-requests ./github.db opnsense/tools --merged-by\r\n$ echo \"select id, url, merged_by from pull_requests;\" | sqlite3 ./github.db \r\n83533612|https://github.com/opnsense/tools/pull/39|1915288\r\n102632885|https://github.com/opnsense/tools/pull/43|1915288\r\n149114810|https://github.com/opnsense/tools/pull/57|1915288\r\n160394495|https://github.com/opnsense/tools/pull/64|1915288\r\n163308408|https://github.com/opnsense/tools/pull/67|1915288\r\n169723264|https://github.com/opnsense/tools/pull/69|1915288\r\n171381422|https://github.com/opnsense/tools/pull/72|1915288\r\n179938195|https://github.com/opnsense/tools/pull/77|1915288\r\n196233824|https://github.com/opnsense/tools/pull/82|1915288\r\n215289964|https://github.com/opnsense/tools/pull/93|\r\n219696100|https://github.com/opnsense/tools/pull/97|1915288\r\n223664843|https://github.com/opnsense/tools/pull/99|\r\n228446172|https://github.com/opnsense/tools/pull/103|1915288\r\n238930434|https://github.com/opnsense/tools/pull/110|1915288\r\n255507110|https://github.com/opnsense/tools/pull/119|1915288\r\n255980675|https://github.com/opnsense/tools/pull/120|1915288\r\n261906770|https://github.com/opnsense/tools/pull/125|\r\n263800503|https://github.com/opnsense/tools/pull/127|1915288\r\n264038685|https://github.com/opnsense/tools/pull/128|1915288\r\n264696704|https://github.com/opnsense/tools/pull/129|1915288\r\n266660547|https://github.com/opnsense/tools/pull/130|1915288\r\n273120409|https://github.com/opnsense/tools/pull/133|1915288\r\n274370803|https://github.com/opnsense/tools/pull/135|\r\n276600629|https://github.com/opnsense/tools/pull/139|\r\n277303655|https://github.com/opnsense/tools/pull/141|1915288\r\n293033714|https://github.com/opnsense/tools/pull/145|\r\n294827649|https://github.com/opnsense/tools/pull/146|\r\n295140008|https://github.com/opnsense/tools/pull/147|1915288\r\n305690829|https://github.com/opnsense/tools/pull/150|9783985\r\n307077931|https://github.com/opnsense/tools/pull/152|1915288\r\n321782100|https://github.com/opnsense/tools/pull/155|\r\n337265672|https://github.com/opnsense/tools/pull/160|\r\n337267484|https://github.com/opnsense/tools/pull/161|1915288\r\n368251763|https://github.com/opnsense/tools/pull/169|\r\n428262505|https://github.com/opnsense/tools/pull/181|\r\n437557011|https://github.com/opnsense/tools/pull/182|1915288\r\n447079893|https://github.com/opnsense/tools/pull/185|\r\n461822092|https://github.com/opnsense/tools/pull/191|\r\n463290142|https://github.com/opnsense/tools/pull/193|1915288\r\n470112962|https://github.com/opnsense/tools/pull/194|1915288\r\n472644649|https://github.com/opnsense/tools/pull/195|1915288\r\n488696898|https://github.com/opnsense/tools/pull/198|\r\n513289902|https://github.com/opnsense/tools/pull/201|\r\n522530265|https://github.com/opnsense/tools/pull/203|\r\n564443347|https://github.com/opnsense/tools/pull/213|\r\n597579516|https://github.com/opnsense/tools/pull/220|1915288\r\n602860357|https://github.com/opnsense/tools/pull/221|1915288\r\n608744738|https://github.com/opnsense/tools/pull/222|1915288\r\n623279673|https://github.com/opnsense/tools/pull/228|1915288\r\n664656182|https://github.com/opnsense/tools/pull/233|\r\n664781786|https://github.com/opnsense/tools/pull/234|1915288\r\n670683636|https://github.com/opnsense/tools/pull/235|1915288\r\n683150764|https://github.com/opnsense/tools/pull/237|\r\n685016233|https://github.com/opnsense/tools/pull/238|\r\n687099825|https://github.com/opnsense/tools/pull/239|1915288\r\n715705652|https://github.com/opnsense/tools/pull/244|1915288\r\n715721248|https://github.com/opnsense/tools/pull/245|1915288\r\n```\r\n`userid` are now present for those PRs that were merged.\r\n\r\nWithout the flag the `merged_by` behavior remains missing as expected when get PRs bulk:\r\n\r\n```\r\n$ github-to-sqlite pull-requests ./github.db opnsense/tools\r\n$ echo \"select id, url, merged_by from pull_requests;\" | sqlite3 ./github.db \r\n83533612|https://github.com/opnsense/tools/pull/39|\r\n102632885|https://github.com/opnsense/tools/pull/43|\r\n149114810|https://github.com/opnsense/tools/pull/57|\r\n160394495|https://github.com/opnsense/tools/pull/64|\r\n163308408|https://github.com/opnsense/tools/pull/67|\r\n169723264|https://github.com/opnsense/tools/pull/69|\r\n171381422|https://github.com/opnsense/tools/pull/72|\r\n179938195|https://github.com/opnsense/tools/pull/77|\r\n196233824|https://github.com/opnsense/tools/pull/82|\r\n215289964|https://github.com/opnsense/tools/pull/93|\r\n219696100|https://github.com/opnsense/tools/pull/97|\r\n223664843|https://github.com/opnsense/tools/pull/99|\r\n228446172|https://github.com/opnsense/tools/pull/103|\r\n238930434|https://github.com/opnsense/tools/pull/110|\r\n255507110|https://github.com/opnsense/tools/pull/119|\r\n255980675|https://github.com/opnsense/tools/pull/120|\r\n261906770|https://github.com/opnsense/tools/pull/125|\r\n263800503|https://github.com/opnsense/tools/pull/127|\r\n264038685|https://github.com/opnsense/tools/pull/128|\r\n264696704|https://github.com/opnsense/tools/pull/129|\r\n266660547|https://github.com/opnsense/tools/pull/130|\r\n273120409|https://github.com/opnsense/tools/pull/133|\r\n274370803|https://github.com/opnsense/tools/pull/135|\r\n276600629|https://github.com/opnsense/tools/pull/139|\r\n277303655|https://github.com/opnsense/tools/pull/141|\r\n293033714|https://github.com/opnsense/tools/pull/145|\r\n294827649|https://github.com/opnsense/tools/pull/146|\r\n295140008|https://github.com/opnsense/tools/pull/147|\r\n305690829|https://github.com/opnsense/tools/pull/150|\r\n307077931|https://github.com/opnsense/tools/pull/152|\r\n321782100|https://github.com/opnsense/tools/pull/155|\r\n337265672|https://github.com/opnsense/tools/pull/160|\r\n337267484|https://github.com/opnsense/tools/pull/161|\r\n368251763|https://github.com/opnsense/tools/pull/169|\r\n428262505|https://github.com/opnsense/tools/pull/181|\r\n437557011|https://github.com/opnsense/tools/pull/182|\r\n447079893|https://github.com/opnsense/tools/pull/185|\r\n461822092|https://github.com/opnsense/tools/pull/191|\r\n463290142|https://github.com/opnsense/tools/pull/193|\r\n470112962|https://github.com/opnsense/tools/pull/194|\r\n472644649|https://github.com/opnsense/tools/pull/195|\r\n488696898|https://github.com/opnsense/tools/pull/198|\r\n513289902|https://github.com/opnsense/tools/pull/201|\r\n522530265|https://github.com/opnsense/tools/pull/203|\r\n564443347|https://github.com/opnsense/tools/pull/213|\r\n597579516|https://github.com/opnsense/tools/pull/220|\r\n602860357|https://github.com/opnsense/tools/pull/221|\r\n608744738|https://github.com/opnsense/tools/pull/222|\r\n623279673|https://github.com/opnsense/tools/pull/228|\r\n664656182|https://github.com/opnsense/tools/pull/233|\r\n664781786|https://github.com/opnsense/tools/pull/234|\r\n670683636|https://github.com/opnsense/tools/pull/235|\r\n683150764|https://github.com/opnsense/tools/pull/237|\r\n685016233|https://github.com/opnsense/tools/pull/238|\r\n687099825|https://github.com/opnsense/tools/pull/239|\r\n715705652|https://github.com/opnsense/tools/pull/244|\r\n715721248|https://github.com/opnsense/tools/pull/245|\r\n```\r\n\r\nIndividual PRs passed via `--pull-request` flag behaves as expected (unchanged):\r\n\r\n```\r\n$ github-to-sqlite pull-requests ./github.db opnsense/tools --pull-request 39 --pull-request 237\r\n$ echo \"select id, url, merged_by from pull_requests;\" | sqlite3 ./github.db\r\n83533612|https://github.com/opnsense/tools/pull/39|1915288\r\n683150764|https://github.com/opnsense/tools/pull/237|\r\n```\r\n> Picking 1 PR that has a merged_by (39) and one that does not (237)", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/66/reactions\", \"total_count\": 3, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 981690086, "node_id": "MDExOlB1bGxSZXF1ZXN0NzIxNjg2NzIx", "number": 67, "title": "Replacing step ID key with step_id", "user": {"value": 16374374, "label": "jshcmpbll"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-08-28T01:26:41Z", "updated_at": "2021-08-28T01:27:00Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/67", "body": "Workflows that have an `id` in any step result in the following error when running `workflows`:\r\n\r\ne.g.`github-to-sqlite workflows github.db nixos/nixpkgs`\r\n\r\n```Traceback (most recent call last):\r\n File \"/usr/local/bin/github-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1668, in invoke```Traceback (most recent call last):\r\n File \"/usr/local/bin/github-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.8/dist-packages/github_to_sqlite/cli.py\", line 601, in workflows\r\n utils.save_workflow(db, repo_id, filename, content)\r\n File \"/usr/local/lib/python3.8/dist-packages/github_to_sqlite/utils.py\", line 865, in save_workflow\r\n db[\"steps\"].insert_all(\r\n File \"/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py\", line 2596, in insert_all\r\n self.insert_chunk(\r\n File \"/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py\", line 2378, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py\", line 419, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.IntegrityError: datatype mismatch\r\n```\r\n\r\n - [Information about the ID key in a step for GHA](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepsid)\r\n - [An example workflow from a public repo](https://github.com/NixOS/nixpkgs/blob/b4cc66827745e525ce7bb54659845ac89788a597/.github/workflows/direct-push.yml#L16)\r\n\r\n# Changes\r\nI'm proposing that the key for `id` in step is replaced with `step_id` so that it no longer interferes with the table `id` for tracking the record.\r\n\r\nSpecial thanks to @sarcasticadmin @egiffen and @ruebenramirez for helping a bit on this \ud83d\ude04 ", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/67/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1013506559, "node_id": "PR_kwDODFdgUs4skaNS", "number": 68, "title": "Add support for retrieving teams / members", "user": {"value": 68329, "label": "philwills"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-10-01T15:55:02Z", "updated_at": "2021-10-01T15:59:53Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/68", "body": "Adds a method for retrieving all the teams within an organisation and all the members in those teams. The latter is stored as a join table `team_members` beteween `teams` and `users`.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/68/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1071071397, "node_id": "I_kwDODFdgUs4_10Cl", "number": 69, "title": "View that combines issues and issue comments", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-04T00:34:33Z", "updated_at": "2021-12-04T00:34:52Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I want to see a reverse chronologically ordered interface onto both issues and comments - essentially a unified log of comments and issues opened across one or multiple projects.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1149402080, "node_id": "PR_kwDODFdgUs4zaUta", "number": 70, "title": "scrape-dependents: enable paging through package menu option if present", "user": {"value": 36061055, "label": "stanbiryukov"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-02-24T15:07:25Z", "updated_at": "2022-02-24T15:07:25Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/70", "body": "Some repos organize network dependents by a Package toggle. This PR adds the ability to page through those options and scrape underlying dependents.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/70/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1177059481, "node_id": "I_kwDODFdgUs5GKICZ", "number": 71, "title": "Store commit parents", "user": {"value": 64686, "label": "carltongibson"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-03-22T17:06:48Z", "updated_at": "2022-04-22T12:44:04Z", "closed_at": "2022-04-22T12:44:04Z", "author_association": "NONE", "pull_request": null, "body": "Hi @simonw \ud83d\udc4b \r\n\r\nCurrently, stored commit data doesn't quite give me the information I'm needing... \r\n\r\nCommitter date and author date are not 100% reliable for dividing a commit history up by release or branch. \r\nA PR created before a release but merged after can have earlier dates\u2026 \u2014 this can be quite frustrating if you're trying to pin down commits for a release: _It should be there!_, but then isn't. (This gets worse using release branches.) \r\n\r\nWould you be open to adding the `sha` of a `parent` of a commit to the commit table? (As an FK? \ud83e\udd14 \u2014\u00a0likely not feasible.)\r\n\r\nIt's part of the [response body](https://docs.github.com/en/rest/reference/commits#get-a-commit): \r\n\r\n```\r\n \"parents\": [\r\n {\r\n \"url\": \"https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e\",\r\n \"sha\": \"6dcb09b5b57875f334f61aebed695e2e4193db5e\"\r\n }\r\n ],\r\n```\r\n\r\nI think this list should only have a single entry. (\ud83e\udd14 \u2014\u00a0not sure why it's a list then...)\r\n\r\nWith this it would be possible to build/reconstruct a chain of commits from the history, that I don't **think** is available as yet (unless you know a better way). \r\n\r\nIt is certainly possible to get sequential lists of commits out of git directly, so the same would be possible combining tools, but wondering if a single tool could do it. \r\n\r\nWhat do you think? \r\nThanks! \ud83c\udfc5 \r\n\r\n", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/71/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1211283427, "node_id": "I_kwDODFdgUs5IMrfj", "number": 72, "title": "feature: display progress bar when downloading multi-page responses", "user": {"value": 9020979, "label": "hydrosquall"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-04-21T16:37:12Z", "updated_at": "2022-04-21T17:29:31Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "## Motivation\r\n\r\nFor a long running command (longer than 1 minute) for a big table (like pull requests or commits), it can be tricky to know if the script is still running, or if a rate limit/error was encountered\r\n\r\nWe know how many pages there are, so it may be possible to indicate how many remain.\r\n\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/a6e237f75a4b86963d91dcb5c9582e3a1b3349d6/github_to_sqlite/utils.py#L367\r\n\r\n## Resources\r\n\r\n- Using the existing Click API: \r\n - https://click.palletsprojects.com/en/5.x/utils/#showing-progress-bars\r\n- Loading spinner: https://github.com/pavdmyt/yaspin\r\n- Progress bar: https://github.com/tqdm/tqdm", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/72/reactions\", \"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1261884917, "node_id": "PR_kwDODFdgUs45K1L3", "number": 73, "title": "Fixing 'NoneType' object has no attribute 'items'", "user": {"value": 1224205, "label": "empjustine"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-06-06T13:58:11Z", "updated_at": "2022-07-18T19:40:12Z", "closed_at": "2022-07-18T19:40:12Z", "author_association": "CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/73", "body": "Under some conditions, GitHub caches removed starred repositories and ends up leaving dangling `None` user references.\r\n\r\n Traceback (most recent call last):\r\n File \"/home/dogsheep/dogsheep/github-to-sqlite/bin/github-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py\", line 1657, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/cli.py\", line 181, in starred\r\n utils.save_stars(db, user, stars)\r\n File \"/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py\", line 494, in save_stars\r\n repo_id = save_repo(db, repo)\r\n File \"/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py\", line 308, in save_repo\r\n to_save[\"owner\"] = save_user(db, to_save[\"owner\"])\r\n File \"/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py\", line 229, in save_user\r\n for key, value in user.items()\r\n AttributeError: 'NoneType' object has no attribute 'items'", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/73/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1308461063, "node_id": "I_kwDODFdgUs5N_YgH", "number": 74, "title": "500 error in github-to-sqlite demo", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2022-07-18T19:39:32Z", "updated_at": "2022-07-18T21:16:18Z", "closed_at": "2022-07-18T21:14:22Z", "author_association": "MEMBER", "pull_request": null, "body": "https://github-to-sqlite.dogsheep.net/github/issue_comments throws a 500:\r\n\r\n> `cannot import name 'etree' from 'markdown.util' (/usr/local/lib/python3.8/site-packages/markdown/util.py)`\r\n\r\nhttps://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/metrics?project=datasette-222320 suggests this started happening 3 days ago.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1363244199, "node_id": "I_kwDODFdgUs5RQXSn", "number": 75, "title": "Fetch repos doesn't support organisations", "user": {"value": 2757699, "label": "OverkillGuy"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-09-06T12:55:06Z", "updated_at": "2022-09-06T12:55:06Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Say I want to get all my Github Org's repos info, for data analysis. Not just the public repos, but also the private/internal repos.\r\n\r\nThe endpoints are different for organisation, and this tool doesn't take it into account:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/ace13ec3d98090d99bd71871c286a4a612c96a50/github_to_sqlite/utils.py#L453\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/ace13ec3d98090d99bd71871c286a4a612c96a50/github_to_sqlite/utils.py#L455\r\n\r\nThe endpoints for organisation repos is instead ([source](https://docs.github.com/en/rest/repos/repos#list-organization-repositories)):\r\n`url = \"https://api.github.com/orgs/{}/repos\".format(username)`\r\n\r\nLet's add support for organisations repo scraping.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/75/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1363280254, "node_id": "PR_kwDODFdgUs4-cIa_", "number": 76, "title": "Add organization support to repos command", "user": {"value": 2757699, "label": "OverkillGuy"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-09-06T13:21:42Z", "updated_at": "2022-09-06T13:59:08Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/76", "body": "New --organization flag to signify all given \"usernames\" are private\r\norgs. Adapts API URL to the organization path instead.\r\n\r\nNot the best implementation, but a first draft to talk around\r\n\r\nFixes #75 (badly, no tests, overly vague, untested)", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/76/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1410548368, "node_id": "I_kwDODFdgUs5UE0KQ", "number": 77, "title": "Feature: Support GitHub discussions", "user": {"value": 631242, "label": "frosencrantz"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-10-16T16:53:38Z", "updated_at": "2022-10-16T16:53:38Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Hi @simonw I've been a happy user of this tool. Thank you for writing it and sharing it.\r\n\r\nI wanted to suggest a feature request to support Discussions. For example the VisiData project has discussions https://github.com/saulpw/visidata/discussions , and it would be useful if there was a way to pull that data into the database.\r\n\r\nHowever, I'm not offering a pull request.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/77/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1505411725, "node_id": "I_kwDODFdgUs5ZusKN", "number": 78, "title": "self-hosted or corp github enterprise", "user": {"value": 549431, "label": "ebdavison"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-12-20T22:51:45Z", "updated_at": "2022-12-20T22:51:45Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "We use github enterprise at work and I would like to use this tool to pull info from that site rather than the public github.com instance. Is there an option for this? If not, can one be added for a custom repo URL?", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/78/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1570375808, "node_id": "I_kwDODFdgUs5dmgiA", "number": 79, "title": "Deploy demo job is failing due to rate limit", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-02-03T20:05:01Z", "updated_at": "2023-04-05T21:12:55Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "https://github.com/dogsheep/github-to-sqlite/actions/runs/4080058087/jobs/7032116511", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}