{"id": 672053811, "node_id": "MDExOlB1bGxSZXF1ZXN0NjcyMDUzODEx", "number": 65, "state": "open", "locked": 0, "title": "basic support for events", "user": {"value": 231498, "label": "khimaros"}, "body": "a quick first pass at implementing the feature requested in https://github.com/dogsheep/github-to-sqlite/issues/64\r\n\r\ntesting instructions:\r\n\r\n```\r\n$ github-to-sqlite events events.db user/khimaros\r\n```\r\n\r\nif the specified user is the authenticated user, it will also include private events.\r\n\r\ncaveat: pagination appears to be broken (i don't see `next` in the response JSON from GitHub)", "created_at": "2021-06-17T00:51:30Z", "updated_at": "2022-10-03T22:35:03Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "0a252a06a15e307c8a67b2e0aac0907e2566bf19", "assignee": null, "milestone": null, "draft": 0, "head": "82da9f91deda81d92ec64c9eda960aa64340c169", "base": "0e45b72312a0756e5a562effbba08cb8de1e480b", "author_association": "FIRST_TIME_CONTRIBUTOR", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "url": "https://github.com/dogsheep/github-to-sqlite/pull/65", "merged_by": null, "auto_merge": null}
{"id": 716357982, "node_id": "MDExOlB1bGxSZXF1ZXN0NzE2MzU3OTgy", "number": 66, "state": "open", "locked": 0, "title": "Add --merged-by flag to pull-requests sub command", "user": {"value": 30531572, "label": "sarcasticadmin"}, "body": "## Description\r\n\r\nProposing a solution to the API limitation for `merged_by` in pull_requests. Specifically the following called out in the readme:\r\n\r\n```\r\nNote that the merged_by column on the pull_requests table will only be populated for pull requests that are loaded using the --pull-request option - the GitHub API does not return this field for pull requests that are loaded in bulk.\r\n```\r\n\r\nThis approach might cause larger repos to hit rate limits called out in https://github.com/dogsheep/github-to-sqlite/issues/51 but seems to work well in the repos I tested and included below.\r\n\r\n## Old Behavior\r\n- Had to list out the pull-requests individually via multiple `--pull-request` flags\r\n\r\n## New Behavior\r\n\r\n- `--merged-by` flag for getting 'merge_by' information out of pull-requests without having to specify individual PR numbers.\r\n\r\n# Testing\r\n\r\nPicking some repo that has more than one merger (datasette only has 1 \ud83d\ude09 )\r\n\r\n```\r\n$ github-to-sqlite pull-requests ./github.db opnsense/tools --merged-by\r\n$ echo \"select id, url, merged_by from pull_requests;\" | sqlite3 ./github.db \r\n83533612|https://github.com/opnsense/tools/pull/39|1915288\r\n102632885|https://github.com/opnsense/tools/pull/43|1915288\r\n149114810|https://github.com/opnsense/tools/pull/57|1915288\r\n160394495|https://github.com/opnsense/tools/pull/64|1915288\r\n163308408|https://github.com/opnsense/tools/pull/67|1915288\r\n169723264|https://github.com/opnsense/tools/pull/69|1915288\r\n171381422|https://github.com/opnsense/tools/pull/72|1915288\r\n179938195|https://github.com/opnsense/tools/pull/77|1915288\r\n196233824|https://github.com/opnsense/tools/pull/82|1915288\r\n215289964|https://github.com/opnsense/tools/pull/93|\r\n219696100|https://github.com/opnsense/tools/pull/97|1915288\r\n223664843|https://github.com/opnsense/tools/pull/99|\r\n228446172|https://github.com/opnsense/tools/pull/103|1915288\r\n238930434|https://github.com/opnsense/tools/pull/110|1915288\r\n255507110|https://github.com/opnsense/tools/pull/119|1915288\r\n255980675|https://github.com/opnsense/tools/pull/120|1915288\r\n261906770|https://github.com/opnsense/tools/pull/125|\r\n263800503|https://github.com/opnsense/tools/pull/127|1915288\r\n264038685|https://github.com/opnsense/tools/pull/128|1915288\r\n264696704|https://github.com/opnsense/tools/pull/129|1915288\r\n266660547|https://github.com/opnsense/tools/pull/130|1915288\r\n273120409|https://github.com/opnsense/tools/pull/133|1915288\r\n274370803|https://github.com/opnsense/tools/pull/135|\r\n276600629|https://github.com/opnsense/tools/pull/139|\r\n277303655|https://github.com/opnsense/tools/pull/141|1915288\r\n293033714|https://github.com/opnsense/tools/pull/145|\r\n294827649|https://github.com/opnsense/tools/pull/146|\r\n295140008|https://github.com/opnsense/tools/pull/147|1915288\r\n305690829|https://github.com/opnsense/tools/pull/150|9783985\r\n307077931|https://github.com/opnsense/tools/pull/152|1915288\r\n321782100|https://github.com/opnsense/tools/pull/155|\r\n337265672|https://github.com/opnsense/tools/pull/160|\r\n337267484|https://github.com/opnsense/tools/pull/161|1915288\r\n368251763|https://github.com/opnsense/tools/pull/169|\r\n428262505|https://github.com/opnsense/tools/pull/181|\r\n437557011|https://github.com/opnsense/tools/pull/182|1915288\r\n447079893|https://github.com/opnsense/tools/pull/185|\r\n461822092|https://github.com/opnsense/tools/pull/191|\r\n463290142|https://github.com/opnsense/tools/pull/193|1915288\r\n470112962|https://github.com/opnsense/tools/pull/194|1915288\r\n472644649|https://github.com/opnsense/tools/pull/195|1915288\r\n488696898|https://github.com/opnsense/tools/pull/198|\r\n513289902|https://github.com/opnsense/tools/pull/201|\r\n522530265|https://github.com/opnsense/tools/pull/203|\r\n564443347|https://github.com/opnsense/tools/pull/213|\r\n597579516|https://github.com/opnsense/tools/pull/220|1915288\r\n602860357|https://github.com/opnsense/tools/pull/221|1915288\r\n608744738|https://github.com/opnsense/tools/pull/222|1915288\r\n623279673|https://github.com/opnsense/tools/pull/228|1915288\r\n664656182|https://github.com/opnsense/tools/pull/233|\r\n664781786|https://github.com/opnsense/tools/pull/234|1915288\r\n670683636|https://github.com/opnsense/tools/pull/235|1915288\r\n683150764|https://github.com/opnsense/tools/pull/237|\r\n685016233|https://github.com/opnsense/tools/pull/238|\r\n687099825|https://github.com/opnsense/tools/pull/239|1915288\r\n715705652|https://github.com/opnsense/tools/pull/244|1915288\r\n715721248|https://github.com/opnsense/tools/pull/245|1915288\r\n```\r\n`userid` are now present for those PRs that were merged.\r\n\r\nWithout the flag the `merged_by` behavior remains missing as expected when get PRs bulk:\r\n\r\n```\r\n$ github-to-sqlite pull-requests ./github.db opnsense/tools\r\n$ echo \"select id, url, merged_by from pull_requests;\" | sqlite3 ./github.db \r\n83533612|https://github.com/opnsense/tools/pull/39|\r\n102632885|https://github.com/opnsense/tools/pull/43|\r\n149114810|https://github.com/opnsense/tools/pull/57|\r\n160394495|https://github.com/opnsense/tools/pull/64|\r\n163308408|https://github.com/opnsense/tools/pull/67|\r\n169723264|https://github.com/opnsense/tools/pull/69|\r\n171381422|https://github.com/opnsense/tools/pull/72|\r\n179938195|https://github.com/opnsense/tools/pull/77|\r\n196233824|https://github.com/opnsense/tools/pull/82|\r\n215289964|https://github.com/opnsense/tools/pull/93|\r\n219696100|https://github.com/opnsense/tools/pull/97|\r\n223664843|https://github.com/opnsense/tools/pull/99|\r\n228446172|https://github.com/opnsense/tools/pull/103|\r\n238930434|https://github.com/opnsense/tools/pull/110|\r\n255507110|https://github.com/opnsense/tools/pull/119|\r\n255980675|https://github.com/opnsense/tools/pull/120|\r\n261906770|https://github.com/opnsense/tools/pull/125|\r\n263800503|https://github.com/opnsense/tools/pull/127|\r\n264038685|https://github.com/opnsense/tools/pull/128|\r\n264696704|https://github.com/opnsense/tools/pull/129|\r\n266660547|https://github.com/opnsense/tools/pull/130|\r\n273120409|https://github.com/opnsense/tools/pull/133|\r\n274370803|https://github.com/opnsense/tools/pull/135|\r\n276600629|https://github.com/opnsense/tools/pull/139|\r\n277303655|https://github.com/opnsense/tools/pull/141|\r\n293033714|https://github.com/opnsense/tools/pull/145|\r\n294827649|https://github.com/opnsense/tools/pull/146|\r\n295140008|https://github.com/opnsense/tools/pull/147|\r\n305690829|https://github.com/opnsense/tools/pull/150|\r\n307077931|https://github.com/opnsense/tools/pull/152|\r\n321782100|https://github.com/opnsense/tools/pull/155|\r\n337265672|https://github.com/opnsense/tools/pull/160|\r\n337267484|https://github.com/opnsense/tools/pull/161|\r\n368251763|https://github.com/opnsense/tools/pull/169|\r\n428262505|https://github.com/opnsense/tools/pull/181|\r\n437557011|https://github.com/opnsense/tools/pull/182|\r\n447079893|https://github.com/opnsense/tools/pull/185|\r\n461822092|https://github.com/opnsense/tools/pull/191|\r\n463290142|https://github.com/opnsense/tools/pull/193|\r\n470112962|https://github.com/opnsense/tools/pull/194|\r\n472644649|https://github.com/opnsense/tools/pull/195|\r\n488696898|https://github.com/opnsense/tools/pull/198|\r\n513289902|https://github.com/opnsense/tools/pull/201|\r\n522530265|https://github.com/opnsense/tools/pull/203|\r\n564443347|https://github.com/opnsense/tools/pull/213|\r\n597579516|https://github.com/opnsense/tools/pull/220|\r\n602860357|https://github.com/opnsense/tools/pull/221|\r\n608744738|https://github.com/opnsense/tools/pull/222|\r\n623279673|https://github.com/opnsense/tools/pull/228|\r\n664656182|https://github.com/opnsense/tools/pull/233|\r\n664781786|https://github.com/opnsense/tools/pull/234|\r\n670683636|https://github.com/opnsense/tools/pull/235|\r\n683150764|https://github.com/opnsense/tools/pull/237|\r\n685016233|https://github.com/opnsense/tools/pull/238|\r\n687099825|https://github.com/opnsense/tools/pull/239|\r\n715705652|https://github.com/opnsense/tools/pull/244|\r\n715721248|https://github.com/opnsense/tools/pull/245|\r\n```\r\n\r\nIndividual PRs passed via `--pull-request` flag behaves as expected (unchanged):\r\n\r\n```\r\n$ github-to-sqlite pull-requests ./github.db opnsense/tools --pull-request 39 --pull-request 237\r\n$ echo \"select id, url, merged_by from pull_requests;\" | sqlite3 ./github.db\r\n83533612|https://github.com/opnsense/tools/pull/39|1915288\r\n683150764|https://github.com/opnsense/tools/pull/237|\r\n```\r\n> Picking 1 PR that has a merged_by (39) and one that does not (237)", "created_at": "2021-08-20T00:57:55Z", "updated_at": "2021-09-28T21:50:31Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "6b4276d9469e4579c81588ac9e3d128026d919a0", "assignee": null, "milestone": null, "draft": 0, "head": "a92a31d5d446022baeaf7f3c9ea107094637e64d", "base": "ed3752022e45b890af63996efec804725e95d0d4", "author_association": "FIRST_TIME_CONTRIBUTOR", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "url": "https://github.com/dogsheep/github-to-sqlite/pull/66", "merged_by": null, "auto_merge": null}
{"id": 721686721, "node_id": "MDExOlB1bGxSZXF1ZXN0NzIxNjg2NzIx", "number": 67, "state": "open", "locked": 0, "title": "Replacing step ID key with step_id", "user": {"value": 16374374, "label": "jshcmpbll"}, "body": "Workflows that have an `id` in any step result in the following error when running `workflows`:\r\n\r\ne.g.`github-to-sqlite workflows github.db nixos/nixpkgs`\r\n\r\n```Traceback (most recent call last):\r\n  File \"/usr/local/bin/github-to-sqlite\", line 8, in <module>\r\n    sys.exit(cli())\r\n  File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1137, in __call__\r\n    return self.main(*args, **kwargs)\r\n  File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1062, in main\r\n    rv = self.invoke(ctx)\r\n  File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1668, in invoke```Traceback (most recent call last):\r\n  File \"/usr/local/bin/github-to-sqlite\", line 8, in <module>\r\n    sys.exit(cli())\r\n  File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1137, in __call__\r\n    return self.main(*args, **kwargs)\r\n  File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1062, in main\r\n    rv = self.invoke(ctx)\r\n  File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1668, in invoke\r\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n  File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 1404, in invoke\r\n    return ctx.invoke(self.callback, **ctx.params)\r\n  File \"/usr/local/lib/python3.8/dist-packages/click/core.py\", line 763, in invoke\r\n    return __callback(*args, **kwargs)\r\n  File \"/usr/local/lib/python3.8/dist-packages/github_to_sqlite/cli.py\", line 601, in workflows\r\n    utils.save_workflow(db, repo_id, filename, content)\r\n  File \"/usr/local/lib/python3.8/dist-packages/github_to_sqlite/utils.py\", line 865, in save_workflow\r\n    db[\"steps\"].insert_all(\r\n  File \"/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py\", line 2596, in insert_all\r\n    self.insert_chunk(\r\n  File \"/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py\", line 2378, in insert_chunk\r\n    result = self.db.execute(query, params)\r\n  File \"/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py\", line 419, in execute\r\n    return self.conn.execute(sql, parameters)\r\nsqlite3.IntegrityError: datatype mismatch\r\n```\r\n\r\n - [Information about the ID key in a step for GHA](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepsid)\r\n - [An example workflow from a public repo](https://github.com/NixOS/nixpkgs/blob/b4cc66827745e525ce7bb54659845ac89788a597/.github/workflows/direct-push.yml#L16)\r\n\r\n# Changes\r\nI'm proposing that the key for `id` in step is replaced with `step_id` so that it no longer interferes with the table `id` for tracking the record.\r\n\r\nSpecial thanks to @sarcasticadmin @egiffen and @ruebenramirez for helping a bit on this \ud83d\ude04 ", "created_at": "2021-08-28T01:26:41Z", "updated_at": "2021-08-28T01:27:00Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "9f73c9bf29dec9a1482d9af56b9fac271869585c", "assignee": null, "milestone": null, "draft": 0, "head": "9b5acceb25cf48b00e9c6c8293358b036440deb2", "base": "ed3752022e45b890af63996efec804725e95d0d4", "author_association": "FIRST_TIME_CONTRIBUTOR", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "url": "https://github.com/dogsheep/github-to-sqlite/pull/67", "merged_by": null, "auto_merge": null}
{"id": 747742034, "node_id": "PR_kwDODFdgUs4skaNS", "number": 68, "state": "open", "locked": 0, "title": "Add support for retrieving teams / members", "user": {"value": 68329, "label": "philwills"}, "body": "Adds a method for retrieving all the teams within an organisation and all the members in those teams. The latter is stored as a join table `team_members` beteween `teams` and `users`.", "created_at": "2021-10-01T15:55:02Z", "updated_at": "2021-10-01T15:59:53Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "f46e276c356c893370d5893296f4b69f08baf02c", "assignee": null, "milestone": null, "draft": 0, "head": "cc838e87b1eb19b299f277a07802923104f35ce2", "base": "ed3752022e45b890af63996efec804725e95d0d4", "author_association": "FIRST_TIME_CONTRIBUTOR", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "url": "https://github.com/dogsheep/github-to-sqlite/pull/68", "merged_by": null, "auto_merge": null}
{"id": 862538586, "node_id": "PR_kwDODFdgUs4zaUta", "number": 70, "state": "open", "locked": 0, "title": "scrape-dependents: enable paging through package menu option if present", "user": {"value": 36061055, "label": "stanbiryukov"}, "body": "Some repos organize network dependents by a Package toggle. This PR adds the ability to page through those options and scrape underlying dependents.", "created_at": "2022-02-24T15:07:25Z", "updated_at": "2022-02-24T15:07:25Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "36cca3584a07d88d1e505111d1b23294d66ba73e", "assignee": null, "milestone": null, "draft": 0, "head": "cc8f276a474525e55ed0bcacb0cd8cc560f89614", "base": "751bc900366ca52e662ea383b858cbf4365093d9", "author_association": "FIRST_TIME_CONTRIBUTOR", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "url": "https://github.com/dogsheep/github-to-sqlite/pull/70", "merged_by": null, "auto_merge": null}
{"id": 1047561919, "node_id": "PR_kwDODFdgUs4-cIa_", "number": 76, "state": "open", "locked": 0, "title": "Add organization support to repos command", "user": {"value": 2757699, "label": "OverkillGuy"}, "body": "New --organization flag to signify all given \"usernames\" are private\r\norgs. Adapts API URL to the organization path instead.\r\n\r\nNot the best implementation, but a first draft to talk around\r\n\r\nFixes #75 (badly, no tests, overly vague, untested)", "created_at": "2022-09-06T13:21:42Z", "updated_at": "2022-09-06T13:59:08Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "1514acfa87f57261547bc3d7fc4f161e34285d76", "assignee": null, "milestone": null, "draft": 0, "head": "bb959b46e8a7647755c14dee180fdd5209451954", "base": "ace13ec3d98090d99bd71871c286a4a612c96a50", "author_association": "FIRST_TIME_CONTRIBUTOR", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "url": "https://github.com/dogsheep/github-to-sqlite/pull/76", "merged_by": null, "auto_merge": null}