issue_comments
9,947 rows sorted by issue_url
This data as json, CSV (advanced)
id | html_url | issue_url ▼ | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
623044643 | https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623044643 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38 | MDEyOklzc3VlQ29tbWVudDYyMzA0NDY0Mw== | zzeleznick 5779832 | 2020-05-03T02:34:32Z | 2020-05-03T02:34:32Z | NONE | 1. More than glad to share feedback from the sidelines as a [starrer](https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++starred_at%2C%0D%0A++starred_by%2C%0D%0A++full_name+as+repo_name%0D%0Afrom%0D%0A++repos_starred%0D%0Awhere%0D%0A++starred_by+%3D+%22zzeleznick%22%0D%0Aorder+by%0D%0A++starred_at+desc). ``` -- Motivation: -- Datasette is a data hammer and I'm looking for nails -- e.g. Find which repos a user has starred => trigger a TBD downstream action select starred_at, starred_by, full_name as repo_name from repos_starred where starred_by = "zzeleznick" order by starred_at desc ``` | starred_at | starred_by | repo_name | | --- | --- | --- | | 2020-02-11T01:08:59Z | zzeleznick | dogsheep/twitter-to-sqlite | | 2020-01-11T21:57:34Z | zzeleznick | simonw/datasette | 2. In my day job, I use [airflow](https://github.com/apache/airflow), and that's the mental model I'm bringing to [datasette](https://github.com/simonw/datasette). 3. I see your project like [twitter-to-sqlite](https://github.com/dogsheep/twitter-to-sqlite) akin to [Operators](https://airflow.apache.org/docs/stable/_api/index.html#pythonapi-operators) in Airflow world. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | [Feature Request] Support Repo Name in Search 🥺 611284481 | |
675509550 | https://github.com/dogsheep/github-to-sqlite/issues/39#issuecomment-675509550 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDY3NTUwOTU1MA== | simonw 9599 | 2020-08-18T14:23:56Z | 2020-08-18T14:23:56Z | MEMBER | I think this is fixed: https://github-to-sqlite.dogsheep.net/github/issues?_facet=repo | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | issues foreign key to repo isn't working 613777056 | |
531517138 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-531517138 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDUzMTUxNzEzOA== | simonw 9599 | 2019-09-14T21:59:59Z | 2019-09-14T21:59:59Z | MEMBER | Paginate through https://api.github.com/repos/simonw/datasette/stargazers Send `Accept: application/vnd.github.v3.star+json` to get the `starred_at` dates. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
550388354 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-550388354 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDU1MDM4ODM1NA== | simonw 9599 | 2019-11-06T16:26:55Z | 2019-11-06T16:26:55Z | MEMBER | Here's a query I figured out using a window function that shows cumulative stargazers over time: ```sql select yyyymmdd, sum(n) over ( order by yyyymmdd rows unbounded preceding ) as cumulative_count from ( select substr(starred_at, 0, 11) as yyyymmdd, count(*) as n from stars group by yyyymmdd ) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
622990947 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-622990947 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDYyMjk5MDk0Nw== | simonw 9599 | 2020-05-02T17:54:16Z | 2020-05-02T17:54:16Z | MEMBER | I could add that window function query as a view, but only if the detected version of SQLite supports window functions. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
623000814 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623000814 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDYyMzAwMDgxNA== | simonw 9599 | 2020-05-02T19:15:23Z | 2020-05-02T19:15:23Z | MEMBER | I'm not going to do the `--sql` bit just yet. I have patterns for working around this for other commands which are working fine: https://github.com/dogsheep/github-to-sqlite/blob/d00a53061556dc403c166b443d141c4e1adbd64a/.github/workflows/deploy-demo.yml#L53-L70 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
623004836 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623004836 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDYyMzAwNDgzNg== | simonw 9599 | 2020-05-02T19:49:51Z | 2020-05-02T19:49:51Z | MEMBER | Alternative pattern: ``` sqlite-utils releases.db 'select full_name from repos' --csv --no-headers \ | tr -d '\r' \ | xargs github-to-sqlite stargazers stars.db ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
623006154 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623006154 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDYyMzAwNjE1NA== | simonw 9599 | 2020-05-02T20:01:39Z | 2020-05-02T20:01:54Z | MEMBER | Needs tests and documentation. I shipped it early to check that the live demo works. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
623007441 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623007441 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDYyMzAwNzQ0MQ== | simonw 9599 | 2020-05-02T20:13:37Z | 2020-05-02T20:13:37Z | MEMBER | Datasette cumulative stars over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
623010272 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623010272 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDYyMzAxMDI3Mg== | simonw 9599 | 2020-05-02T20:39:14Z | 2020-05-02T20:39:14Z | MEMBER | Graph of cumulative stars for Datasette over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29#g.mark=line&g.x_column=yyyymmdd&g.x_type=temporal&g.y_column=cumulative_count&g.y_type=quantitative <img width="291" alt="github__select_yyyymmdd__sum_n__over___order_by_yyyymmdd_rows_unbounded_preceding___as_cumulative_count_from___select_substr_starred_at__0__11__as_yyyymmdd__count____as_n_from_stars_where_repo___107914493_group_by_yyyymmdd__" src="https://user-images.githubusercontent.com/9599/80891642-31587180-8c7a-11ea-9169-60eb3cb6d3e0.png"> Stars per day (as a label bar chart, so very wide): https://github-to-sqlite.dogsheep.net/github?sql=%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++#g.mark=bar&g.x_column=yyyymmdd&g.x_type=ordinal&g.y_column=n&g.y_type=quantitative <img width="1424" alt="github__select_substr_starred_at__0__11__as_yyyymmdd__count____as_n_from_stars_where_repo___107914493_group_by_yyyymmdd" src="https://user-images.githubusercontent.com/9599/80891656-4e8d4000-8c7a-11ea-9ff2-02c9cf4cb5d5.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
643393506 | https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643393506 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDY0MzM5MzUwNg== | simonw 9599 | 2020-06-12T17:21:14Z | 2020-06-12T17:21:14Z | MEMBER | I only install SQLite for this: https://github.com/dogsheep/github-to-sqlite/blob/c0d54e0260468be38152293df5abd775c068495d/.github/workflows/deploy-demo.yml#L77-L78 I'm going to remove the need to install sqlite3 by making this possible with sqlite-utils: https://github.com/simonw/sqlite-utils/issues/115 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo deploy is broken 637899539 | |
643414646 | https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643414646 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDY0MzQxNDY0Ng== | simonw 9599 | 2020-06-12T18:06:48Z | 2020-06-12T18:06:48Z | MEMBER | That fixed it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo deploy is broken 637899539 | |
653947916 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653947916 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk0NzkxNg== | simonw 9599 | 2020-07-05T22:40:47Z | 2020-07-05T22:40:47Z | MEMBER | Might be that it's not got enough RAM. I'll try deploying to a larger instance. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653960989 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653960989 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MDk4OQ== | simonw 9599 | 2020-07-06T00:32:34Z | 2020-07-06T00:32:34Z | MEMBER | Same error. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653962418 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962418 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjQxOA== | simonw 9599 | 2020-07-06T00:41:38Z | 2020-07-06T00:41:38Z | MEMBER | https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/logs?project=datasette-222320 has some clues. <img width="818" alt="github-to-sqlite_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/86545887-92086280-bee6-11ea-8787-b3b85a7bb4b9.png"> <img width="1451" alt="github-to-sqlite_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/86545917-c11ed400-bee6-11ea-9cf8-957911ca9f6a.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653962530 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962530 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjUzMA== | simonw 9599 | 2020-07-06T00:42:13Z | 2020-07-06T00:42:13Z | MEMBER | So it looks like it's the ASGI lifespan change I made in https://github.com/simonw/datasette/commit/16f592247a2a0e140ada487e9972645406dcae69 - It must be incompatible with one of the plugins. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653962669 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962669 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjY2OQ== | simonw 9599 | 2020-07-06T00:42:57Z | 2020-07-06T00:42:57Z | MEMBER | https://github-to-sqlite.dogsheep.net/-/plugins ```json [ { "name": "datasette-json-html", "static": false, "templates": false, "version": "0.6", "hooks": [ "prepare_connection", "render_cell" ] }, { "name": "datasette-render-markdown", "static": false, "templates": false, "version": "1.1.2", "hooks": [ "extra_template_vars", "render_cell" ] }, { "name": "datasette-pretty-json", "static": false, "templates": false, "version": "0.2", "hooks": [ "render_cell" ] }, { "name": "datasette-search-all", "static": false, "templates": true, "version": "0.2.1", "hooks": [ "asgi_wrapper", "extra_template_vars" ] }, { "name": "datasette-vega", "static": true, "templates": false, "version": "0.6.2", "hooks": [ "extra_css_urls", "extra_js_urls" ] } ] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653962708 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962708 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjcwOA== | simonw 9599 | 2020-07-06T00:43:10Z | 2020-07-06T00:43:10Z | MEMBER | I bet it's datasette-search-all. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653966670 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653966670 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2NjY3MA== | simonw 9599 | 2020-07-06T01:07:02Z | 2020-07-06T01:07:02Z | MEMBER | OK that fix worked.https://github.com/dogsheep/github-to-sqlite/runs/839764768?check_suite_focus=true | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
660536265 | https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660536265 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDY2MDUzNjI2NQ== | simonw 9599 | 2020-07-18T20:15:12Z | 2020-07-18T20:15:12Z | MEMBER | I want to create a SQL query which shows me all of my repositories that have commits that are NOT in the most recent release. The releases table doesn't have enough information for this because it doesn't tell you the commit hash associated with each release, just the tag: https://github-to-sqlite.dogsheep.net/github/releases | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite tags command for fetching tags 660355904 | |
660547502 | https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660547502 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDY2MDU0NzUwMg== | simonw 9599 | 2020-07-18T21:50:37Z | 2020-07-18T21:50:37Z | MEMBER | ``` $ github-to-sqlite tags tags.db simonw/datasette dogsheep/github-to-sqlite $ sqlite-utils tables tags.db --counts [{"table": "users", "count": 2}, {"table": "licenses", "count": 1}, {"table": "repos", "count": 2}, {"table": "tags", "count": 76}, {"table": "licenses_fts", "count": 1}, {"table": "licenses_fts_data", "count": 3}, {"table": "licenses_fts_idx", "count": 1}, {"table": "licenses_fts_docsize", "count": 1}, {"table": "licenses_fts_config", "count": 1}, {"table": "repos_fts", "count": 2}, {"table": "repos_fts_data", "count": 3}, {"table": "repos_fts_idx", "count": 1}, {"table": "repos_fts_docsize", "count": 2}, {"table": "repos_fts_config", "count": 1}, {"table": "users_fts", "count": 2}, {"table": "users_fts_data", "count": 3}, {"table": "users_fts_idx", "count": 1}, {"table": "users_fts_docsize", "count": 2}, {"table": "users_fts_config", "count": 1}] $ sqlite-utils rows tags.db tags [{"repo_id": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"}, {"repo_id": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"}, {"repo_id": 107914493, "name": "0.45a4", "sha": "265483173bc8341dc02c8b782b9b59d2ce8bbedb"}, {"repo_id": 107914493, "name": "0.45a3", "sha": "1f55a4a2b68fa65e56a28baeb7f44122fdeca7e7"}, {"repo_id": 107914493, "name": "0.45a2", "sha": "1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f"}, {"repo_id": 107914493, "name": "0.45a1", "sha": "b59b92b1b0517cf18fa748ff9d0a0bf86298dd43"}, {"repo_id": 107914493, "name": "0.45a0", "sha": "dda932d818b34ccab11730a76554f0a3748d8348"}, {"repo_id": 107914493, "name": "0.44", "sha": "b906030235efbdff536405d66078f4868ce0d3bd"}, {"repo_id": 107914493, "name": "0.43", "sha": "d56f402822df102f9cf1a9a056449d01a15e3aae"}, {"repo_id": 107914493, "name": "0.42", "sha": "af6c6c5d6f929f951c0e63bfd1c82e37a071b50f"}, {"repo_id": 107914493, "name": "0.41", "sha": "182e5c8745c94576718315f7596ccc81e5e2417b"}, {"repo_id": 107914493, "name": "0.40", "sha": "8da108193b08ab… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite tags command for fetching tags 660355904 | |
660548780 | https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660548780 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDY2MDU0ODc4MA== | simonw 9599 | 2020-07-18T22:02:37Z | 2020-07-18T23:05:56Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/tags?_facet=repo | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite tags command for fetching tags 660355904 | |
660551397 | https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660551397 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDY2MDU1MTM5Nw== | simonw 9599 | 2020-07-18T22:27:32Z | 2020-07-18T23:05:45Z | MEMBER | ```sql with most_recent_releases as ( with ranked as ( select repo, tag_name, published_at, row_number() OVER ( partition BY repo ORDER BY published_at DESC ) rank FROM releases ) select * from ranked where rank = 1 ) select repos.full_name as repo, most_recent_releases.tag_name as release, commits.committer_date as release_commit_date, ( select count(*) from commits c2 where c2.repo = repos.id and c2.committer_date > commits.committer_date ) as commits_since_release, 'https://github.com/' || repos.full_name || '/compare/' || most_recent_releases.tag_name || '...' || repos.default_branch as view_commits from most_recent_releases join repos on most_recent_releases.repo = repos.id join tags on tags.repo = repos.id and tags.name = most_recent_releases.tag_name join commits on tags.sha = commits.sha order by commits_since_release desc ``` repo | release | release_commit_date | commits_since_release | view_commits -- | -- | -- | -- | -- simonw/datasette | 0.45 | 2020-07-01T21:43:07Z | 9 | https://github.com/simonw/datasette/compare/0.45...master dogsheep/twitter-to-sqlite | 0.21.1 | 2020-04-30T18:20:43Z | 2 | https://github.com/dogsheep/twitter-to-sqlite/compare/0.21.1...master dogsheep/github-to-sqlite | 2.3 | 2020-07-09T23:26:34Z | 2 | https://github.com/dogsheep/github-to-sqlite/compare/2.3...master dogsheep/dogsheep-photos | 0.4.1 | 2020-05-25T20:11:20Z | 2 | https://github.com/dogsheep/dogsheep-photos/compare/0.4.1...master dogsheep/swarm-to-sqlite | 0.3.1 | 2020-03-28T02:29:41Z | 1 | https://github.com/dogsheep/swarm-to-sqlite/compare/0.3.1...master dogsheep/hacker-news-to-sqlite | 0.3.1 | 2020-03-21T22:39:34Z | 1 | https://github.com/dogsheep/hacker-news-to-sqlite/compare/0.3.1...master simonw/sqlite-utils | 2.11 | 2020-07-08T17:36:07Z | 0 | https://github.com/simonw/sqlite-utils/compare/2.11...master dog… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite tags command for fetching tags 660355904 | |
660553646 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553646 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1MzY0Ng== | simonw 9599 | 2020-07-18T22:51:41Z | 2020-07-18T22:51:41Z | MEMBER | I could fix this by putting `REFRESH_DB` in a commit message: https://github.com/dogsheep/github-to-sqlite/blob/4ae4aa6f172344b19ff3513707195ee6d2654bd4/.github/workflows/deploy-demo.yml#L41-L46 But... doing so would lose the data I've collected in https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc concerning the first time each dependent repo was spotted. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660553711 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553711 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1MzcxMQ== | simonw 9599 | 2020-07-18T22:52:16Z | 2020-07-18T22:52:16Z | MEMBER | I think the best fix is to download the `github.db` database, manually fix it and then manually deploy it to Cloud Run from my laptop. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660554162 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554162 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1NDE2Mg== | simonw 9599 | 2020-07-18T22:56:58Z | 2020-07-18T22:56:58Z | MEMBER | Manually fixing the database: ``` $ wget 'https://github-to-sqlite.dogsheep.net/github.db' --2020-07-18 15:52:33-- https://github-to-sqlite.dogsheep.net/github.db Resolving github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)... 172.217.5.115 Connecting to github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)|172.217.5.115|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 14626816 (14M) [application/octet-stream] Saving to: ‘github.db’ github.db 100%[============================================================================>] 13.95M 1.22MB/s in 18s 2020-07-18 15:52:53 (773 KB/s) - ‘github.db’ saved [14626816/14626816] $ sqlite3 github.db SQLite version 3.28.0 2019-04-15 14:49:49 Enter ".help" for usage hints. sqlite> drop table tags; sqlite> ^D $ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils dogsheep/healthkit-to-sqlite dogsheep/swarm-to-sqlite dogsheep/twitter-to-sqlite dogsheep/inaturalist-to-sqlite dogsheep/google-takeout-to-sqlite dogsheep/github-to-sqlite dogsheep/genome-to-sqlite dogsheep/pocket-to-sqlite dogsheep/hacker-news-to-sqlite dogsheep/dogsheep-photos $ sqlite-utils tables github.db --counts [{"table": "users", "count": 4048}, {"table": "repos", "count": 210}, ... {"table": "stars", "count": 4140}, {"table": "tags", "count": 188}] $ sqlite-utils rows github.db tags [{"repo": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"}, {"repo": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"}, ... ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660554299 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554299 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1NDI5OQ== | simonw 9599 | 2020-07-18T22:58:24Z | 2020-07-18T23:02:52Z | MEMBER | Deploying the fixed version like this: ``` $ gcloud config set run/region us-central1 $ gcloud config set project datasette-222320 $ datasette publish cloudrun /tmp/github.db \ -m demo-metadata.json \ --service github-to-sqlite \ --install=py-gfm \ --install='datasette-search-all>=0.3' \ --install='datasette-render-markdown>=1.1.2' \ --install=datasette-pretty-json \ --install=datasette-json-html \ --install=datasette-vega ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660554811 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554811 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1NDgxMQ== | simonw 9599 | 2020-07-18T23:03:13Z | 2020-07-18T23:03:13Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/tags now shows a `repo` column instead of a `repo_id` column. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
675259273 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-675259273 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDY3NTI1OTI3Mw== | simonw 9599 | 2020-08-18T05:28:32Z | 2020-08-18T05:28:32Z | MEMBER | Oh that's interesting - i didn't realize "reviews" were a separate concept. I'd definitely accept a pull request adding those! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: pull request reviews and comments 664485022 | |
735482187 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482187 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDczNTQ4MjE4Nw== | simonw 9599 | 2020-11-30T00:20:11Z | 2020-11-30T00:20:11Z | MEMBER | Pull request are now added, thanks to @adamjonas. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: pull request reviews and comments 664485022 | |
735482546 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482546 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDczNTQ4MjU0Ng== | simonw 9599 | 2020-11-30T00:22:02Z | 2020-11-30T00:22:02Z | MEMBER | As for reviews... here's the output of `github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews --accept 'application/vnd.github.v3+json'` ```json [ { "id": 503368921, "node_id": "MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTAzMzY4OTIx", "user": { "login": "simonw", "id": 9599, "node_id": "MDQ6VXNlcjk1OTk=", "avatar_url": "https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4", "gravatar_id": "", "url": "https://api.github.com/users/simonw", "html_url": "https://github.com/simonw", "followers_url": "https://api.github.com/users/simonw/followers", "following_url": "https://api.github.com/users/simonw/following{/other_user}", "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}", "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/simonw/subscriptions", "organizations_url": "https://api.github.com/users/simonw/orgs", "repos_url": "https://api.github.com/users/simonw/repos", "events_url": "https://api.github.com/users/simonw/events{/privacy}", "received_events_url": "https://api.github.com/users/simonw/received_events", "type": "User", "site_admin": false }, "body": "", "state": "CHANGES_REQUESTED", "html_url": "https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-503368921", "pull_request_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48", "author_association": "MEMBER", "_links": { "html": { "href": "https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-503368921" }, "pull_request": { "href": "https://api.github.c… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: pull request reviews and comments 664485022 | |
735483604 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483604 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDczNTQ4MzYwNA== | simonw 9599 | 2020-11-30T00:26:50Z | 2020-11-30T00:26:50Z | MEMBER | It seems like there's a lot missing from that - those aren't particularly interesting given the data that is returned. From the docs at https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#reviews it looks like each review consists of multiple comments, and the comments are where the useful material is - https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#list-comments-for-a-pull-request-review `github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews/503368921/comments --accept 'application/vnd.github.v3+json'` ```json [ { "id": 500603838, "node_id": "MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwMzgzOA==", "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500603838", "pull_request_review_id": 503368921, "diff_hunk": "@@ -0,0 +1,370 @@\n+[\n+ {\n+ \"url\": \"https://api.github.com/repos/simonw/datasette/pulls/571\",\n+ \"id\": 313384926,\n+ \"node_id\": \"MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2\",\n+ \"html_url\": \"https://github.com/simonw/datasette/pull/571\",\n+ \"diff_url\": \"https://github.com/simonw/datasette/pull/571.diff\",\n+ \"patch_url\": \"https://github.com/simonw/datasette/pull/571.patch\",\n+ \"issue_url\": \"https://api.github.com/repos/simonw/datasette/issues/571\",\n+ \"number\": 571,\n+ \"state\": \"closed\",\n+ \"locked\": false,\n+ \"title\": \"detect_fts now works with alternative table escaping\",\n+ \"user\": {\n+ \"login\": \"simonw\",\n+ \"id\": 9599,\n+ \"node_id\": \"MDQ6VXNlcjk1OTk=\",\n+ \"avatar_url\": \"https://avatars0.githubusercontent.com/u/9599?v=4\",\n+ \"gravatar_id\": \"\",\n+ \"url\": \"https://api.github.com/users/simonw\",\n+ \"html_url\": \"https://github.com/simonw\",\n+ \"followers_url\": \"https://api.github.com/users/simonw/followers\",\n+ \"following_ur… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: pull request reviews and comments 664485022 | |
735483820 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483820 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDczNTQ4MzgyMA== | simonw 9599 | 2020-11-30T00:27:47Z | 2020-11-30T00:27:47Z | MEMBER | So it looks like anything that pulls reviews needs to pull each review, then for each one pull the comments. I'm going to consider this blocked on smarter rate limit handling in #51. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: pull request reviews and comments 664485022 | |
1359468823 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-1359468823 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | IC_kwDODFdgUs5RB9kX | choldgraf 1839645 | 2022-12-20T14:39:39Z | 2022-12-20T14:40:15Z | NONE | Just a quick +1 to this one from me - I would like to do a better job of tracking who is reviewing one another's pull requests in repositories, since this is a specific kind of maintenance work that I think often goes unrewarded. I can't seem to figure this out just by looking at the `pull_request` or `issue_comments` tables, so I think it would be helpful to support PR reviews natively (even if just for summary statistics). Alternatively if there is a way in the API to tell if an issue comment is part of a review, then perhaps you could quickly calculate the number of unique reviews that an author performed. But that was beyond my SQL-foo :-) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: pull request reviews and comments 664485022 | |
675523053 | https://github.com/dogsheep/github-to-sqlite/issues/47#issuecomment-675523053 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/47 | MDEyOklzc3VlQ29tbWVudDY3NTUyMzA1Mw== | simonw 9599 | 2020-08-18T14:45:53Z | 2020-08-18T14:45:53Z | MEMBER | ``` % github-to-sqlite emojis emojis.db --fetch [########----------------------------] 397/1682 23% 00:03:43 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | emojis command 681086659 | |
704503719 | https://github.com/dogsheep/github-to-sqlite/pull/48#issuecomment-704503719 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48 | MDEyOklzc3VlQ29tbWVudDcwNDUwMzcxOQ== | adamjonas 755825 | 2020-10-06T19:26:59Z | 2020-10-06T19:26:59Z | CONTRIBUTOR | ref #46 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add pull requests 681228542 | |
704553385 | https://github.com/dogsheep/github-to-sqlite/pull/48#issuecomment-704553385 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48 | MDEyOklzc3VlQ29tbWVudDcwNDU1MzM4NQ== | simonw 9599 | 2020-10-06T21:07:44Z | 2020-10-06T21:07:44Z | MEMBER | Sorry for not looking at this sooner, trying it out now - pull request looks great! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add pull requests 681228542 | |
549094229 | https://github.com/dogsheep/github-to-sqlite/issues/5#issuecomment-549094229 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDU0OTA5NDIyOQ== | simonw 9599 | 2019-11-03T00:44:03Z | 2019-11-03T00:44:03Z | MEMBER | Might not need an incomplete boolean - may be possible to handle this with `alter=True` and then by filtering for users with null values in certain columns. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add "incomplete" boolean to users table for incomplete profiles 493671014 | |
602806530 | https://github.com/dogsheep/github-to-sqlite/issues/5#issuecomment-602806530 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDYwMjgwNjUzMA== | simonw 9599 | 2020-03-23T19:23:30Z | 2020-03-23T19:23:30Z | MEMBER | Yeah `alter=True` means that this isn't necessary. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add "incomplete" boolean to users table for incomplete profiles 493671014 | |
693773191 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693773191 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDY5Mzc3MzE5MQ== | simonw 9599 | 2020-09-17T02:39:26Z | 2020-09-17T02:39:26Z | MEMBER | I'm going to start with `github-to-sqlite get` and `github-to-sqlite post` - I may add `put` and suchlike later on. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
693775622 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693775622 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDY5Mzc3NTYyMg== | simonw 9599 | 2020-09-17T02:48:34Z | 2020-09-17T02:48:34Z | MEMBER | I'd like a `--paginate` option that does the same thing as https://github.com/simonw/paginate-json | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
693788032 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693788032 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDY5Mzc4ODAzMg== | simonw 9599 | 2020-09-17T03:35:22Z | 2020-09-17T03:35:22Z | MEMBER | Documentation: https://github.com/dogsheep/github-to-sqlite/blob/b02bf135485c0a7a3768868967f45a6b5e515289/README.md#making-authenticated-api-calls | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
693788387 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693788387 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDY5Mzc4ODM4Nw== | simonw 9599 | 2020-09-17T03:36:47Z | 2020-09-17T03:36:58Z | MEMBER | Fun demo of the `--nl` option: github-to-sqlite get /users/simonw/repos --paginate --nl | sqlite-utils insert simonw.db repos - --nl | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
693789129 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693789129 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDY5Mzc4OTEyOQ== | simonw 9599 | 2020-09-17T03:40:01Z | 2020-09-17T03:40:01Z | MEMBER | Bug with endpoints that return dictionaries rather than arrays: ``` github-to-sqlite get /users/simonw [ "login", "id", "node_id", "avatar_url", "gravatar_id", "url", "html_url", "followers_url", "following_url", "gists_url", "starred_url", "subscriptions_url", "organizations_url", "repos_url", "events_url", "received_events_url", "type", "site_admin", "name", "company", "blog", "location", "email", "hireable", "bio", "twitter_username", "public_repos", "public_gists", "followers", "following", "created_at", "updated_at" ] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
693794700 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693794700 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDY5Mzc5NDcwMA== | simonw 9599 | 2020-09-17T04:02:39Z | 2020-09-17T04:02:39Z | MEMBER | It would be useful if you could pass an `--accept` option to this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
711569063 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-711569063 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDcxMTU2OTA2Mw== | simonw 9599 | 2020-10-19T05:01:29Z | 2020-10-19T05:01:29Z | MEMBER | Demo of `--accept`: github-to-sqlite get /repos/simonw/datasette/readme --accept 'application/vnd.github.VERSION.html' | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
735484186 | https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-735484186 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51 | MDEyOklzc3VlQ29tbWVudDczNTQ4NDE4Ng== | simonw 9599 | 2020-11-30T00:29:31Z | 2020-11-30T00:29:31Z | MEMBER | This just caused a failure in deploying the demo: https://github.com/dogsheep/github-to-sqlite/runs/1471304407?check_suite_focus=true ``` File "/opt/hostedtoolcache/Python/3.8.6/x64/bin/github-to-sqlite", line 33, in <module> sys.exit(load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')()) File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 142, in issue_comments for comment in utils.fetch_issue_comments(repo, token, issue): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 380, in fetch_issue_comments for comments in paginate(url, headers): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 472, in paginate raise GitHubError.from_response(response) github_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403) Error: Process completed with exit code 1. ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite should handle rate limits better 703246031 | |
770150526 | https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-770150526 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51 | MDEyOklzc3VlQ29tbWVudDc3MDE1MDUyNg== | daniel-butler 22578954 | 2021-01-30T03:44:19Z | 2021-01-30T03:47:24Z | CONTRIBUTOR | I don't have much experience with github's rate limiting. In my day job we use the [tenacity library](https://github.com/jd/tenacity) to handle http errors we get. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite should handle rate limits better 703246031 | |
1208757153 | https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-1208757153 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51 | IC_kwDODFdgUs5IDCuh | hydrosquall 9020979 | 2022-08-09T00:29:44Z | 2022-08-09T00:29:44Z | NONE | I've been looking into how to to get this data out of Github (especially now there are "secondary rate limits" without an advertised allowance separate from the regular rate limits. I've had decent success with the Airbyte github extractor (aside from one data quality issue https://github.com/airbytehq/airbyte/pull/15420 ). Airbyte splits data extraction between the GraphQL and REST endpoints depending on the resource type, but they're very comprehensive. https://github.com/airbytehq/airbyte/blob/306a75ef5370728e0912cf52a1a898a530db0c90/airbyte-integrations/connectors/source-github/source_github/streams.py#L22-L122 Before this, I tried a few solutions in my own custom wrapper mentioned in this thread + its children https://github.com/PyGithub/PyGithub/issues/1989 , but they weren't working as expected. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite should handle rate limits better 703246031 | |
1279224780 | https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-1279224780 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51 | IC_kwDODFdgUs5MP2vM | chapmanjacobd 7908073 | 2022-10-14T16:34:07Z | 2022-10-14T16:34:07Z | NONE | also, it says that authenticated requests have a much higher "rate limit". Unauthenticated requests only get 60 req/hour ?? seems more like a quota than a "rate limit" (although I guess that is semantic equivalence) You would want to use `x-ratelimit-reset` ``` time.sleep(r['x-ratelimit-reset'] + 1 - time.time()) ``` But a more complete solution would bring authenticated requests to the other subcommands. I'm surprised only `github-to-sqlite get` is using the `--auth=` CLI flag | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite should handle rate limits better 703246031 | |
735485677 | https://github.com/dogsheep/github-to-sqlite/issues/53#issuecomment-735485677 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/53 | MDEyOklzc3VlQ29tbWVudDczNTQ4NTY3Nw== | simonw 9599 | 2020-11-30T00:36:09Z | 2020-11-30T00:36:09Z | MEMBER | Given rate limits (see #51) this command might be better implemented by running a `git clone` into a temporary directory - doing so would retrieve all of the files in one go. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for fetching file contents 753000405 | |
735464438 | https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735464438 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54 | MDEyOklzc3VlQ29tbWVudDczNTQ2NDQzOA== | simonw 9599 | 2020-11-29T21:57:08Z | 2020-11-29T21:57:08Z | MEMBER | Inspired by this tweet from Michael Heap https://twitter.com/mheap/status/1333108608817631238 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite workflows command 753026003 | |
735464493 | https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735464493 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54 | MDEyOklzc3VlQ29tbWVudDczNTQ2NDQ5Mw== | simonw 9599 | 2020-11-29T21:57:32Z | 2020-11-29T21:57:32Z | MEMBER | `$ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite workflows command 753026003 | |
735465708 | https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735465708 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54 | MDEyOklzc3VlQ29tbWVudDczNTQ2NTcwOA== | simonw 9599 | 2020-11-29T22:08:46Z | 2020-11-29T22:08:46Z | MEMBER | Demo: - https://github-to-sqlite.dogsheep.net/github/steps?_facet=repo - https://github-to-sqlite.dogsheep.net/github/workflows - https://github-to-sqlite.dogsheep.net/github/jobs | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite workflows command 753026003 | |
746734412 | https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746734412 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58 | MDEyOklzc3VlQ29tbWVudDc0NjczNDQxMg== | simonw 9599 | 2020-12-16T17:58:56Z | 2020-12-16T17:58:56Z | MEMBER | I'm going to rewrite those `<a href="#filtering-tables">` links to `<a href="#user-content-filtering-tables">` - but only if a corresponding `id="user-content-filtering-tables"` element exists. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Readme HTML has broken internal links 769150394 | |
746735889 | https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746735889 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58 | MDEyOklzc3VlQ29tbWVudDc0NjczNTg4OQ== | simonw 9599 | 2020-12-16T17:59:50Z | 2020-12-16T17:59:50Z | MEMBER | I don't want to add a full HTML parser (like BeautifulSoup) as a dependency for this feature. Since the HTML comes from a single, trusted source (GitHub) I could probably handle this using [regular expressions](https://stackoverflow.com/a/1732454). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Readme HTML has broken internal links 769150394 | |
751375487 | https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-751375487 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59 | MDEyOklzc3VlQ29tbWVudDc1MTM3NTQ4Nw== | frosencrantz 631242 | 2020-12-26T17:08:44Z | 2020-12-26T17:08:44Z | CONTRIBUTOR | Hi @simonw, do I need to do anything else for this PR to be considered to be included? I've tried using this project and it is quite nice to be able to explore a repository, but noticed that a couple commands don't allow you to use authorization from the environment variable. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove unneeded exists=True for -a/--auth flag. 771872303 | |
844250232 | https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-844250232 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59 | MDEyOklzc3VlQ29tbWVudDg0NDI1MDIzMg== | simonw 9599 | 2021-05-19T16:08:10Z | 2021-05-19T16:08:10Z | MEMBER | Thanks for catching this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove unneeded exists=True for -a/--auth flag. 771872303 | |
846413174 | https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-846413174 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59 | MDEyOklzc3VlQ29tbWVudDg0NjQxMzE3NA== | frosencrantz 631242 | 2021-05-22T14:06:19Z | 2021-05-22T14:06:19Z | CONTRIBUTOR | Thanks Simon! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove unneeded exists=True for -a/--auth flag. 771872303 | |
541387822 | https://github.com/dogsheep/github-to-sqlite/issues/6#issuecomment-541387822 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDU0MTM4NzgyMg== | simonw 9599 | 2019-10-13T05:27:39Z | 2019-10-13T05:27:39Z | MEMBER | This should be fixed by https://github.com/dogsheep/github-to-sqlite/commit/552543a74970f8a3a3f87f887be23a0c6eb1cb5b | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite3.OperationalError: table users has no column named bio 504238461 | |
541387941 | https://github.com/dogsheep/github-to-sqlite/issues/6#issuecomment-541387941 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDU0MTM4Nzk0MQ== | simonw 9599 | 2019-10-13T05:30:19Z | 2019-10-13T05:30:19Z | MEMBER | Fix released in 0.5: https://github.com/dogsheep/github-to-sqlite/releases/tag/0.5 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite3.OperationalError: table users has no column named bio 504238461 | |
770069864 | https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770069864 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60 | MDEyOklzc3VlQ29tbWVudDc3MDA2OTg2NA== | daniel-butler 22578954 | 2021-01-29T21:52:05Z | 2021-02-12T18:29:43Z | CONTRIBUTOR | For the purposes below I am assuming the organization I would get all the repositories and their related commits from is called `gh-organization`. The github's owner id of gh-orgnization is `123456789`. ```bash github-to-sqlite repos github.db gh-organization ``` I'm on a windows computer running git bash to be able to use the `|` command. This works for me ```bash sqlite3 github.db "SELECT full_name FROM repos WHERE owner = '123456789';" | tr '\n\r' ' ' | xargs | { read repos; github-to-sqlite commits github.db $repos; } ``` On a pure linux system I think this would work because the new line character is normally `\n` ```bash sqlite3 github.db "SELECT full_name FROM repos WHERE owner = '123456789';" | tr '\n' ' ' | xargs | { read repos; github-to-sqlite commits github.db $repos; }` ``` As expected I ran into rate limit issues #51 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Use Data from SQLite in other commands 797097140 | |
770071568 | https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770071568 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60 | MDEyOklzc3VlQ29tbWVudDc3MDA3MTU2OA== | simonw 9599 | 2021-01-29T21:56:15Z | 2021-01-29T21:56:15Z | MEMBER | I really like the way you're using pipes here - really smart. It's similar to how I build the demo database in this GitHub Actions workflow: https://github.com/dogsheep/github-to-sqlite/blob/62dfd3bc4014b108200001ef4bc746feb6f33b45/.github/workflows/deploy-demo.yml#L52-L82 `twitter-to-sqlite` actually has a mechanism for doing this kind of thing, documented at https://github.com/dogsheep/twitter-to-sqlite#providing-input-from-a-sql-query-with---sql-and---attach It lets you do things like: ``` $ twitter-to-sqlite users-lookup my.db --sql="select follower_id from following" --ids ``` Maybe I should add something similar to `github-to-sqlite`? Feels like it could be really useful. | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Use Data from SQLite in other commands 797097140 | |
770112248 | https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770112248 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60 | MDEyOklzc3VlQ29tbWVudDc3MDExMjI0OA== | daniel-butler 22578954 | 2021-01-30T00:01:03Z | 2021-01-30T01:14:42Z | CONTRIBUTOR | Yes that would be cool! I wouldn't mind helping. Is this the meat of it? https://github.com/dogsheep/twitter-to-sqlite/blob/21fc1cad6dd6348c67acff90a785b458d3a81275/twitter_to_sqlite/utils.py#L512 It looks like the cli option is added with this decorator : https://github.com/dogsheep/twitter-to-sqlite/blob/21fc1cad6dd6348c67acff90a785b458d3a81275/twitter_to_sqlite/cli.py#L14 I looked a bit at utils.py in the GitHub repository. I was surprised at the amount of manual mapping of the API response you had to do to get this to work. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Use Data from SQLite in other commands 797097140 | |
844249385 | https://github.com/dogsheep/github-to-sqlite/pull/61#issuecomment-844249385 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/61 | MDEyOklzc3VlQ29tbWVudDg0NDI0OTM4NQ== | simonw 9599 | 2021-05-19T16:07:06Z | 2021-05-19T16:07:06Z | MEMBER | Thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | fixing typo in get cli help text 797108702 | |
860895838 | https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-860895838 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64 | MDEyOklzc3VlQ29tbWVudDg2MDg5NTgzOA== | khimaros 231498 | 2021-06-14T18:23:21Z | 2021-06-14T21:37:35Z | NONE | i have a basic working version at https://github.com/khimaros/github-to-sqlite this can be tested with `github-to-sqlite events.db khimaros/events` caveat: the GitHub API doesn't seem to provide a complete history of events. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | feature: support "events" 920636216 | |
861035862 | https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861035862 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64 | MDEyOklzc3VlQ29tbWVudDg2MTAzNTg2Mg== | khimaros 231498 | 2021-06-14T22:29:20Z | 2021-06-14T22:29:20Z | NONE | it looks like the v4 GraphQL API is the only way to get data beyond 90 days from GitHub. this is significant change, but may be worth considering in the future. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | feature: support "events" 920636216 | |
861041597 | https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861041597 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64 | MDEyOklzc3VlQ29tbWVudDg2MTA0MTU5Nw== | simonw 9599 | 2021-06-14T22:44:54Z | 2021-06-14T22:44:54Z | MEMBER | Have you found a way to access events in GraphQL? I can only see way to access a timeline of events for a single issue or a single pull request. See also https://github.community/t/get-event-equivalent-for-v4/13600/2 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | feature: support "events" 920636216 | |
861042050 | https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861042050 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64 | MDEyOklzc3VlQ29tbWVudDg2MTA0MjA1MA== | simonw 9599 | 2021-06-14T22:45:42Z | 2021-06-14T22:45:42Z | MEMBER | I'm definitely interested in supporting events in this tool - see #14. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | feature: support "events" 920636216 | |
861087651 | https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861087651 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64 | MDEyOklzc3VlQ29tbWVudDg2MTA4NzY1MQ== | khimaros 231498 | 2021-06-15T00:48:37Z | 2021-06-15T00:48:37Z | NONE | @simonw -- i've created an omega-query that fetched most of what was interesting to me for a single user. found by poking around in the "Explorer" tab in https://docs.github.com/en/graphql/overview/explorer note: pagination is still required via `first` and `last` but it seems to allow unlimited history. ``` query MyQuery { __typename user(login: "<user>") { id pinnedItems(first: 100) { edges { node } } pullRequests(first: 100) { nodes { body title state createdAt } } createdAt issues(first: 100) { pageInfo { endCursor startCursor } nodes { title url createdAt body } } issueComments(first: 100) { edges { node { id updatedAt url body } } } repositories(first: 100) { nodes { createdAt description parent { name } pinnedIssues(first: 100) { edges { node { id } } } pinnedDiscussions(first: 100) { edges { node { id } } } } } starredRepositories(first: 100) { edges { node { id } } } } } ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | feature: support "events" 920636216 | |
885964242 | https://github.com/dogsheep/github-to-sqlite/pull/65#issuecomment-885964242 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65 | IC_kwDODFdgUs40zr3S | khimaros 231498 | 2021-07-23T23:45:35Z | 2021-07-23T23:45:35Z | NONE | @simonw is this PR of interest to you? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | basic support for events 923270900 | |
1266141699 | https://github.com/dogsheep/github-to-sqlite/pull/65#issuecomment-1266141699 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65 | IC_kwDODFdgUs5Ld8oD | khimaros 231498 | 2022-10-03T22:35:03Z | 2022-10-03T22:35:03Z | NONE | @simonw rebased against latest, please let me know if i should drop this PR. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | basic support for events 923270900 | |
929651819 | https://github.com/dogsheep/github-to-sqlite/pull/66#issuecomment-929651819 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/66 | IC_kwDODFdgUs43aVxr | sarcasticadmin 30531572 | 2021-09-28T21:50:31Z | 2021-09-28T21:50:31Z | NONE | @simonw any feedback/thoughts? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add --merged-by flag to pull-requests sub command 975161924 | |
985928838 | https://github.com/dogsheep/github-to-sqlite/issues/69#issuecomment-985928838 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69 | IC_kwDODFdgUs46xBSG | simonw 9599 | 2021-12-04T00:34:52Z | 2021-12-04T00:34:52Z | MEMBER | First attempt at this: ```sql select 'issues' as "table", id, node_id, title, user, created_at, body, repo from issues union all select 'issue_comments' as "table", issue_comments.id, issue_comments.node_id, '' as title, issue_comments.user, issue_comments.created_at, issue_comments.body, issues.repo from issue_comments join issues on issues.id = issue_comments.issue order by created_at desc ``` https://github-to-sqlite.dogsheep.net/github?sql=select+%27issues%27+as+%22table%22%2C+id%2C+node_id%2C+title%2C+user%2C+created_at%2C+body%2C+repo%0D%0Afrom+issues%0D%0Aunion+all%0D%0Aselect+%27issue_comments%27+as+%22table%22%2C+issue_comments.id%2C+issue_comments.node_id%2C+%27%27+as+title%2C+issue_comments.user%2C+issue_comments.created_at%2C+issue_comments.body%2C+issues.repo%0D%0Afrom+issue_comments+join+issues+on+issues.id+%3D+issue_comments.issue%0D%0Aorder+by+created_at+desc | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | View that combines issues and issue comments 1071071397 | |
541721437 | https://github.com/dogsheep/github-to-sqlite/issues/7#issuecomment-541721437 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/7 | MDEyOklzc3VlQ29tbWVudDU0MTcyMTQzNw== | simonw 9599 | 2019-10-14T14:44:12Z | 2019-10-14T14:44:12Z | MEMBER | Docs: https://github.com/dogsheep/github-to-sqlite/blob/0.5/README.md#retrieving-issue-comments-for-a-repository | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | issue-comments command for importing issue comments 506276893 | |
1105474232 | https://github.com/dogsheep/github-to-sqlite/issues/72#issuecomment-1105474232 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/72 | IC_kwDODFdgUs5B5DK4 | simonw 9599 | 2022-04-21T17:02:15Z | 2022-04-21T17:02:15Z | MEMBER | That's interesting - yeah it looks like the number of pages can be derived from the `Link` header, which is enough information to show a progress bar, probably using Click just to avoid adding another dependency. https://docs.github.com/en/rest/guides/traversing-with-pagination | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | feature: display progress bar when downloading multi-page responses 1211283427 | |
1188223299 | https://github.com/dogsheep/github-to-sqlite/pull/73#issuecomment-1188223299 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/73 | IC_kwDODFdgUs5G0tlD | simonw 9599 | 2022-07-18T19:40:06Z | 2022-07-18T19:40:06Z | MEMBER | Thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fixing 'NoneType' object has no attribute 'items' 1261884917 | |
1188223933 | https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188223933 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74 | IC_kwDODFdgUs5G0tu9 | simonw 9599 | 2022-07-18T19:40:50Z | 2022-07-18T19:42:41Z | MEMBER | Here's how the demo is deployed: https://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/.github/workflows/deploy-demo.yml#L103-L119 I'm suspicious of `py-gfm`, which is used like this: https://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/demo-metadata.json#L49-L51 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 500 error in github-to-sqlite demo 1308461063 | |
1188225625 | https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188225625 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74 | IC_kwDODFdgUs5G0uJZ | simonw 9599 | 2022-07-18T19:41:52Z | 2022-07-18T19:41:52Z | MEMBER | https://github.com/Zopieux/py-gfm says that library is no longer maintained, and suggests https://github.com/Zopieux/pycmarkgfm as an alternative. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 500 error in github-to-sqlite demo 1308461063 | |
1188228964 | https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188228964 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74 | IC_kwDODFdgUs5G0u9k | simonw 9599 | 2022-07-18T19:45:30Z | 2022-07-18T19:47:35Z | MEMBER | `pycmarkgfm` doesn't implement the Markdown plugin extension I was using. I'm going to drop the GFM rendering from the demo, and just treat it as regular markdown. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 500 error in github-to-sqlite demo 1308461063 | |
1188233729 | https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188233729 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74 | IC_kwDODFdgUs5G0wIB | simonw 9599 | 2022-07-18T19:51:02Z | 2022-07-18T19:51:02Z | MEMBER | Takes 30m to deploy the demo! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 500 error in github-to-sqlite demo 1308461063 | |
1188317682 | https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188317682 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74 | IC_kwDODFdgUs5G1Eny | simonw 9599 | 2022-07-18T21:14:22Z | 2022-07-18T21:14:22Z | MEMBER | That fixed it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 500 error in github-to-sqlite demo 1308461063 | |
1238190601 | https://github.com/dogsheep/github-to-sqlite/pull/76#issuecomment-1238190601 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/76 | IC_kwDODFdgUs5JzUoJ | OverkillGuy 2757699 | 2022-09-06T13:58:20Z | 2022-09-06T13:59:08Z | NONE | Tested PR just now in private org, fetched >2k repos infos flawlessly! poetry run github-to-sqlite repos --organization github.db MYORG | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add organization support to repos command 1363280254 | |
1498167714 | https://github.com/dogsheep/github-to-sqlite/issues/79#issuecomment-1498167714 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79 | IC_kwDODFdgUs5ZTDmi | simonw 9599 | 2023-04-05T21:12:55Z | 2023-04-05T21:12:55Z | MEMBER | It's a rate limiting problem: https://github.com/dogsheep/github-to-sqlite/actions/runs/4622674009/jobs/8175633155 ``` File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 407, in commits utils.save_commits(db, commits, repo_full["id"]) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 635, in save_commits for commit in commits: File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 417, in fetch_commits for commits in paginate(url, headers): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 478, in paginate raise GitHubError.from_response(response) github_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403) Error: Process completed with exit code 1. ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Deploy demo job is failing due to rate limit 1570375808 | |
549094195 | https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549094195 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDU0OTA5NDE5NQ== | simonw 9599 | 2019-11-03T00:43:16Z | 2019-11-03T00:43:28Z | MEMBER | Also need to take #5 into account - if this command creates incomplete user records, how do we repair them? And make sure that if we run this command first any future commands that populate users don't break (probably just a case of using `alter=True` in a few places). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | stargazers command, refs #4 516763727 | |
549230583 | https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549230583 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDU0OTIzMDU4Mw== | simonw 9599 | 2019-11-04T05:49:26Z | 2019-11-04T05:49:26Z | MEMBER | Adding the view from #10 would be useful here too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | stargazers command, refs #4 516763727 | |
549233778 | https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549233778 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDU0OTIzMzc3OA== | simonw 9599 | 2019-11-04T06:14:40Z | 2019-11-04T06:14:40Z | MEMBER | Spotted a tricky problem: running `github-to-sqlite starred stargazers.db` results in an incomplete `simonw` record. It creates a proper record for me thanks to this bit: https://github.com/dogsheep/github-to-sqlite/blob/ea07274667a08c67907e8bfbbccb6f0fb95ce817/github_to_sqlite/cli.py#L120-L126 But then... when it gets to the `datasette` repository which I have starred it over-writes my full user record with one that's missing most of the details, thanks to this bit: https://github.com/dogsheep/github-to-sqlite/blob/ea07274667a08c67907e8bfbbccb6f0fb95ce817/github_to_sqlite/utils.py#L117-L124 I need to find a way of NOT over-writing a good record with a thinner one. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | stargazers command, refs #4 516763727 | |
594154644 | https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-594154644 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDU5NDE1NDY0NA== | simonw 9599 | 2020-03-03T20:33:57Z | 2020-03-03T20:33:57Z | MEMBER | `sqlite-utils` supports proper upserts now so this problem should be easy to fix. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | stargazers command, refs #4 516763727 | |
623006004 | https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-623006004 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDYyMzAwNjAwNA== | simonw 9599 | 2020-05-02T20:00:26Z | 2020-05-02T20:00:26Z | MEMBER | I'm abandoning this in favour of a new implementation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | stargazers command, refs #4 516763727 | |
1073152522 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/10#issuecomment-1073152522 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/10 | IC_kwDODFE5qs4_9wIK | csusanu 9290214 | 2022-03-20T02:38:07Z | 2022-03-20T02:38:07Z | NONE | [This line](https://github.com/dogsheep/google-takeout-to-sqlite/blob/e54e544427f1cc3ea8189f0e95f54046301a8645/google_takeout_to_sqlite/utils.py) needs to say `"MyActivity.json"` instead of `"My Activity.json"`. Google must have changed the file name. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite3.OperationalError: no such table: main.my_activity 1123393829 | |
747126777 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/2#issuecomment-747126777 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDc0NzEyNjc3Nw== | simonw 9599 | 2020-12-17T00:36:52Z | 2020-12-17T00:36:52Z | MEMBER | The memory profiler tricks I used in https://github.com/dogsheep/healthkit-to-sqlite/issues/7 could help figure out what's going on here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | killed by oomkiller on large location-history 769376447 | |
747130908 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/2#issuecomment-747130908 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDc0NzEzMDkwOA== | khimaros 231498 | 2020-12-17T00:47:04Z | 2020-12-17T00:47:43Z | NONE | it looks like almost all of the memory consumption is coming from `json.load()`. another direction here may be to use the new "Semantic Location History" data which is already broken down by year and month. it also provides much more interesting data, such as estimated address, form of travel, etc. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | killed by oomkiller on large location-history 769376447 | |
780817596 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-780817596 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDc4MDgxNzU5Ng== | UtahDave 306240 | 2021-02-17T20:01:35Z | 2021-02-17T20:01:35Z | NONE | I've got this almost working. Just needs some polish | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature Request: Gmail 778380836 | |
781451701 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-781451701 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDc4MTQ1MTcwMQ== | Btibert3 203343 | 2021-02-18T16:06:21Z | 2021-02-18T16:06:21Z | NONE | Awesome! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature Request: Gmail 778380836 | |
783688547 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-783688547 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDc4MzY4ODU0Nw== | UtahDave 306240 | 2021-02-22T21:31:28Z | 2021-02-22T21:31:28Z | NONE | @Btibert3 I've opened a PR with my initial attempt at this. Would you be willing to give this a try? https://github.com/dogsheep/google-takeout-to-sqlite/pull/5 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature Request: Gmail 778380836 | |
790198930 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790198930 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDc5MDE5ODkzMA== | Btibert3 203343 | 2021-03-04T00:58:40Z | 2021-03-04T00:58:40Z | NONE | I am just seeing this sorry, yes! I will kick the tires later on tonight. My apologies for the delay. | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature Request: Gmail 778380836 | |
790934616 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790934616 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDc5MDkzNDYxNg== | Btibert3 203343 | 2021-03-04T20:54:44Z | 2021-03-04T20:54:44Z | NONE | Sorry for the delay, I got sidetracked after class last night. I am getting the following error: ``` /content# google-takeout-to-sqlite mbox takeout.db Takeout/Mail/gmail.mbox Usage: google-takeout-to-sqlite [OPTIONS] COMMAND [ARGS]...Try 'google-takeout-to-sqlite --help' for help. Error: No such command 'mbox'. ``` On the box, I installed with pip after cloning: https://github.com/UtahDave/google-takeout-to-sqlite.git | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature Request: Gmail 778380836 | |
783794520 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-783794520 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc4Mzc5NDUyMA== | UtahDave 306240 | 2021-02-23T01:13:54Z | 2021-02-23T01:13:54Z | NONE | Also, @simonw I created a test based off the existing tests. I think it's working correctly | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | WIP: Add Gmail takeout mbox import 813880401 | |
784638394 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-784638394 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc4NDYzODM5NA== | UtahDave 306240 | 2021-02-24T00:36:18Z | 2021-02-24T00:36:18Z | NONE | I noticed that @simonw is using black for formatting. I ran black on my additions in this PR. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | WIP: Add Gmail takeout mbox import 813880401 | |
786925280 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-786925280 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc4NjkyNTI4MA== | simonw 9599 | 2021-02-26T22:23:10Z | 2021-02-26T22:23:10Z | MEMBER | Thanks! I requested my Gmail export from takeout - once that arrives I'll test it against this and then merge the PR. | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | WIP: Add Gmail takeout mbox import 813880401 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
author_association 4 ✖