{"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785086", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6", "id": 706785086, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc4NTA4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T23:28:50Z", "updated_at": "2020-10-11T23:28:50Z", "author_association": "MEMBER", "body": "The XML for the OCR stuff is a bit weird. Currently I'm doing this to it:\r\n\r\nhttps://github.com/dogsheep/evernote-to-sqlite/blob/c33d7b043a45eb3e88676e5fa3ce31755199d9f8/evernote_to_sqlite/utils.py#L70-L78\r\n\r\nThis can produce some odd results, for example:\r\n\r\n> Sure 'Sure, 'Sure. Sure, Sure. sure sure. sure ? If you If Yau [you live jive In m 1n an area devoid of natural wonders, wanders, wonders ? wonders wonders. your mind will be blown, blown' blown. blown ? -e i ? ,1 IL it ? at ? KY ? fl ft bat at\r\n\r\nWhich came from this image:\r\n\r\n![image](https://user-images.githubusercontent.com/9599/95692952-5dd7c880-0bde-11eb-939a-d10b800a4105.png)\r\n\r\nThe XML for that is:\r\n\r\n```xml\r\n\r\n \r\n Sure\r\n 'Sure,\r\n 'Sure.\r\n Sure,\r\n Sure.\r\n \r\n \r\n sure\r\n sure.\r\n sure ?\r\n \r\n \r\n If you\r\n If Yau\r\n [you\r\n \r\n \r\n live\r\n jive\r\n \r\n \r\n In\r\n m\r\n 1n\r\n \r\n \r\n an\r\n \r\n \r\n area\r\n \r\n \r\n devoid\r\n \r\n \r\n of\r\n \r\n \r\n natural\r\n \r\n \r\n wonders,\r\n wanders,\r\n wonders ?\r\n wonders\r\n wonders.\r\n \r\n \r\n your\r\n \r\n \r\n mind\r\n \r\n \r\n will\r\n \r\n \r\n be\r\n \r\n \r\n blown,\r\n blown'\r\n blown.\r\n blown ?\r\n \r\n \r\n -e\r\n \r\n \r\n i ?\r\n \r\n \r\n ,1\r\n \r\n \r\n IL\r\n \r\n \r\n it ?\r\n at ?\r\n KY ?\r\n \r\n \r\n fl\r\n ft\r\n bat\r\n at\r\n \r\n\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718949182, "label": "Better handling of OCR data"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785201", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6", "id": 706785201, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc4NTIwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T23:29:39Z", "updated_at": "2020-10-11T23:29:39Z", "author_association": "MEMBER", "body": "It looks to me like each of those `` blocks has a number of guesses in order of confidence:\r\n```xml\r\n \r\n wonders,\r\n wanders,\r\n wonders ?\r\n wonders\r\n wonders.\r\n \r\n```\r\nSo maybe the best approach here is to just take the first `t` element within each `item`.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718949182, "label": "Better handling of OCR data"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/7#issuecomment-777827396", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7", "id": 777827396, "node_id": "MDEyOklzc3VlQ29tbWVudDc3NzgyNzM5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-11T22:13:14Z", "updated_at": "2021-02-11T22:13:14Z", "author_association": "MEMBER", "body": "My best guess is that you have an older version of `sqlite-utils` installed here - the `replace=True` argument was added in version 2.0. I've bumped the dependency in `setup.py`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 743297582, "label": "evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace'"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/9#issuecomment-777821383", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9", "id": 777821383, "node_id": "MDEyOklzc3VlQ29tbWVudDc3NzgyMTM4Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-11T22:01:28Z", "updated_at": "2021-02-11T22:01:28Z", "author_association": "MEMBER", "body": "Aha! I think I've figured out what's going on here.\r\n\r\nThe CData blocks containing the notes look like this:\r\n\r\n`
This note includes two images.

...`\r\n\r\nThe DTD at http://xml.evernote.com/pub/enml2.dtd includes some entities:\r\n\r\n```\r\n\r\n\r\n\r\n%HTMLlat1;\r\n\r\n\r\n%HTMLsymbol;\r\n\r\n\r\n%HTMLspecial;\r\n```\r\nSo I need to be able to handle all of those different entities. I think I can do that using `html.entities.entitydefs` from the Python standard library, which looks a bit like this:\r\n\r\n```python\r\n{'Aacute': '\u00c1',\r\n 'aacute': '\u00e1',\r\n 'Aacute;': '\u00c1',\r\n 'aacute;': '\u00e1',\r\n 'Abreve;': '\u0102',\r\n 'abreve;': '\u0103',\r\n 'ac;': '\u223e',\r\n 'acd;': '\u223f',\r\n# ...\r\n}\r\n```\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 748372469, "label": "ParseError: undefined entity š"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/pull/10#issuecomment-777839351", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10", "id": 777839351, "node_id": "MDEyOklzc3VlQ29tbWVudDc3NzgzOTM1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-11T22:37:55Z", "updated_at": "2021-02-11T22:37:55Z", "author_association": "MEMBER", "body": "I've merged these changes by hand now, thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 770712149, "label": "BugFix for encoding and not update info."}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544646516", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 544646516, "node_id": "MDEyOklzc3VlQ29tbWVudDU0NDY0NjUxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-21T18:30:14Z", "updated_at": "2019-10-21T18:30:14Z", "author_association": "MEMBER", "body": "Thanks to help from Dr. Laura Cantino at Science Hack Day San Francisco I've been able to pull together this query:\r\n\r\n```sql\r\nselect rsid, genotype, case genotype\r\n when 'AA' then 'brown eye color, 80% of the time'\r\n when 'AG' then 'brown eye color'\r\n when 'GG' then 'blue eye color, 99% of the time'\r\nend as interpretation from genome where rsid = 'rs12913832'\r\n```\r\n\r\nSee also https://www.snpedia.com/index.php/Rs12913832 - in particular this table:\r\n\r\n\"rs12913832_-_SNPedia\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544648863", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 544648863, "node_id": "MDEyOklzc3VlQ29tbWVudDU0NDY0ODg2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-21T18:36:03Z", "updated_at": "2019-10-21T18:36:03Z", "author_association": "MEMBER", "body": "\"natalie__select_rsid__genotype__case_genotype_when__AA__then__brown_eye_color__80__of_the_time__when__AG__then__brown_eye_color__when__GG__then__blue_eye_color__99__of_the_time__end_as_interpretation_from_genome_where_rsid____rs12913832__an\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765495861", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 765495861, "node_id": "MDEyOklzc3VlQ29tbWVudDc2NTQ5NTg2MQ==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-01-22T15:44:00Z", "updated_at": "2021-01-22T15:44:00Z", "author_association": "NONE", "body": "Risk of autoimmune disorders: https://www.snpedia.com/index.php/Genotype\r\n```\r\nselect rsid, genotype, case genotype\r\n when 'AA' then '2x risk of rheumatoid arthritis and other autoimmune diseases'\r\n when 'GG' then 'Normal risk for autoimmune disorders'\r\nend as interpretation from genome where rsid = 'rs2476601'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765498984", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 765498984, "node_id": "MDEyOklzc3VlQ29tbWVudDc2NTQ5ODk4NA==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-01-22T15:48:25Z", "updated_at": "2021-01-22T15:49:33Z", "author_association": "NONE", "body": "The \"Warrior Gene\" https://www.snpedia.com/index.php/Rs4680\r\n\r\n```\r\nselect rsid, genotype, case genotype\r\n when 'AA' then '(worrier) advantage in memory and attention tasks'\r\n when 'AG' then 'Intermediate dopamine levels, other effects'\r\n when 'GG' then '(warrior) multiple associations, see details'\r\nend as interpretation from genome where rsid = 'rs4680'\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765502845", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 765502845, "node_id": "MDEyOklzc3VlQ29tbWVudDc2NTUwMjg0NQ==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-01-22T15:53:19Z", "updated_at": "2021-01-22T15:53:19Z", "author_association": "NONE", "body": "rs7903146 Influences risk of Type-2 diabetes\r\nhttps://www.snpedia.com/index.php/Rs7903146\r\n```\r\nselect rsid, genotype, case genotype\r\n when 'CC' then 'Normal (lower) risk of Type 2 Diabetes and Gestational Diabetes.'\r\n when 'CT' then '1.4x increased risk for diabetes (and perhaps colon cancer).'\r\n when 'TT' then '2x increased risk for Type-2 diabetes'\r\nend as interpretation from genome where rsid = 'rs7903146'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765506901", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 765506901, "node_id": "MDEyOklzc3VlQ29tbWVudDc2NTUwNjkwMQ==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-01-22T15:58:41Z", "updated_at": "2021-01-22T15:58:58Z", "author_association": "NONE", "body": "Both rs10757274 and rs2383206 can both indicate higher risks of heart disease\r\nhttps://www.snpedia.com/index.php/Rs2383206\r\n\r\n```\r\nselect rsid, genotype, case genotype\r\n when 'AA' then 'Normal'\r\n when 'AG' then '~1.2x increased risk for heart disease'\r\n when 'GG' then '~1.3x increased risk for heart disease'\r\nend as interpretation from genome where rsid = 'rs10757274'\r\n```\r\n\r\n```\r\nselect rsid, genotype, case genotype\r\n when 'AA' then 'Normal'\r\n when 'AG' then '1.4x increased risk for heart disease'\r\n when 'GG' then '1.7x increased risk for heart disease'\r\nend as interpretation from genome where rsid = 'rs2383206'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765523517", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 765523517, "node_id": "MDEyOklzc3VlQ29tbWVudDc2NTUyMzUxNw==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-01-22T16:20:25Z", "updated_at": "2021-01-22T16:20:25Z", "author_association": "NONE", "body": "rs53576: the oxytocin receptor (OXTR) gene\r\n\r\n```\r\nselect rsid, genotype, case genotype\r\n when 'AA' then 'Lack of empathy?'\r\n when 'AG' then 'Lack of empathy?'\r\n when 'GG' then 'Optimistic and empathetic; handle stress well'\r\nend as interpretation from genome where rsid = 'rs53576'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765525338", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 765525338, "node_id": "MDEyOklzc3VlQ29tbWVudDc2NTUyNTMzOA==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-01-22T16:22:44Z", "updated_at": "2021-01-22T16:22:44Z", "author_association": "NONE", "body": "rs1333049 associated with coronary artery disease\r\nhttps://www.snpedia.com/index.php/Rs1333049\r\n```\r\n\r\nselect rsid, genotype, case genotype\r\n when 'CC' then '1.9x increased risk for coronary artery disease'\r\n when 'CG' then '1.5x increased risk for CAD'\r\n when 'GG' then 'normal'\r\nend as interpretation from genome where rsid = 'rs1333049'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-831004775", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 831004775, "node_id": "MDEyOklzc3VlQ29tbWVudDgzMTAwNDc3NQ==", "user": {"value": 25372415, "label": "cobiadigital"}, "created_at": "2021-05-03T03:46:23Z", "updated_at": "2021-05-03T03:46:23Z", "author_association": "NONE", "body": "RS1800955 is related to novelty seeking and ADHD\r\nhttps://www.snpedia.com/index.php/Rs1800955\r\n\r\n`select rsid, genotype, case genotype\r\n when 'CC' then 'increased susceptibility to novelty seeking'\r\n when 'CT' then 'increased susceptibility to novelty seeking'\r\n when 'TT' then 'normal'\r\nend as interpretation from genome where rsid = 'rs1800955'`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-549230337", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10", "id": 549230337, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTIzMDMzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-04T05:47:18Z", "updated_at": "2019-11-04T05:47:18Z", "author_association": "MEMBER", "body": "This definition isn't quite right - it's not pulling the identity of the user who starred the repo (`users.login` ends up being the owner login instead).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 516967682, "label": "Add this repos_starred view"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622461122", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10", "id": 622461122, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTEyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:34:39Z", "updated_at": "2020-05-01T16:34:39Z", "author_association": "MEMBER", "body": "Blocked on #37", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 516967682, "label": "Add this repos_starred view"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622980203", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10", "id": 622980203, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk4MDIwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T16:34:29Z", "updated_at": "2020-05-02T16:34:29Z", "author_association": "MEMBER", "body": "Fixed definition:\r\n```sql\r\nselect\r\n stars.starred_at,\r\n starring_user.login as starred_by,\r\n repos.*\r\nfrom\r\n repos\r\n join stars on repos.id = stars.repo\r\n join users as starring_user on stars.user = starring_user.id\r\n join users on repos.owner = users.id\r\norder by\r\n starred_at desc;\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 516967682, "label": "Add this repos_starred view"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594151327", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 594151327, "node_id": "MDEyOklzc3VlQ29tbWVudDU5NDE1MTMyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-03T20:26:15Z", "updated_at": "2020-03-03T20:32:23Z", "author_association": "MEMBER", "body": "Better version (since this also includes JSON array of repository topics):\r\n```sql\r\nCREATE VIEW recent_releases AS select\r\n repos.rowid as rowid,\r\n json_object(\"label\", repos.full_name, \"href\", repos.html_url) as repo,\r\n json_object(\r\n \"href\",\r\n releases.html_url,\r\n \"label\",\r\n releases.name\r\n ) as release,\r\n substr(releases.published_at, 0, 11) as date,\r\n releases.body as body_markdown,\r\n releases.published_at,\r\n coalesce(repos.topics, '[]') as topics\r\nfrom\r\n releases\r\n join repos on repos.id = releases.repo\r\norder by\r\n releases.published_at desc\r\n```\r\nThat `repos.rowid as rowid` bit is necessary because otherwise clicking on a link in facet-by-topic doesn't return any results.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594155249", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 594155249, "node_id": "MDEyOklzc3VlQ29tbWVudDU5NDE1NTI0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-03T20:35:17Z", "updated_at": "2020-03-03T20:35:17Z", "author_association": "MEMBER", "body": "`swarm-to-sqlite` has an example of adding views here: https://github.com/dogsheep/swarm-to-sqlite/blob/f2c89dd613fb8a7f14e5267ccc2145463b996190/swarm_to_sqlite/utils.py#L141\r\n\r\nI think that approach can be approved by first checking if the view exists, then dropping it, then recreating it. Could even try to see if the view exists and matches what we were going to set it to and do nothing if that is the case.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 622461223, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:34:52Z", "updated_at": "2020-05-01T16:34:52Z", "author_association": "MEMBER", "body": "Blocked on #37", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 622989874, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T17:46:14Z", "updated_at": "2020-05-02T17:46:14Z", "author_association": "MEMBER", "body": "Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 622998813, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T18:58:17Z", "updated_at": "2020-05-02T18:58:17Z", "author_association": "MEMBER", "body": "Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602861730", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602861730, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2MTczMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:19:16Z", "updated_at": "2020-03-23T21:19:42Z", "author_association": "MEMBER", "body": "I'm going to do this with a scheduled GitHub Action in this repo.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862236", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602862236, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2MjIzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:20:26Z", "updated_at": "2020-03-23T21:20:26Z", "author_association": "MEMBER", "body": "I'll run the `commits` and `issues` and `issue-comments` commands in addition to the `releases` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862967", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602862967, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg2Mjk2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T21:22:04Z", "updated_at": "2020-03-23T21:22:04Z", "author_association": "MEMBER", "body": "Following these instructions: https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602895896", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602895896, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg5NTg5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T22:42:25Z", "updated_at": "2020-03-23T22:42:25Z", "author_association": "MEMBER", "body": "Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602916947", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602916947, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxNjk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:38:06Z", "updated_at": "2020-03-23T23:38:06Z", "author_association": "MEMBER", "body": "Woohoo! https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602917713", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602917713, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxNzcxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:40:29Z", "updated_at": "2020-03-23T23:40:29Z", "author_association": "MEMBER", "body": "Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602918689", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602918689, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxODY4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:43:39Z", "updated_at": "2020-03-23T23:47:50Z", "author_association": "MEMBER", "body": "I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602919058", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602919058, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkxOTA1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:44:48Z", "updated_at": "2020-03-23T23:44:48Z", "author_association": "MEMBER", "body": "Next step: use a `metadata.json` file to add some extras. And add the `datasette-render-markdown` plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602924714", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13", "id": 602924714, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyNDcxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-24T00:03:25Z", "updated_at": "2020-03-24T00:03:25Z", "author_association": "MEMBER", "body": "This is good enough for the 1.0 release.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521275281, "label": "Set up a live demo Datasette instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559883311", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14", "id": 559883311, "node_id": "MDEyOklzc3VlQ29tbWVudDU1OTg4MzMxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-29T21:30:37Z", "updated_at": "2019-11-29T21:30:37Z", "author_association": "MEMBER", "body": "I should build the command to persist ETags and obey their polling guidelines:\r\n\r\n> Events are optimized for polling with the \"ETag\" header. If no new events have been triggered, you will see a \"304 Not Modified\" response, and your current rate limit will be untouched. There is also an \"X-Poll-Interval\" header that specifies how often (in seconds) you are allowed to poll. In times of high server load, the time may increase. Please obey the header.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 530491074, "label": "Command for importing events"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559902818", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14", "id": 559902818, "node_id": "MDEyOklzc3VlQ29tbWVudDU1OTkwMjgxOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-30T01:32:38Z", "updated_at": "2019-11-30T01:32:38Z", "author_association": "MEMBER", "body": "Prototype:\r\n```\r\npip install sqlite-utils paginate-json\r\npaginate-json \"https://api.github.com/users/simonw/events\" | sqlite-utils insert /tmp/events.db events - --pk=id\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 530491074, "label": "Command for importing events"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-613641947", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14", "id": 613641947, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMzY0MTk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-14T19:38:24Z", "updated_at": "2020-04-14T19:38:34Z", "author_association": "MEMBER", "body": "Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 530491074, "label": "Command for importing events"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602309825", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602309825, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMwOTgyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T00:52:38Z", "updated_at": "2020-03-23T00:52:38Z", "author_association": "MEMBER", "body": "Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the `releases` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602311701", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602311701, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMxMTcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T01:05:15Z", "updated_at": "2020-03-23T01:06:14Z", "author_association": "MEMBER", "body": "None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0\r\n\r\n```\r\ngithub-to-sqlite releases releases.db instrumenta/conftest\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602312054", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 602312054, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMxMjA1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T01:07:10Z", "updated_at": "2020-03-23T01:07:10Z", "author_association": "MEMBER", "body": "Each asset looks like this:\r\n```json\r\n {\r\n \"url\": \"https://api.github.com/repos/instrumenta/conftest/releases/assets/11811946\",\r\n \"id\": 11811946,\r\n \"node_id\": \"MDEyOlJlbGVhc2VBc3NldDExODExOTQ2\",\r\n \"name\": \"checksums.txt\",\r\n \"label\": \"\",\r\n \"uploader\": {\r\n \"login\": \"garethr\",\r\n \"id\": 2029,\r\n \"node_id\": \"MDQ6VXNlcjIwMjk=\",\r\n \"avatar_url\": \"https://avatars2.githubusercontent.com/u/2029?v=4\",\r\n \"gravatar_id\": \"\",\r\n \"url\": \"https://api.github.com/users/garethr\",\r\n \"html_url\": \"https://github.com/garethr\",\r\n \"followers_url\": \"https://api.github.com/users/garethr/followers\",\r\n \"following_url\": \"https://api.github.com/users/garethr/following{/other_user}\",\r\n \"gists_url\": \"https://api.github.com/users/garethr/gists{/gist_id}\",\r\n \"starred_url\": \"https://api.github.com/users/garethr/starred{/owner}{/repo}\",\r\n \"subscriptions_url\": \"https://api.github.com/users/garethr/subscriptions\",\r\n \"organizations_url\": \"https://api.github.com/users/garethr/orgs\",\r\n \"repos_url\": \"https://api.github.com/users/garethr/repos\",\r\n \"events_url\": \"https://api.github.com/users/garethr/events{/privacy}\",\r\n \"received_events_url\": \"https://api.github.com/users/garethr/received_events\",\r\n \"type\": \"User\",\r\n \"site_admin\": false\r\n },\r\n \"content_type\": \"text/plain; charset=utf-8\",\r\n \"state\": \"uploaded\",\r\n \"size\": 600,\r\n \"download_count\": 2,\r\n \"created_at\": \"2019-03-30T16:56:44Z\",\r\n \"updated_at\": \"2019-03-30T16:56:44Z\",\r\n \"browser_download_url\": \"https://github.com/instrumenta/conftest/releases/download/v0.1.0/checksums.txt\"\r\n }\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-605439685", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15", "id": 605439685, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNTQzOTY4NQ==", "user": {"value": 2029, "label": "garethr"}, "created_at": "2020-03-28T12:17:01Z", "updated_at": "2020-03-28T12:17:01Z", "author_association": "NONE", "body": "That looks great, thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 544571092, "label": "Assets table with downloads"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-571412923", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16", "id": 571412923, "node_id": "MDEyOklzc3VlQ29tbWVudDU3MTQxMjkyMw==", "user": {"value": 15092, "label": "jayvdb"}, "created_at": "2020-01-07T03:06:46Z", "updated_at": "2020-01-07T03:06:46Z", "author_association": "NONE", "body": "I re-tried after doing `auth`, and I get the same result.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546051181, "label": "Exception running first command: IndexError: list index out of range"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-601979709", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16", "id": 601979709, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTk3OTcwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-21T02:20:08Z", "updated_at": "2020-03-21T02:20:08Z", "author_association": "MEMBER", "body": "I'm pretty sure this means I need to upgrade the underlying `sqlite-utils` dependency. I saw the same bug in `twitter-to-sqlite`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546051181, "label": "Exception running first command: IndexError: list index out of range"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-602136481", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16", "id": 602136481, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjEzNjQ4MQ==", "user": {"value": 15092, "label": "jayvdb"}, "created_at": "2020-03-22T02:08:57Z", "updated_at": "2020-03-22T02:08:57Z", "author_association": "NONE", "body": "I'd love to be using your library as a better cached gh layer for a new library I have built, replacing large parts of the very ugly https://github.com/jayvdb/pypidb/blob/master/pypidb/_github.py , and then probably being able to rebuild the setuppy chunk as a feature here at a later stage.\r\n\r\nI would also need tokenless and netrc support, but I would be happy to add those bits.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546051181, "label": "Exception running first command: IndexError: list index out of range"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-613611455", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16", "id": 613611455, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMzYxMTQ1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-14T18:37:21Z", "updated_at": "2020-04-14T18:37:21Z", "author_association": "MEMBER", "body": "This should have been fixed by #20 and #23 \r\n\r\n@jayvdb I'm definitely interested in this tool working as a library - it's purely designed as a CLI tool at the moment, but cleaning it up to work better as a dependency is totally in-scope for the project. https://sqlite-utils.readthedocs.io/ is an example of a tool I've built that works for both.\r\n\r\nFeel free to open a new issue here with some notes on what you would need for this to work as a library for your project!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546051181, "label": "Exception running first command: IndexError: list index out of range"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597354514", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17", "id": 597354514, "node_id": "MDEyOklzc3VlQ29tbWVudDU5NzM1NDUxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-10T22:37:45Z", "updated_at": "2020-03-10T22:37:45Z", "author_association": "MEMBER", "body": "I should add an option to stop the moment you see a commit you have fetched before.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 578883725, "label": "Command for importing commits"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597358364", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17", "id": 597358364, "node_id": "MDEyOklzc3VlQ29tbWVudDU5NzM1ODM2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-10T22:50:20Z", "updated_at": "2020-03-11T01:18:36Z", "author_association": "MEMBER", "body": "By default it will stop when it sees a commit that has already been stored. You will be able to over-ride that behaviour using `--all`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 578883725, "label": "Command for importing commits"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980724", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 601980724, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTk4MDcyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-21T02:28:05Z", "updated_at": "2020-03-21T02:28:05Z", "author_association": "MEMBER", "body": "This suggests that `commit[\"author\"]` can be `None` in some cases?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980957", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 601980957, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTk4MDk1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-21T02:30:03Z", "updated_at": "2020-03-21T02:30:03Z", "author_association": "MEMBER", "body": "https://github.community/t5/GitHub-API-Development-and/Request-for-commits-quot-author-quot-null-and-quot-committer/m-p/35842/highlight/true#M3372\r\n> Commits aren't always associated with a GitHub user. For example, perhaps a friend of mine and I were working on a project together. I have a GitHub account and my friend doesn't. If we both add commits to the repository using our own email addresses and names and then I push the repository to GitHub, my commits will be associated with my GitHub user account but my friends' commits will show up with `author` and `committer` as `null`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602807178", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602807178, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgwNzE3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:24:43Z", "updated_at": "2020-03-23T19:24:43Z", "author_association": "MEMBER", "body": "I need to find an example before I work on this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811468", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602811468, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxMTQ2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:33:14Z", "updated_at": "2020-03-23T19:33:14Z", "author_association": "MEMBER", "body": "Found one: https://api.github.com/repos/simonw/simonw.github.com/commits", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811785", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602811785, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxMTc4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:33:56Z", "updated_at": "2020-03-23T19:34:06Z", "author_association": "MEMBER", "body": "```json\r\n[\r\n {\r\n \"sha\": \"a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"node_id\": \"MDY6Q29tbWl0OTMyMDk6YThkYzkxNDA4OWQzOTlkOWI1MjJlYmI1MWI2N2Y5YWMyZThhYTZiMA==\",\r\n \"commit\": {\r\n \"author\": {\r\n \"name\": \"Simon Willison\",\r\n \"email\": \"simon@...\",\r\n \"date\": \"2008-12-18T23:17:12Z\"\r\n },\r\n \"committer\": {\r\n \"name\": \"Simon Willison\",\r\n \"email\": \"simon@...\",\r\n \"date\": \"2008-12-18T23:17:12Z\"\r\n },\r\n \"message\": \"First commit\",\r\n \"tree\": {\r\n \"sha\": \"ac2dfb75e2592c59165c2880f3f7a16dafd452a1\",\r\n \"url\": \"https://api.github.com/repos/simonw/simonw.github.com/git/trees/ac2dfb75e2592c59165c2880f3f7a16dafd452a1\"\r\n },\r\n \"url\": \"https://api.github.com/repos/simonw/simonw.github.com/git/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"comment_count\": 0,\r\n \"verification\": {\r\n \"verified\": false,\r\n \"reason\": \"unsigned\",\r\n \"signature\": null,\r\n \"payload\": null\r\n }\r\n },\r\n \"url\": \"https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"html_url\": \"https://github.com/simonw/simonw.github.com/commit/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0\",\r\n \"comments_url\": \"https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0/comments\",\r\n \"author\": null,\r\n \"committer\": null,\r\n \"parents\": []\r\n }\r\n]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602813731", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602813731, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxMzczMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:38:03Z", "updated_at": "2020-03-23T20:44:49Z", "author_association": "MEMBER", "body": "So it turns out `\"author\"` and `\"committer\"` on the commit are `null` if the email address in the nested `\"commit\"` doesn't match an existing GitHub user.\r\n\r\nMaybe I should be storing the nested data somewhere as well?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602815120", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602815120, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjgxNTEyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T19:40:55Z", "updated_at": "2020-03-23T19:43:19Z", "author_association": "MEMBER", "body": "I could pull a pk-hashed version of the name/email into separate `raw_author` and `raw_committer` columns perhaps - against a `commit_authors` table. Could be interesting.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602846293", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18", "id": 602846293, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg0NjI5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T20:44:40Z", "updated_at": "2020-03-23T20:44:40Z", "author_association": "MEMBER", "body": "I implemented the `raw_authors` idea.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585411547, "label": "Commits in GitHub API can have null author"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602304531", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19", "id": 602304531, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjMwNDUzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T00:20:28Z", "updated_at": "2020-03-23T00:20:28Z", "author_association": "MEMBER", "body": "Current FTS code:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/e35eec4343aa560c58c1634cc228d0d46c442304/github_to_sqlite/utils.py#L136-L143", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585850715, "label": "Enable full-text search for more stuff (like commits, issues and issue_comments)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602782437", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19", "id": 602782437, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjc4MjQzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T18:36:12Z", "updated_at": "2020-03-23T18:51:23Z", "author_association": "MEMBER", "body": "* `commits`: `message`\r\n* `issue_comments`: `body`\r\n* `issues`: `title`, `body`\r\n* `labels`: `name`, `description`\r\n* `licenses`: `name`\r\n* `milestones`: `title`, `description`\r\n* `releases`: `name`, `body`\r\n* `repos`: `name`, `description`\r\n* `users`: `login`, `name` (maybe `company`, `bio`)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585850715, "label": "Enable full-text search for more stuff (like commits, issues and issue_comments)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602896434", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21", "id": 602896434, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjg5NjQzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T22:43:37Z", "updated_at": "2020-03-23T22:43:37Z", "author_association": "MEMBER", "body": "I'm going to do this now to help figure out the latest error in #13:\r\n```\r\nTraceback (most recent call last):\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite\", line 11, in \r\n load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')()\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 237, in commits\r\n utils.save_commits(db, commits, repo_full[\"id\"])\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 345, in save_commits\r\n for commit in commits:\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 207, in fetch_commits\r\n if stop_when(commit):\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 224, in stop_when\r\n db[\"commits\"].get(commit[\"sha\"])\r\nTypeError: string indices must be integers\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 586561727, "label": "Turn GitHub API errors into exceptions"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602920163", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21", "id": 602920163, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyMDE2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-23T23:48:22Z", "updated_at": "2020-03-23T23:48:22Z", "author_association": "MEMBER", "body": "I'm happy with this pattern:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 586561727, "label": "Turn GitHub API errors into exceptions"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/23#issuecomment-602928533", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23", "id": 602928533, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMjkyODUzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-24T00:15:49Z", "updated_at": "2020-03-24T00:15:49Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/github-to-sqlite/releases/tag/1.0", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 586595839, "label": "Release 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/25#issuecomment-614810417", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25", "id": 614810417, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDgxMDQxNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T18:07:11Z", "updated_at": "2020-04-16T18:07:11Z", "author_association": "MEMBER", "body": "Turns out the main problem was #26 - now fixed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601265023, "label": "Improvements to demo instance"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614794739", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26", "id": 614794739, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDc5NDczOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T17:38:28Z", "updated_at": "2020-04-16T17:38:28Z", "author_association": "MEMBER", "body": "I'm already doing this here: https://github.com/dogsheep/github-to-sqlite/blob/c4aaa50e167cfa9021c7c94260bc3e89e10947bf/github_to_sqlite/utils.py#L246-L250", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601271612, "label": "Topics are missing from repositories"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614795712", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26", "id": 614795712, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDc5NTcxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T17:40:27Z", "updated_at": "2020-04-16T17:40:27Z", "author_association": "MEMBER", "body": "Aha! it was missing from the `fetch_repo()` function.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601271612, "label": "Topics are missing from repositories"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831451", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 614831451, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDgzMTQ1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T18:47:25Z", "updated_at": "2020-04-16T18:47:25Z", "author_association": "MEMBER", "body": "Is it possible for a repo to have an `owner` that differs from its `organization`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831842", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 614831842, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDgzMTg0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T18:48:18Z", "updated_at": "2020-04-16T18:48:18Z", "author_association": "MEMBER", "body": "I'm going to make `organization` another foreign key to the `users` table just in case it IS possible (maybe with GitHub Enterprise or similar?)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 614843406, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNDg0MzQwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-16T19:11:53Z", "updated_at": "2020-04-16T19:20:23Z", "author_association": "MEMBER", "body": "This didn't quite work: the column type is incorrect, so the foreign key relationship isn't sticking:\r\n\r\nhttps://github-to-sqlite.dogsheep.net/github/repos?organization=53015001\r\n\r\n`[organization] TEXT REFERENCES [users]([id])` - should be `INTEGER`.\r\n\r\nThe problem is that if the first repo inserted has no organization it's set to `null`, which `sqlite-utils` derives as a `TEXT` column.\r\n\r\nOne solution would be to create the column explicitly with a type, but this could get messy.\r\n\r\nI think I want a new sqlite-utils feature for this instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615518606", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 615518606, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxODYwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T00:14:32Z", "updated_at": "2020-04-18T00:14:32Z", "author_association": "MEMBER", "body": "https://github.com/simonw/sqlite-utils/issues/100 is done and released in sqlite-utils 2.7.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615519409", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27", "id": 615519409, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxOTQwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T00:19:16Z", "updated_at": "2020-04-18T00:19:16Z", "author_association": "MEMBER", "body": "```\r\n$ github-to-sqlite repos b.db dogsheep\r\n$ sqlite3 b.db '.schema repos'\r\nCREATE TABLE [repos] (\r\n [id] INTEGER PRIMARY KEY,\r\n...\r\n [permissions] TEXT,\r\n [organization] INTEGER REFERENCES [users]([id]),\r\n FOREIGN KEY(license) REFERENCES licenses(key)\r\n);\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601330277, "label": "Repos have a big blob of JSON in the organization column"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883040", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28", "id": 615883040, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTg4MzA0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T14:45:38Z", "updated_at": "2020-04-18T14:45:38Z", "author_association": "MEMBER", "body": "```\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 219, in contributors\r\n utils.save_contributors(db, contributors, repo_full[\"id\"])\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 354, in save_contributors\r\n for contributor in contributors:\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 228, in fetch_contributors\r\n for contributors in paginate(url, headers):\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 286, in paginate\r\n data = response.json()\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/requests/models.py\", line 898, in json\r\n return complexjson.loads(self.text, **kwargs)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/__init__.py\", line 357, in loads\r\n return _default_decoder.decode(s)\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py\", line 337, in decode\r\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\r\n File \"/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py\", line 355, in raw_decode\r\n raise JSONDecodeError(\"Expecting value\", s, err.value) from None\r\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601333634, "label": "Pull repository contributors"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883687", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28", "id": 615883687, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTg4MzY4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T14:49:58Z", "updated_at": "2020-04-18T14:49:58Z", "author_association": "MEMBER", "body": "That happened trying to pull contributors for `dogsheep/beta` - an empty repository. Turns out it was returning a `204 no content`:\r\n```\r\n~ $ curl -i 'https://api.github.com/repos/dogsheep/beta/contributors'\r\nHTTP/1.1 204 No Content\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601333634, "label": "Pull repository contributors"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615886206", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28", "id": 615886206, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTg4NjIwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T15:04:59Z", "updated_at": "2020-04-18T15:04:59Z", "author_association": "MEMBER", "body": "Demo: https://github-to-sqlite.dogsheep.net/github/contributors\r\n\r\nDocumentation: https://github.com/dogsheep/github-to-sqlite/blob/13f8868fb5efa01c263b24f6dd91c617e6e938e1/README.md#fetching-contributors-to-a-repository ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 601333634, "label": "Pull repository contributors"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29", "id": 616883275, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjg4MzI3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T00:43:28Z", "updated_at": "2020-04-21T00:43:28Z", "author_association": "MEMBER", "body": "I'm copying repo from issue, which surprisingly is a string, not an integer ID.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603617013, "label": "Milestones should have foreign key to creator and repo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531516956", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3", "id": 531516956, "node_id": "MDEyOklzc3VlQ29tbWVudDUzMTUxNjk1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-14T21:56:31Z", "updated_at": "2019-09-14T21:56:31Z", "author_association": "MEMBER", "body": "https://api.github.com/users/simonw/repos\r\n\r\nIt would be useful to be able to fetch stargazers, forks etc as well. Not sure if that should be a separate command or a `--stargazers` option to this command.\r\n\r\nProbably a separate command since `issues` is a separate command already.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 493670426, "label": "Command to fetch all repos belonging to a user or organization"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531517083", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3", "id": 531517083, "node_id": "MDEyOklzc3VlQ29tbWVudDUzMTUxNzA4Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-14T21:58:42Z", "updated_at": "2019-09-14T21:58:42Z", "author_association": "MEMBER", "body": "Split stargazers into #4", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 493670426, "label": "Command to fetch all repos belonging to a user or organization"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616879753", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30", "id": 616879753, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjg3OTc1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T00:29:29Z", "updated_at": "2020-04-21T00:29:29Z", "author_association": "MEMBER", "body": "`assignee` looks like it's the incorrect type too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603618244, "label": "Issues milestone column is the wrong type"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616883726", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30", "id": 616883726, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjg4MzcyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T00:45:23Z", "updated_at": "2020-04-21T00:45:23Z", "author_association": "MEMBER", "body": "Demo of fix: https://github-to-sqlite.dogsheep.net/github/issues?assignee__notblank=1&milestone__notblank=1\r\n\r\n![github__issues__4_rows_where_where_assignee_is_not_blank_and_milestone_is_not_blank_sorted_by_updated_at_descending](https://user-images.githubusercontent.com/9599/79812758-b49ace80-832e-11ea-81db-bdf993b872cc.png)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603618244, "label": "Issues milestone column is the wrong type"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-616884647", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31", "id": 616884647, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjg4NDY0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T00:49:16Z", "updated_at": "2020-04-21T00:50:20Z", "author_association": "MEMBER", "body": "The API just gives us the `repository_url`: https://api.github.com/repos/simonw/datasette/issues\r\n\r\n![Mozilla_Firefox_and_Topic__Week_2__Discussion__Submit_your_six_story_points_here](https://user-images.githubusercontent.com/9599/79812950-283cdb80-832f-11ea-8759-9633087d1e7e.png)\r\n\r\nWe currently turn that into a `simonw/datasette` string here:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/e0e8d8caa9657b04bfb8a2cf16c9b580f38b1805/github_to_sqlite/utils.py#L43-L46", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603624862, "label": "Issue and milestone should have foreign key to repo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617348174", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31", "id": 617348174, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNzM0ODE3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T18:50:29Z", "updated_at": "2020-04-21T18:50:29Z", "author_association": "MEMBER", "body": "Since this represents a breaking schema change for anyone running SQL queries against these tables, I'm going to do a major version bump to 2.0 when I release this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603624862, "label": "Issue and milestone should have foreign key to repo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617491607", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31", "id": 617491607, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNzQ5MTYwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-22T01:20:19Z", "updated_at": "2020-04-22T01:20:19Z", "author_association": "MEMBER", "body": "https://github-to-sqlite.dogsheep.net/github/milestones now link to repo:\r\n\r\n\"Screen\r\n\r\nAnd so do issues: https://github-to-sqlite.dogsheep.net/github/issues\r\n\r\n\"Screen\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 603624862, "label": "Issue and milestone should have foreign key to repo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617364956", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32", "id": 617364956, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNzM2NDk1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T19:24:45Z", "updated_at": "2020-04-21T19:24:45Z", "author_association": "MEMBER", "body": "That's because I just broke this code:\r\n\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/2cf75a0a036719eb7e57fdc7c5c2ea0f4c26978a/github_to_sqlite/utils.py#L131-L139\r\n\r\nIt expects the `repo` column to be `simonw/datasette` but it's now an ID instead.\r\n\r\nI should add a test for this as part of the fix.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 604222295, "label": "Issue comments don't appear to populate issues foreign key"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617369247", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32", "id": 617369247, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNzM2OTI0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-21T19:33:03Z", "updated_at": "2020-04-21T19:33:03Z", "author_association": "MEMBER", "body": "Caused by #31.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 604222295, "label": "Issue comments don't appear to populate issues foreign key"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617490914", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32", "id": 617490914, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNzQ5MDkxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-22T01:17:44Z", "updated_at": "2020-04-22T01:17:44Z", "author_association": "MEMBER", "body": "https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101 now shows issues.\r\n\r\nAnd https://github-to-sqlite.dogsheep.net/github/issue_comments links to them:\r\n\r\n\"Screen\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 604222295, "label": "Issue comments don't appear to populate issues foreign key"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622169728", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33", "id": 622169728, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjE2OTcyOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T23:18:51Z", "updated_at": "2020-04-30T23:18:51Z", "author_association": "MEMBER", "body": "Sure, that sounds fine to me.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 609950090, "label": "Fall back to authentication via ENV"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622171097", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33", "id": 622171097, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjE3MTA5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T23:22:45Z", "updated_at": "2020-04-30T23:23:57Z", "author_association": "MEMBER", "body": "The `auth.json` mechanism this uses is standard across all of the other Dogsheep tools - it's actually designed so you can have one `auth.json` with a bunch of different credentials for different tools:\r\n\r\n```json\r\n{\r\n \"goodreads_personal_token\": \"...\",\r\n \"goodreads_user_id\": \"...\",\r\n \"github_personal_token\": \"...\",\r\n \"pocket_consumer_key\": \"...\",\r\n \"pocket_username\": \"...\",\r\n \"pocket_access_token\": \"...\"\r\n}\r\n```\r\nBut... `github-to-sqlite` does feel like it deserves a special case here, since it's such a good fit for running inside of GitHub Actions - which even provide a `GITHUB_TOKEN` for you to use!\r\n\r\nSo I don't think it will harm the family of tools too much if this has an environment variable alternative to the `-a` file.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 609950090, "label": "Fall back to authentication via ENV"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622279374", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33", "id": 622279374, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjI3OTM3NA==", "user": {"value": 2029, "label": "garethr"}, "created_at": "2020-05-01T07:12:47Z", "updated_at": "2020-05-01T07:12:47Z", "author_association": "NONE", "body": "I also go it working with:\r\n\r\n```yaml\r\nrun: echo ${{ secrets.github_token }} | github-to-sqlite auth\r\n```", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 609950090, "label": "Fall back to authentication via ENV"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622997410", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33", "id": 622997410, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk5NzQxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T18:46:10Z", "updated_at": "2020-05-02T18:46:10Z", "author_association": "MEMBER", "body": "Documented here: https://github.com/dogsheep/github-to-sqlite/blob/10fb34de41aaa35681f08b5991540d65bfcf2e2e/README.md#authentication", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 609950090, "label": "Fall back to authentication via ENV"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133298", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34", "id": 622133298, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjEzMzI5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T21:48:24Z", "updated_at": "2020-04-30T21:48:24Z", "author_association": "MEMBER", "body": "Unfortunately it's not available through any GitHub API - I managed to figure out how to get dependencies, but I need dependents. https://github.com/simonw/til/blob/master/github/dependencies-graphql-api.md", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610408908, "label": "Command for retrieving dependents for a repo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133422", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34", "id": 622133422, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjEzMzQyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T21:48:39Z", "updated_at": "2020-04-30T21:48:39Z", "author_association": "MEMBER", "body": "It looks like the only option is to scrape them. I'll do that and then replace with an API as soon as one becomes available.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610408908, "label": "Command for retrieving dependents for a repo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133775", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34", "id": 622133775, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjEzMzc3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T21:49:27Z", "updated_at": "2020-04-30T21:49:27Z", "author_association": "MEMBER", "body": "Proposed command:\r\n\r\n github-to-sqlite scrape-dependents github.db simonw/datasette\r\n\r\nI'll pull full details of the scraped repos from the regular API. I'll also record when they were \"first seen\" by the command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610408908, "label": "Command for retrieving dependents for a repo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622135654", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34", "id": 622135654, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjEzNTY1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T21:53:44Z", "updated_at": "2020-04-30T21:56:06Z", "author_association": "MEMBER", "body": "I think this is the neatest scraping pattern:\r\n```python\r\n[a[\"href\"].lstrip(\"/\") for a in soup.select(\"a[data-hovercard-type=repository]\")]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610408908, "label": "Command for retrieving dependents for a repo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622136585", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34", "id": 622136585, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjEzNjU4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T21:55:51Z", "updated_at": "2020-04-30T21:55:51Z", "author_association": "MEMBER", "body": "And to find the \"Next\" pagination link:\r\n```python\r\nsoup.select(\".paginate-container\")[0].find(\"a\", text=\"Next\")\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610408908, "label": "Command for retrieving dependents for a repo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622162835", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34", "id": 622162835, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjE2MjgzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T22:59:26Z", "updated_at": "2020-04-30T22:59:26Z", "author_association": "MEMBER", "body": "Documentation: https://github.com/dogsheep/github-to-sqlite/blob/c9f48404481882e8b3af06f35e4801a80ac79ed6/README.md#scraping-dependents-for-a-repository", "reactions": "{\"total_count\": 2, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 2, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610408908, "label": "Command for retrieving dependents for a repo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622213950", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35", "id": 622213950, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjIxMzk1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T02:09:04Z", "updated_at": "2020-05-01T02:09:04Z", "author_association": "MEMBER", "body": "It sped up this query a lot - 2.5s down to 300ms:\r\n```sql\r\nselect\r\n repos.full_name,\r\n json_object(\r\n 'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number,\r\n 'label', '#' || issues.number\r\n ) as issue,\r\n issues.title,\r\n users.login,\r\n users.id,\r\n issues.state,\r\n issues.locked,\r\n issues.assignee,\r\n issues.milestone,\r\n issues.comments,\r\n issues.created_at,\r\n issues.updated_at,\r\n issues.closed_at,\r\n issues.author_association,\r\n issues.pull_request,\r\n issues.repo,\r\n issues.type\r\nfrom\r\n issues\r\n join repos on repos.id = issues.repo\r\n join users on issues.user = users.id\r\nwhere\r\n issues.state = 'open'\r\n and issues.user not in (9599, 27856297)\r\n and not exists (\r\n select\r\n id\r\n from\r\n issue_comments\r\n where\r\n issue_comments.user = 9599\r\n and issues.id = issue_comments.issue\r\n )\r\norder by\r\n issues.updated_at desc;\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610511450, "label": "Create index on issue_comments(user) and other foreign keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622214262", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35", "id": 622214262, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjIxNDI2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T02:10:32Z", "updated_at": "2020-05-01T02:11:19Z", "author_association": "MEMBER", "body": "This sped that query up even more - down to 4ms.\r\n```sql\r\ncreate index issue_comments_issue on issue_comments(issue);\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610511450, "label": "Create index on issue_comments(user) and other foreign keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622982667", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35", "id": 622982667, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk4MjY2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T16:52:53Z", "updated_at": "2020-05-02T16:52:53Z", "author_association": "MEMBER", "body": "Easiest option: use `db.index_foreign_keys()`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#adding-indexes-for-all-foreign-keys", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610511450, "label": "Create index on issue_comments(user) and other foreign keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622461025", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36", "id": 622461025, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTAyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:34:24Z", "updated_at": "2020-05-01T16:34:24Z", "author_association": "MEMBER", "body": "Blocked on #37", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610842926, "label": "Add view for better display of dependent repos"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622982346", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36", "id": 622982346, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk4MjM0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T16:50:31Z", "updated_at": "2020-05-02T16:50:31Z", "author_association": "MEMBER", "body": "Demo: https://github-to-sqlite.dogsheep.net/github/dependent_repos", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610842926, "label": "Add view for better display of dependent repos"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461537", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37", "id": 622461537, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTUzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:35:40Z", "updated_at": "2020-05-01T16:35:40Z", "author_association": "MEMBER", "body": "This will check if the view exists and has the exact same matching definition as the one we want. If it doesn't, we will drop it (if it exists) and recreate it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610843136, "label": "Mechanism for creating views if they don't yet exist"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461948", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37", "id": 622461948, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTk0OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:36:42Z", "updated_at": "2020-05-01T16:36:42Z", "author_association": "MEMBER", "body": "It should only create views if the underlying tables exist.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610843136, "label": "Mechanism for creating views if they don't yet exist"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622978173", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37", "id": 622978173, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk3ODE3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T16:19:31Z", "updated_at": "2020-05-02T16:19:47Z", "author_association": "MEMBER", "body": "I can use the new `.create_view(..., replace=True)` parameter in `sqlite-utils` 2.7.2 for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610843136, "label": "Mechanism for creating views if they don't yet exist"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623027889", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38", "id": 623027889, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzAyNzg4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T23:15:11Z", "updated_at": "2020-05-02T23:15:11Z", "author_association": "MEMBER", "body": "This is one of the use-cases for the `repos_starred` view: it allows you to easily run this kid of query without having to construct the SQL by hand. Here's a demo:\r\n\r\nhttps://github-to-sqlite.dogsheep.net/github/repos_starred?name__contains=twitter\r\n\r\nMy philosophy here is to keep the raw tables (like `stars`) as normalized as possible, then use SQL views which expose the data in a form that's easier to query.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 611284481, "label": "[Feature Request] Support Repo Name in Search \ud83e\udd7a"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038148", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38", "id": 623038148, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzAzODE0OA==", "user": {"value": 5779832, "label": "zzeleznick"}, "created_at": "2020-05-03T01:18:57Z", "updated_at": "2020-05-03T01:18:57Z", "author_association": "NONE", "body": "Thanks, @simonw! \r\n\r\nI feel a little foolish in hindsight, but I'm on the same page now and am glad to have discovered first-hand a motivation for this `repos_starred` use case.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 611284481, "label": "[Feature Request] Support Repo Name in Search \ud83e\udd7a"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038378", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38", "id": 623038378, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzAzODM3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T01:21:13Z", "updated_at": "2020-05-03T01:21:13Z", "author_association": "MEMBER", "body": "No this is really useful feedback! I'm so close to this project that I miss what's not obvious to people dropping in for the first time.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 611284481, "label": "[Feature Request] Support Repo Name in Search \ud83e\udd7a"}, "performed_via_github_app": null}