This note includes two images.
...`
The DTD at http://xml.evernote.com/pub/enml2.dtd includes some entities:
```
%HTMLlat1;
%HTMLsymbol;
%HTMLspecial;
```
So I need to be able to handle all of those different entities. I think I can do that using `html.entities.entitydefs` from the Python standard library, which looks a bit like this:
```python
{'Aacute': 'Á',
'aacute': 'á',
'Aacute;': 'Á',
'aacute;': 'á',
'Abreve;': 'Ă',
'abreve;': 'ă',
'ac;': '∾',
'acd;': '∿',
# ...
}
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",748372469,
https://github.com/dogsheep/evernote-to-sqlite/pull/10#issuecomment-777839351,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10,777839351,MDEyOklzc3VlQ29tbWVudDc3NzgzOTM1MQ==,9599,2021-02-11T22:37:55Z,2021-02-11T22:37:55Z,MEMBER,"I've merged these changes by hand now, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770712149,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544646516,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,544646516,MDEyOklzc3VlQ29tbWVudDU0NDY0NjUxNg==,9599,2019-10-21T18:30:14Z,2019-10-21T18:30:14Z,MEMBER,"Thanks to help from Dr. Laura Cantino at Science Hack Day San Francisco I've been able to pull together this query:
```sql
select rsid, genotype, case genotype
when 'AA' then 'brown eye color, 80% of the time'
when 'AG' then 'brown eye color'
when 'GG' then 'blue eye color, 99% of the time'
end as interpretation from genome where rsid = 'rs12913832'
```
See also https://www.snpedia.com/index.php/Rs12913832 - in particular this table:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544648863,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,544648863,MDEyOklzc3VlQ29tbWVudDU0NDY0ODg2Mw==,9599,2019-10-21T18:36:03Z,2019-10-21T18:36:03Z,MEMBER,"![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765495861,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765495861,MDEyOklzc3VlQ29tbWVudDc2NTQ5NTg2MQ==,25372415,2021-01-22T15:44:00Z,2021-01-22T15:44:00Z,NONE,"Risk of autoimmune disorders: https://www.snpedia.com/index.php/Genotype
```
select rsid, genotype, case genotype
when 'AA' then '2x risk of rheumatoid arthritis and other autoimmune diseases'
when 'GG' then 'Normal risk for autoimmune disorders'
end as interpretation from genome where rsid = 'rs2476601'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765498984,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765498984,MDEyOklzc3VlQ29tbWVudDc2NTQ5ODk4NA==,25372415,2021-01-22T15:48:25Z,2021-01-22T15:49:33Z,NONE,"The ""Warrior Gene"" https://www.snpedia.com/index.php/Rs4680
```
select rsid, genotype, case genotype
when 'AA' then '(worrier) advantage in memory and attention tasks'
when 'AG' then 'Intermediate dopamine levels, other effects'
when 'GG' then '(warrior) multiple associations, see details'
end as interpretation from genome where rsid = 'rs4680'
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765502845,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765502845,MDEyOklzc3VlQ29tbWVudDc2NTUwMjg0NQ==,25372415,2021-01-22T15:53:19Z,2021-01-22T15:53:19Z,NONE,"rs7903146 Influences risk of Type-2 diabetes
https://www.snpedia.com/index.php/Rs7903146
```
select rsid, genotype, case genotype
when 'CC' then 'Normal (lower) risk of Type 2 Diabetes and Gestational Diabetes.'
when 'CT' then '1.4x increased risk for diabetes (and perhaps colon cancer).'
when 'TT' then '2x increased risk for Type-2 diabetes'
end as interpretation from genome where rsid = 'rs7903146'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765506901,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765506901,MDEyOklzc3VlQ29tbWVudDc2NTUwNjkwMQ==,25372415,2021-01-22T15:58:41Z,2021-01-22T15:58:58Z,NONE,"Both rs10757274 and rs2383206 can both indicate higher risks of heart disease
https://www.snpedia.com/index.php/Rs2383206
```
select rsid, genotype, case genotype
when 'AA' then 'Normal'
when 'AG' then '~1.2x increased risk for heart disease'
when 'GG' then '~1.3x increased risk for heart disease'
end as interpretation from genome where rsid = 'rs10757274'
```
```
select rsid, genotype, case genotype
when 'AA' then 'Normal'
when 'AG' then '1.4x increased risk for heart disease'
when 'GG' then '1.7x increased risk for heart disease'
end as interpretation from genome where rsid = 'rs2383206'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765523517,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765523517,MDEyOklzc3VlQ29tbWVudDc2NTUyMzUxNw==,25372415,2021-01-22T16:20:25Z,2021-01-22T16:20:25Z,NONE,"rs53576: the oxytocin receptor (OXTR) gene
```
select rsid, genotype, case genotype
when 'AA' then 'Lack of empathy?'
when 'AG' then 'Lack of empathy?'
when 'GG' then 'Optimistic and empathetic; handle stress well'
end as interpretation from genome where rsid = 'rs53576'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765525338,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765525338,MDEyOklzc3VlQ29tbWVudDc2NTUyNTMzOA==,25372415,2021-01-22T16:22:44Z,2021-01-22T16:22:44Z,NONE,"rs1333049 associated with coronary artery disease
https://www.snpedia.com/index.php/Rs1333049
```
select rsid, genotype, case genotype
when 'CC' then '1.9x increased risk for coronary artery disease'
when 'CG' then '1.5x increased risk for CAD'
when 'GG' then 'normal'
end as interpretation from genome where rsid = 'rs1333049'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-831004775,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,831004775,MDEyOklzc3VlQ29tbWVudDgzMTAwNDc3NQ==,25372415,2021-05-03T03:46:23Z,2021-05-03T03:46:23Z,NONE,"RS1800955 is related to novelty seeking and ADHD
https://www.snpedia.com/index.php/Rs1800955
`select rsid, genotype, case genotype
when 'CC' then 'increased susceptibility to novelty seeking'
when 'CT' then 'increased susceptibility to novelty seeking'
when 'TT' then 'normal'
end as interpretation from genome where rsid = 'rs1800955'`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,
https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-549230337,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10,549230337,MDEyOklzc3VlQ29tbWVudDU0OTIzMDMzNw==,9599,2019-11-04T05:47:18Z,2019-11-04T05:47:18Z,MEMBER,This definition isn't quite right - it's not pulling the identity of the user who starred the repo (`users.login` ends up being the owner login instead).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516967682,
https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622461122,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10,622461122,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTEyMg==,9599,2020-05-01T16:34:39Z,2020-05-01T16:34:39Z,MEMBER,Blocked on #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516967682,
https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622980203,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10,622980203,MDEyOklzc3VlQ29tbWVudDYyMjk4MDIwMw==,9599,2020-05-02T16:34:29Z,2020-05-02T16:34:29Z,MEMBER,"Fixed definition:
```sql
select
stars.starred_at,
starring_user.login as starred_by,
repos.*
from
repos
join stars on repos.id = stars.repo
join users as starring_user on stars.user = starring_user.id
join users on repos.owner = users.id
order by
starred_at desc;
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516967682,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594151327,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,594151327,MDEyOklzc3VlQ29tbWVudDU5NDE1MTMyNw==,9599,2020-03-03T20:26:15Z,2020-03-03T20:32:23Z,MEMBER,"Better version (since this also includes JSON array of repository topics):
```sql
CREATE VIEW recent_releases AS select
repos.rowid as rowid,
json_object(""label"", repos.full_name, ""href"", repos.html_url) as repo,
json_object(
""href"",
releases.html_url,
""label"",
releases.name
) as release,
substr(releases.published_at, 0, 11) as date,
releases.body as body_markdown,
releases.published_at,
coalesce(repos.topics, '[]') as topics
from
releases
join repos on repos.id = releases.repo
order by
releases.published_at desc
```
That `repos.rowid as rowid` bit is necessary because otherwise clicking on a link in facet-by-topic doesn't return any results.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594155249,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,594155249,MDEyOklzc3VlQ29tbWVudDU5NDE1NTI0OQ==,9599,2020-03-03T20:35:17Z,2020-03-03T20:35:17Z,MEMBER,"`swarm-to-sqlite` has an example of adding views here: https://github.com/dogsheep/swarm-to-sqlite/blob/f2c89dd613fb8a7f14e5267ccc2145463b996190/swarm_to_sqlite/utils.py#L141
I think that approach can be approved by first checking if the view exists, then dropping it, then recreating it. Could even try to see if the view exists and matches what we were going to set it to and do nothing if that is the case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,622461223,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw==,9599,2020-05-01T16:34:52Z,2020-05-01T16:34:52Z,MEMBER,Blocked on #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,622989874,MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA==,9599,2020-05-02T17:46:14Z,2020-05-02T17:46:14Z,MEMBER,Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,622998813,MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw==,9599,2020-05-02T18:58:17Z,2020-05-02T18:58:17Z,MEMBER,Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602861730,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602861730,MDEyOklzc3VlQ29tbWVudDYwMjg2MTczMA==,9599,2020-03-23T21:19:16Z,2020-03-23T21:19:42Z,MEMBER,I'm going to do this with a scheduled GitHub Action in this repo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862236,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602862236,MDEyOklzc3VlQ29tbWVudDYwMjg2MjIzNg==,9599,2020-03-23T21:20:26Z,2020-03-23T21:20:26Z,MEMBER,I'll run the `commits` and `issues` and `issue-comments` commands in addition to the `releases` command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862967,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602862967,MDEyOklzc3VlQ29tbWVudDYwMjg2Mjk2Nw==,9599,2020-03-23T21:22:04Z,2020-03-23T21:22:04Z,MEMBER,Following these instructions: https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602895896,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602895896,MDEyOklzc3VlQ29tbWVudDYwMjg5NTg5Ng==,9599,2020-03-23T22:42:25Z,2020-03-23T22:42:25Z,MEMBER,"Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602916947,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602916947,MDEyOklzc3VlQ29tbWVudDYwMjkxNjk0Nw==,9599,2020-03-23T23:38:06Z,2020-03-23T23:38:06Z,MEMBER,Woohoo! https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602917713,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602917713,MDEyOklzc3VlQ29tbWVudDYwMjkxNzcxMw==,9599,2020-03-23T23:40:29Z,2020-03-23T23:40:29Z,MEMBER,"Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602918689,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602918689,MDEyOklzc3VlQ29tbWVudDYwMjkxODY4OQ==,9599,2020-03-23T23:43:39Z,2020-03-23T23:47:50Z,MEMBER,I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602919058,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602919058,MDEyOklzc3VlQ29tbWVudDYwMjkxOTA1OA==,9599,2020-03-23T23:44:48Z,2020-03-23T23:44:48Z,MEMBER,Next step: use a `metadata.json` file to add some extras. And add the `datasette-render-markdown` plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602924714,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602924714,MDEyOklzc3VlQ29tbWVudDYwMjkyNDcxNA==,9599,2020-03-24T00:03:25Z,2020-03-24T00:03:25Z,MEMBER,This is good enough for the 1.0 release.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,
https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559883311,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14,559883311,MDEyOklzc3VlQ29tbWVudDU1OTg4MzMxMQ==,9599,2019-11-29T21:30:37Z,2019-11-29T21:30:37Z,MEMBER,"I should build the command to persist ETags and obey their polling guidelines:
> Events are optimized for polling with the ""ETag"" header. If no new events have been triggered, you will see a ""304 Not Modified"" response, and your current rate limit will be untouched. There is also an ""X-Poll-Interval"" header that specifies how often (in seconds) you are allowed to poll. In times of high server load, the time may increase. Please obey the header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",530491074,
https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559902818,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14,559902818,MDEyOklzc3VlQ29tbWVudDU1OTkwMjgxOA==,9599,2019-11-30T01:32:38Z,2019-11-30T01:32:38Z,MEMBER,"Prototype:
```
pip install sqlite-utils paginate-json
paginate-json ""https://api.github.com/users/simonw/events"" | sqlite-utils insert /tmp/events.db events - --pk=id
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",530491074,
https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-613641947,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14,613641947,MDEyOklzc3VlQ29tbWVudDYxMzY0MTk0Nw==,9599,2020-04-14T19:38:24Z,2020-04-14T19:38:34Z,MEMBER,"Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",530491074,
https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602309825,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15,602309825,MDEyOklzc3VlQ29tbWVudDYwMjMwOTgyNQ==,9599,2020-03-23T00:52:38Z,2020-03-23T00:52:38Z,MEMBER,Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the `releases` command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",544571092,
https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602311701,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15,602311701,MDEyOklzc3VlQ29tbWVudDYwMjMxMTcwMQ==,9599,2020-03-23T01:05:15Z,2020-03-23T01:06:14Z,MEMBER,"None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0
```
github-to-sqlite releases releases.db instrumenta/conftest
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",544571092,
https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602312054,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15,602312054,MDEyOklzc3VlQ29tbWVudDYwMjMxMjA1NA==,9599,2020-03-23T01:07:10Z,2020-03-23T01:07:10Z,MEMBER,"Each asset looks like this:
```json
{
""url"": ""https://api.github.com/repos/instrumenta/conftest/releases/assets/11811946"",
""id"": 11811946,
""node_id"": ""MDEyOlJlbGVhc2VBc3NldDExODExOTQ2"",
""name"": ""checksums.txt"",
""label"": """",
""uploader"": {
""login"": ""garethr"",
""id"": 2029,
""node_id"": ""MDQ6VXNlcjIwMjk="",
""avatar_url"": ""https://avatars2.githubusercontent.com/u/2029?v=4"",
""gravatar_id"": """",
""url"": ""https://api.github.com/users/garethr"",
""html_url"": ""https://github.com/garethr"",
""followers_url"": ""https://api.github.com/users/garethr/followers"",
""following_url"": ""https://api.github.com/users/garethr/following{/other_user}"",
""gists_url"": ""https://api.github.com/users/garethr/gists{/gist_id}"",
""starred_url"": ""https://api.github.com/users/garethr/starred{/owner}{/repo}"",
""subscriptions_url"": ""https://api.github.com/users/garethr/subscriptions"",
""organizations_url"": ""https://api.github.com/users/garethr/orgs"",
""repos_url"": ""https://api.github.com/users/garethr/repos"",
""events_url"": ""https://api.github.com/users/garethr/events{/privacy}"",
""received_events_url"": ""https://api.github.com/users/garethr/received_events"",
""type"": ""User"",
""site_admin"": false
},
""content_type"": ""text/plain; charset=utf-8"",
""state"": ""uploaded"",
""size"": 600,
""download_count"": 2,
""created_at"": ""2019-03-30T16:56:44Z"",
""updated_at"": ""2019-03-30T16:56:44Z"",
""browser_download_url"": ""https://github.com/instrumenta/conftest/releases/download/v0.1.0/checksums.txt""
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",544571092,
https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-605439685,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15,605439685,MDEyOklzc3VlQ29tbWVudDYwNTQzOTY4NQ==,2029,2020-03-28T12:17:01Z,2020-03-28T12:17:01Z,NONE,"That looks great, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",544571092,
https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-571412923,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16,571412923,MDEyOklzc3VlQ29tbWVudDU3MTQxMjkyMw==,15092,2020-01-07T03:06:46Z,2020-01-07T03:06:46Z,NONE,"I re-tried after doing `auth`, and I get the same result.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546051181,
https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-601979709,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16,601979709,MDEyOklzc3VlQ29tbWVudDYwMTk3OTcwOQ==,9599,2020-03-21T02:20:08Z,2020-03-21T02:20:08Z,MEMBER,I'm pretty sure this means I need to upgrade the underlying `sqlite-utils` dependency. I saw the same bug in `twitter-to-sqlite`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546051181,
https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-602136481,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16,602136481,MDEyOklzc3VlQ29tbWVudDYwMjEzNjQ4MQ==,15092,2020-03-22T02:08:57Z,2020-03-22T02:08:57Z,NONE,"I'd love to be using your library as a better cached gh layer for a new library I have built, replacing large parts of the very ugly https://github.com/jayvdb/pypidb/blob/master/pypidb/_github.py , and then probably being able to rebuild the setuppy chunk as a feature here at a later stage.
I would also need tokenless and netrc support, but I would be happy to add those bits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546051181,
https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-613611455,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16,613611455,MDEyOklzc3VlQ29tbWVudDYxMzYxMTQ1NQ==,9599,2020-04-14T18:37:21Z,2020-04-14T18:37:21Z,MEMBER,"This should have been fixed by #20 and #23
@jayvdb I'm definitely interested in this tool working as a library - it's purely designed as a CLI tool at the moment, but cleaning it up to work better as a dependency is totally in-scope for the project. https://sqlite-utils.readthedocs.io/ is an example of a tool I've built that works for both.
Feel free to open a new issue here with some notes on what you would need for this to work as a library for your project!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546051181,
https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597354514,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17,597354514,MDEyOklzc3VlQ29tbWVudDU5NzM1NDUxNA==,9599,2020-03-10T22:37:45Z,2020-03-10T22:37:45Z,MEMBER,I should add an option to stop the moment you see a commit you have fetched before.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",578883725,
https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597358364,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17,597358364,MDEyOklzc3VlQ29tbWVudDU5NzM1ODM2NA==,9599,2020-03-10T22:50:20Z,2020-03-11T01:18:36Z,MEMBER,By default it will stop when it sees a commit that has already been stored. You will be able to over-ride that behaviour using `--all`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",578883725,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980724,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,601980724,MDEyOklzc3VlQ29tbWVudDYwMTk4MDcyNA==,9599,2020-03-21T02:28:05Z,2020-03-21T02:28:05Z,MEMBER,"This suggests that `commit[""author""]` can be `None` in some cases?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980957,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,601980957,MDEyOklzc3VlQ29tbWVudDYwMTk4MDk1Nw==,9599,2020-03-21T02:30:03Z,2020-03-21T02:30:03Z,MEMBER,"https://github.community/t5/GitHub-API-Development-and/Request-for-commits-quot-author-quot-null-and-quot-committer/m-p/35842/highlight/true#M3372
> Commits aren't always associated with a GitHub user. For example, perhaps a friend of mine and I were working on a project together. I have a GitHub account and my friend doesn't. If we both add commits to the repository using our own email addresses and names and then I push the repository to GitHub, my commits will be associated with my GitHub user account but my friends' commits will show up with `author` and `committer` as `null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602807178,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602807178,MDEyOklzc3VlQ29tbWVudDYwMjgwNzE3OA==,9599,2020-03-23T19:24:43Z,2020-03-23T19:24:43Z,MEMBER,I need to find an example before I work on this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811468,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602811468,MDEyOklzc3VlQ29tbWVudDYwMjgxMTQ2OA==,9599,2020-03-23T19:33:14Z,2020-03-23T19:33:14Z,MEMBER,Found one: https://api.github.com/repos/simonw/simonw.github.com/commits,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811785,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602811785,MDEyOklzc3VlQ29tbWVudDYwMjgxMTc4NQ==,9599,2020-03-23T19:33:56Z,2020-03-23T19:34:06Z,MEMBER,"```json
[
{
""sha"": ""a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0"",
""node_id"": ""MDY6Q29tbWl0OTMyMDk6YThkYzkxNDA4OWQzOTlkOWI1MjJlYmI1MWI2N2Y5YWMyZThhYTZiMA=="",
""commit"": {
""author"": {
""name"": ""Simon Willison"",
""email"": ""simon@..."",
""date"": ""2008-12-18T23:17:12Z""
},
""committer"": {
""name"": ""Simon Willison"",
""email"": ""simon@..."",
""date"": ""2008-12-18T23:17:12Z""
},
""message"": ""First commit"",
""tree"": {
""sha"": ""ac2dfb75e2592c59165c2880f3f7a16dafd452a1"",
""url"": ""https://api.github.com/repos/simonw/simonw.github.com/git/trees/ac2dfb75e2592c59165c2880f3f7a16dafd452a1""
},
""url"": ""https://api.github.com/repos/simonw/simonw.github.com/git/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0"",
""comment_count"": 0,
""verification"": {
""verified"": false,
""reason"": ""unsigned"",
""signature"": null,
""payload"": null
}
},
""url"": ""https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0"",
""html_url"": ""https://github.com/simonw/simonw.github.com/commit/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0"",
""comments_url"": ""https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0/comments"",
""author"": null,
""committer"": null,
""parents"": []
}
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602813731,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602813731,MDEyOklzc3VlQ29tbWVudDYwMjgxMzczMQ==,9599,2020-03-23T19:38:03Z,2020-03-23T20:44:49Z,MEMBER,"So it turns out `""author""` and `""committer""` on the commit are `null` if the email address in the nested `""commit""` doesn't match an existing GitHub user.
Maybe I should be storing the nested data somewhere as well?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602815120,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602815120,MDEyOklzc3VlQ29tbWVudDYwMjgxNTEyMA==,9599,2020-03-23T19:40:55Z,2020-03-23T19:43:19Z,MEMBER,I could pull a pk-hashed version of the name/email into separate `raw_author` and `raw_committer` columns perhaps - against a `commit_authors` table. Could be interesting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602846293,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602846293,MDEyOklzc3VlQ29tbWVudDYwMjg0NjI5Mw==,9599,2020-03-23T20:44:40Z,2020-03-23T20:44:40Z,MEMBER,I implemented the `raw_authors` idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,
https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602304531,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19,602304531,MDEyOklzc3VlQ29tbWVudDYwMjMwNDUzMQ==,9599,2020-03-23T00:20:28Z,2020-03-23T00:20:28Z,MEMBER,"Current FTS code:
https://github.com/dogsheep/github-to-sqlite/blob/e35eec4343aa560c58c1634cc228d0d46c442304/github_to_sqlite/utils.py#L136-L143","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585850715,
https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602782437,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19,602782437,MDEyOklzc3VlQ29tbWVudDYwMjc4MjQzNw==,9599,2020-03-23T18:36:12Z,2020-03-23T18:51:23Z,MEMBER,"* `commits`: `message`
* `issue_comments`: `body`
* `issues`: `title`, `body`
* `labels`: `name`, `description`
* `licenses`: `name`
* `milestones`: `title`, `description`
* `releases`: `name`, `body`
* `repos`: `name`, `description`
* `users`: `login`, `name` (maybe `company`, `bio`)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585850715,
https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602896434,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21,602896434,MDEyOklzc3VlQ29tbWVudDYwMjg5NjQzNA==,9599,2020-03-23T22:43:37Z,2020-03-23T22:43:37Z,MEMBER,"I'm going to do this now to help figure out the latest error in #13:
```
Traceback (most recent call last):
File ""/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite"", line 11, in
load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')()
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 237, in commits
utils.save_commits(db, commits, repo_full[""id""])
File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 345, in save_commits
for commit in commits:
File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 207, in fetch_commits
if stop_when(commit):
File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 224, in stop_when
db[""commits""].get(commit[""sha""])
TypeError: string indices must be integers
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",586561727,
https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602920163,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21,602920163,MDEyOklzc3VlQ29tbWVudDYwMjkyMDE2Mw==,9599,2020-03-23T23:48:22Z,2020-03-23T23:48:22Z,MEMBER,"I'm happy with this pattern:
https://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",586561727,
https://github.com/dogsheep/github-to-sqlite/issues/23#issuecomment-602928533,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23,602928533,MDEyOklzc3VlQ29tbWVudDYwMjkyODUzMw==,9599,2020-03-24T00:15:49Z,2020-03-24T00:15:49Z,MEMBER,https://github.com/dogsheep/github-to-sqlite/releases/tag/1.0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",586595839,
https://github.com/dogsheep/github-to-sqlite/issues/25#issuecomment-614810417,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25,614810417,MDEyOklzc3VlQ29tbWVudDYxNDgxMDQxNw==,9599,2020-04-16T18:07:11Z,2020-04-16T18:07:11Z,MEMBER,Turns out the main problem was #26 - now fixed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601265023,
https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614794739,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26,614794739,MDEyOklzc3VlQ29tbWVudDYxNDc5NDczOQ==,9599,2020-04-16T17:38:28Z,2020-04-16T17:38:28Z,MEMBER,I'm already doing this here: https://github.com/dogsheep/github-to-sqlite/blob/c4aaa50e167cfa9021c7c94260bc3e89e10947bf/github_to_sqlite/utils.py#L246-L250,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601271612,
https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614795712,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26,614795712,MDEyOklzc3VlQ29tbWVudDYxNDc5NTcxMg==,9599,2020-04-16T17:40:27Z,2020-04-16T17:40:27Z,MEMBER,Aha! it was missing from the `fetch_repo()` function.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601271612,
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831451,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,614831451,MDEyOklzc3VlQ29tbWVudDYxNDgzMTQ1MQ==,9599,2020-04-16T18:47:25Z,2020-04-16T18:47:25Z,MEMBER,Is it possible for a repo to have an `owner` that differs from its `organization`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831842,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,614831842,MDEyOklzc3VlQ29tbWVudDYxNDgzMTg0Mg==,9599,2020-04-16T18:48:18Z,2020-04-16T18:48:18Z,MEMBER,I'm going to make `organization` another foreign key to the `users` table just in case it IS possible (maybe with GitHub Enterprise or similar?),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,614843406,MDEyOklzc3VlQ29tbWVudDYxNDg0MzQwNg==,9599,2020-04-16T19:11:53Z,2020-04-16T19:20:23Z,MEMBER,"This didn't quite work: the column type is incorrect, so the foreign key relationship isn't sticking:
https://github-to-sqlite.dogsheep.net/github/repos?organization=53015001
`[organization] TEXT REFERENCES [users]([id])` - should be `INTEGER`.
The problem is that if the first repo inserted has no organization it's set to `null`, which `sqlite-utils` derives as a `TEXT` column.
One solution would be to create the column explicitly with a type, but this could get messy.
I think I want a new sqlite-utils feature for this instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615518606,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,615518606,MDEyOklzc3VlQ29tbWVudDYxNTUxODYwNg==,9599,2020-04-18T00:14:32Z,2020-04-18T00:14:32Z,MEMBER,https://github.com/simonw/sqlite-utils/issues/100 is done and released in sqlite-utils 2.7.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615519409,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,615519409,MDEyOklzc3VlQ29tbWVudDYxNTUxOTQwOQ==,9599,2020-04-18T00:19:16Z,2020-04-18T00:19:16Z,MEMBER,"```
$ github-to-sqlite repos b.db dogsheep
$ sqlite3 b.db '.schema repos'
CREATE TABLE [repos] (
[id] INTEGER PRIMARY KEY,
...
[permissions] TEXT,
[organization] INTEGER REFERENCES [users]([id]),
FOREIGN KEY(license) REFERENCES licenses(key)
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,
https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883040,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28,615883040,MDEyOklzc3VlQ29tbWVudDYxNTg4MzA0MA==,9599,2020-04-18T14:45:38Z,2020-04-18T14:45:38Z,MEMBER,"```
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 219, in contributors
utils.save_contributors(db, contributors, repo_full[""id""])
File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 354, in save_contributors
for contributor in contributors:
File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 228, in fetch_contributors
for contributors in paginate(url, headers):
File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 286, in paginate
data = response.json()
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/requests/models.py"", line 898, in json
return complexjson.loads(self.text, **kwargs)
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/__init__.py"", line 357, in loads
return _default_decoder.decode(s)
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py"", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py"", line 355, in raw_decode
raise JSONDecodeError(""Expecting value"", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601333634,
https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883687,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28,615883687,MDEyOklzc3VlQ29tbWVudDYxNTg4MzY4Nw==,9599,2020-04-18T14:49:58Z,2020-04-18T14:49:58Z,MEMBER,"That happened trying to pull contributors for `dogsheep/beta` - an empty repository. Turns out it was returning a `204 no content`:
```
~ $ curl -i 'https://api.github.com/repos/dogsheep/beta/contributors'
HTTP/1.1 204 No Content
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601333634,
https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615886206,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28,615886206,MDEyOklzc3VlQ29tbWVudDYxNTg4NjIwNg==,9599,2020-04-18T15:04:59Z,2020-04-18T15:04:59Z,MEMBER,"Demo: https://github-to-sqlite.dogsheep.net/github/contributors
Documentation: https://github.com/dogsheep/github-to-sqlite/blob/13f8868fb5efa01c263b24f6dd91c617e6e938e1/README.md#fetching-contributors-to-a-repository ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601333634,
https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29,616883275,MDEyOklzc3VlQ29tbWVudDYxNjg4MzI3NQ==,9599,2020-04-21T00:43:28Z,2020-04-21T00:43:28Z,MEMBER,"I'm copying repo from issue, which surprisingly is a string, not an integer ID.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603617013,
https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531516956,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3,531516956,MDEyOklzc3VlQ29tbWVudDUzMTUxNjk1Ng==,9599,2019-09-14T21:56:31Z,2019-09-14T21:56:31Z,MEMBER,"https://api.github.com/users/simonw/repos
It would be useful to be able to fetch stargazers, forks etc as well. Not sure if that should be a separate command or a `--stargazers` option to this command.
Probably a separate command since `issues` is a separate command already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670426,
https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531517083,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3,531517083,MDEyOklzc3VlQ29tbWVudDUzMTUxNzA4Mw==,9599,2019-09-14T21:58:42Z,2019-09-14T21:58:42Z,MEMBER,Split stargazers into #4,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670426,
https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616879753,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30,616879753,MDEyOklzc3VlQ29tbWVudDYxNjg3OTc1Mw==,9599,2020-04-21T00:29:29Z,2020-04-21T00:29:29Z,MEMBER,`assignee` looks like it's the incorrect type too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603618244,
https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616883726,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30,616883726,MDEyOklzc3VlQ29tbWVudDYxNjg4MzcyNg==,9599,2020-04-21T00:45:23Z,2020-04-21T00:45:23Z,MEMBER,"Demo of fix: https://github-to-sqlite.dogsheep.net/github/issues?assignee__notblank=1&milestone__notblank=1
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603618244,
https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-616884647,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,616884647,MDEyOklzc3VlQ29tbWVudDYxNjg4NDY0Nw==,9599,2020-04-21T00:49:16Z,2020-04-21T00:50:20Z,MEMBER,"The API just gives us the `repository_url`: https://api.github.com/repos/simonw/datasette/issues
We currently turn that into a `simonw/datasette` string here:
https://github.com/dogsheep/github-to-sqlite/blob/e0e8d8caa9657b04bfb8a2cf16c9b580f38b1805/github_to_sqlite/utils.py#L43-L46","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862,
https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617348174,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,617348174,MDEyOklzc3VlQ29tbWVudDYxNzM0ODE3NA==,9599,2020-04-21T18:50:29Z,2020-04-21T18:50:29Z,MEMBER,"Since this represents a breaking schema change for anyone running SQL queries against these tables, I'm going to do a major version bump to 2.0 when I release this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862,
https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617491607,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,617491607,MDEyOklzc3VlQ29tbWVudDYxNzQ5MTYwNw==,9599,2020-04-22T01:20:19Z,2020-04-22T01:20:19Z,MEMBER,"https://github-to-sqlite.dogsheep.net/github/milestones now link to repo:
![]()
And so do issues: https://github-to-sqlite.dogsheep.net/github/issues
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862,
https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617364956,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617364956,MDEyOklzc3VlQ29tbWVudDYxNzM2NDk1Ng==,9599,2020-04-21T19:24:45Z,2020-04-21T19:24:45Z,MEMBER,"That's because I just broke this code:
https://github.com/dogsheep/github-to-sqlite/blob/2cf75a0a036719eb7e57fdc7c5c2ea0f4c26978a/github_to_sqlite/utils.py#L131-L139
It expects the `repo` column to be `simonw/datasette` but it's now an ID instead.
I should add a test for this as part of the fix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295,
https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617369247,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617369247,MDEyOklzc3VlQ29tbWVudDYxNzM2OTI0Nw==,9599,2020-04-21T19:33:03Z,2020-04-21T19:33:03Z,MEMBER,Caused by #31.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295,
https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617490914,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617490914,MDEyOklzc3VlQ29tbWVudDYxNzQ5MDkxNA==,9599,2020-04-22T01:17:44Z,2020-04-22T01:17:44Z,MEMBER,"https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101 now shows issues.
And https://github-to-sqlite.dogsheep.net/github/issue_comments links to them:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622169728,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622169728,MDEyOklzc3VlQ29tbWVudDYyMjE2OTcyOA==,9599,2020-04-30T23:18:51Z,2020-04-30T23:18:51Z,MEMBER,"Sure, that sounds fine to me.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622171097,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622171097,MDEyOklzc3VlQ29tbWVudDYyMjE3MTA5Nw==,9599,2020-04-30T23:22:45Z,2020-04-30T23:23:57Z,MEMBER,"The `auth.json` mechanism this uses is standard across all of the other Dogsheep tools - it's actually designed so you can have one `auth.json` with a bunch of different credentials for different tools:
```json
{
""goodreads_personal_token"": ""..."",
""goodreads_user_id"": ""..."",
""github_personal_token"": ""..."",
""pocket_consumer_key"": ""..."",
""pocket_username"": ""..."",
""pocket_access_token"": ""...""
}
```
But... `github-to-sqlite` does feel like it deserves a special case here, since it's such a good fit for running inside of GitHub Actions - which even provide a `GITHUB_TOKEN` for you to use!
So I don't think it will harm the family of tools too much if this has an environment variable alternative to the `-a` file.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622279374,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622279374,MDEyOklzc3VlQ29tbWVudDYyMjI3OTM3NA==,2029,2020-05-01T07:12:47Z,2020-05-01T07:12:47Z,NONE,"I also go it working with:
```yaml
run: echo ${{ secrets.github_token }} | github-to-sqlite auth
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622997410,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622997410,MDEyOklzc3VlQ29tbWVudDYyMjk5NzQxMA==,9599,2020-05-02T18:46:10Z,2020-05-02T18:46:10Z,MEMBER,Documented here: https://github.com/dogsheep/github-to-sqlite/blob/10fb34de41aaa35681f08b5991540d65bfcf2e2e/README.md#authentication,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133298,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622133298,MDEyOklzc3VlQ29tbWVudDYyMjEzMzI5OA==,9599,2020-04-30T21:48:24Z,2020-04-30T21:48:24Z,MEMBER,"Unfortunately it's not available through any GitHub API - I managed to figure out how to get dependencies, but I need dependents. https://github.com/simonw/til/blob/master/github/dependencies-graphql-api.md","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133422,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622133422,MDEyOklzc3VlQ29tbWVudDYyMjEzMzQyMg==,9599,2020-04-30T21:48:39Z,2020-04-30T21:48:39Z,MEMBER,It looks like the only option is to scrape them. I'll do that and then replace with an API as soon as one becomes available.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133775,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622133775,MDEyOklzc3VlQ29tbWVudDYyMjEzMzc3NQ==,9599,2020-04-30T21:49:27Z,2020-04-30T21:49:27Z,MEMBER,"Proposed command:
github-to-sqlite scrape-dependents github.db simonw/datasette
I'll pull full details of the scraped repos from the regular API. I'll also record when they were ""first seen"" by the command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622135654,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622135654,MDEyOklzc3VlQ29tbWVudDYyMjEzNTY1NA==,9599,2020-04-30T21:53:44Z,2020-04-30T21:56:06Z,MEMBER,"I think this is the neatest scraping pattern:
```python
[a[""href""].lstrip(""/"") for a in soup.select(""a[data-hovercard-type=repository]"")]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622136585,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622136585,MDEyOklzc3VlQ29tbWVudDYyMjEzNjU4NQ==,9599,2020-04-30T21:55:51Z,2020-04-30T21:55:51Z,MEMBER,"And to find the ""Next"" pagination link:
```python
soup.select("".paginate-container"")[0].find(""a"", text=""Next"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622162835,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622162835,MDEyOklzc3VlQ29tbWVudDYyMjE2MjgzNQ==,9599,2020-04-30T22:59:26Z,2020-04-30T22:59:26Z,MEMBER,Documentation: https://github.com/dogsheep/github-to-sqlite/blob/c9f48404481882e8b3af06f35e4801a80ac79ed6/README.md#scraping-dependents-for-a-repository,"{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 2, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,
https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622213950,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622213950,MDEyOklzc3VlQ29tbWVudDYyMjIxMzk1MA==,9599,2020-05-01T02:09:04Z,2020-05-01T02:09:04Z,MEMBER,"It sped up this query a lot - 2.5s down to 300ms:
```sql
select
repos.full_name,
json_object(
'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number,
'label', '#' || issues.number
) as issue,
issues.title,
users.login,
users.id,
issues.state,
issues.locked,
issues.assignee,
issues.milestone,
issues.comments,
issues.created_at,
issues.updated_at,
issues.closed_at,
issues.author_association,
issues.pull_request,
issues.repo,
issues.type
from
issues
join repos on repos.id = issues.repo
join users on issues.user = users.id
where
issues.state = 'open'
and issues.user not in (9599, 27856297)
and not exists (
select
id
from
issue_comments
where
issue_comments.user = 9599
and issues.id = issue_comments.issue
)
order by
issues.updated_at desc;
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450,
https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622214262,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622214262,MDEyOklzc3VlQ29tbWVudDYyMjIxNDI2Mg==,9599,2020-05-01T02:10:32Z,2020-05-01T02:11:19Z,MEMBER,"This sped that query up even more - down to 4ms.
```sql
create index issue_comments_issue on issue_comments(issue);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450,
https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622982667,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622982667,MDEyOklzc3VlQ29tbWVudDYyMjk4MjY2Nw==,9599,2020-05-02T16:52:53Z,2020-05-02T16:52:53Z,MEMBER,Easiest option: use `db.index_foreign_keys()`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#adding-indexes-for-all-foreign-keys,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450,
https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622461025,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36,622461025,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTAyNQ==,9599,2020-05-01T16:34:24Z,2020-05-01T16:34:24Z,MEMBER,Blocked on #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610842926,
https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622982346,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36,622982346,MDEyOklzc3VlQ29tbWVudDYyMjk4MjM0Ng==,9599,2020-05-02T16:50:31Z,2020-05-02T16:50:31Z,MEMBER,Demo: https://github-to-sqlite.dogsheep.net/github/dependent_repos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610842926,
https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461537,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622461537,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTUzNw==,9599,2020-05-01T16:35:40Z,2020-05-01T16:35:40Z,MEMBER,"This will check if the view exists and has the exact same matching definition as the one we want. If it doesn't, we will drop it (if it exists) and recreate it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136,
https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461948,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622461948,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTk0OA==,9599,2020-05-01T16:36:42Z,2020-05-01T16:36:42Z,MEMBER,It should only create views if the underlying tables exist.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136,
https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622978173,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622978173,MDEyOklzc3VlQ29tbWVudDYyMjk3ODE3Mw==,9599,2020-05-02T16:19:31Z,2020-05-02T16:19:47Z,MEMBER,"I can use the new `.create_view(..., replace=True)` parameter in `sqlite-utils` 2.7.2 for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136,
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623027889,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623027889,MDEyOklzc3VlQ29tbWVudDYyMzAyNzg4OQ==,9599,2020-05-02T23:15:11Z,2020-05-02T23:15:11Z,MEMBER,"This is one of the use-cases for the `repos_starred` view: it allows you to easily run this kid of query without having to construct the SQL by hand. Here's a demo:
https://github-to-sqlite.dogsheep.net/github/repos_starred?name__contains=twitter
My philosophy here is to keep the raw tables (like `stars`) as normalized as possible, then use SQL views which expose the data in a form that's easier to query.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481,
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038148,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623038148,MDEyOklzc3VlQ29tbWVudDYyMzAzODE0OA==,5779832,2020-05-03T01:18:57Z,2020-05-03T01:18:57Z,NONE,"Thanks, @simonw!
I feel a little foolish in hindsight, but I'm on the same page now and am glad to have discovered first-hand a motivation for this `repos_starred` use case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481,
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038378,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623038378,MDEyOklzc3VlQ29tbWVudDYyMzAzODM3OA==,9599,2020-05-03T01:21:13Z,2020-05-03T01:21:13Z,MEMBER,No this is really useful feedback! I'm so close to this project that I miss what's not obvious to people dropping in for the first time.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481,
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623044643,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623044643,MDEyOklzc3VlQ29tbWVudDYyMzA0NDY0Mw==,5779832,2020-05-03T02:34:32Z,2020-05-03T02:34:32Z,NONE,"1. More than glad to share feedback from the sidelines as a [starrer](https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++starred_at%2C%0D%0A++starred_by%2C%0D%0A++full_name+as+repo_name%0D%0Afrom%0D%0A++repos_starred%0D%0Awhere%0D%0A++starred_by+%3D+%22zzeleznick%22%0D%0Aorder+by%0D%0A++starred_at+desc).
```
-- Motivation:
-- Datasette is a data hammer and I'm looking for nails
-- e.g. Find which repos a user has starred => trigger a TBD downstream action
select
starred_at,
starred_by,
full_name as repo_name
from
repos_starred
where
starred_by = ""zzeleznick""
order by
starred_at desc
```
| starred_at | starred_by | repo_name |
| --- | --- | --- |
| 2020-02-11T01:08:59Z | zzeleznick | dogsheep/twitter-to-sqlite |
| 2020-01-11T21:57:34Z | zzeleznick | simonw/datasette |
2. In my day job, I use [airflow](https://github.com/apache/airflow), and that's the mental model I'm bringing to [datasette](https://github.com/simonw/datasette).
3. I see your project like [twitter-to-sqlite](https://github.com/dogsheep/twitter-to-sqlite) akin to [Operators](https://airflow.apache.org/docs/stable/_api/index.html#pythonapi-operators) in Airflow world.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481,