issue_comments
9,947 rows sorted by issue_url
This data as json, CSV (advanced)
id | html_url | issue_url ▼ | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
706776680 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776680 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjc3NjY4MA== | simonw 9599 | 2020-10-11T22:22:16Z | 2020-10-11T22:22:16Z | MEMBER | Maybe the best way do this is with a custom route, `/-/evernote/note-id` - that way I can clean the HTML and resolve the other things in the `<en-note>` structure without using `render_cell()` and the like. My concern about using `render_cell()` is that it could lead to weird security problems when combined with `?sql=` queries. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
706776808 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776808 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjc3NjgwOA== | simonw 9599 | 2020-10-11T22:23:14Z | 2020-10-11T22:23:14Z | MEMBER | ... but it's still important to be able to get to the rendered note directly from the browse notes `/evernote/notes` page. Maybe use a simple `render_cell()` hook that just knows how to generate the link to the rendered note page? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
706834800 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706834800 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjgzNDgwMA== | simonw 9599 | 2020-10-12T03:24:57Z | 2020-10-16T20:16:28Z | MEMBER | Here's my first attempt at a plugin for this: ```python from datasette import hookimpl import jinja2 START = "<en-note" END = "</en-note>" TEMPLATE = """ <div style="max-width: 500px; white-space: normal; overflow-wrap: break-word;">{}</div> """.strip() EN_MEDIA_SCRIPT = """ Array.from(document.querySelectorAll('en-media')).forEach(el => { let hash = el.getAttribute('hash'); let type = el.getAttribute('type'); let path = `/evernote/resources_data/${hash}.json?_shape=array`; fetch(path).then(r => r.json()).then(rows => { let b64 = rows[0].data.encoded; let data = `data:${type};base64,${b64}`; el.innerHTML = `<img style="max-width: 300px" src="${data}">`; }); }); """ @hookimpl def render_cell(value, table): if not table: # Don't render content from arbitrary SQL queries, could be XSS hole return if not value or not isinstance(value, str): return value = value.strip() if value.startswith(START) and value.endswith(END): trimmed = value[len(START) : -len(END)] trimmed = trimmed.split(">", 1)[1] # Replace those horrible double newlines trimmed = trimmed.replace("<div><br /></div>", "<br>") return jinja2.Markup(TEMPLATE.format(trimmed)) @hookimpl def extra_body_script(): return EN_MEDIA_SCRIPT ``` It works! It does however demonstrate that Evernote's "clip this webpage" feature means there is a LOT of weird HTML that can get into a note. It looks like they've filtered out the scripts but I wouldn't bet on it - they certainly don't filter out many of the inline styles. So running Bleach is almost certainly a good idea. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
706785086 | https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785086 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDcwNjc4NTA4Ng== | simonw 9599 | 2020-10-11T23:28:50Z | 2020-10-11T23:28:50Z | MEMBER | The XML for the OCR stuff is a bit weird. Currently I'm doing this to it: https://github.com/dogsheep/evernote-to-sqlite/blob/c33d7b043a45eb3e88676e5fa3ce31755199d9f8/evernote_to_sqlite/utils.py#L70-L78 This can produce some odd results, for example: > Sure 'Sure, 'Sure. Sure, Sure. sure sure. sure ? If you If Yau [you live jive In m 1n an area devoid of natural wonders, wanders, wonders ? wonders wonders. your mind will be blown, blown' blown. blown ? -e i ? ,1 IL it ? at ? KY ? fl ft bat at Which came from this image: ![image](https://user-images.githubusercontent.com/9599/95692952-5dd7c880-0bde-11eb-939a-d10b800a4105.png) The XML for that is: ```xml <recoIndex docType="unknown" objType="image" objID="05ffb72b307bf495f064243c7099d94f" engineVersion="6.5.17.7" recoType="service" lang="en" objWidth="1000" objHeight="1504"> <item x="68" y="75" w="104" h="37"> <t w="60">Sure</t> <t w="52">'Sure,</t> <t w="47">'Sure.</t> <t w="33">Sure,</t> <t w="26">Sure.</t> </item> <item x="182" y="83" w="92" h="26"> <t w="62">sure</t> <t w="58">sure.</t> <t w="46">sure ?</t> </item> <item x="69" y="132" w="107" h="45"> <t w="81">If you</t> <t w="64">If Yau</t> <t w="31">[you</t> </item> <item x="186" y="132" w="67" h="35"> <t w="85">live</t> <t w="51">jive</t> </item> <item x="263" y="140" w="36" h="27"> <t w="82">In</t> <t w="56">m</t> <t w="53">1n</t> </item> <item x="309" y="140" w="53" h="27"> <t w="82">an</t> </item> <item x="372" y="141" w="90" h="26"> <t w="94">area</t> </item> <item x="472" y="132" w="138" h="35"> <t w="85">devoid</t> </item> <item x="620" y="132" w="43" h="35"> <t w="82">of</t> </item> <item x="68" y="190" w="137" h="35"> <t w="87">natural</t> </item> <item x="215" y="190" w="187" h="39"> <t w="57">wonders,</t> <t w="55">wanders,</t> <t w="52">wonders ?</t> <t w="45">wonders</t> <t w="42">won… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Better handling of OCR data 718949182 | |
706785201 | https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785201 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDcwNjc4NTIwMQ== | simonw 9599 | 2020-10-11T23:29:39Z | 2020-10-11T23:29:39Z | MEMBER | It looks to me like each of those `<item>` blocks has a number of guesses in order of confidence: ```xml <item x="215" y="190" w="187" h="39"> <t w="57">wonders,</t> <t w="55">wanders,</t> <t w="52">wonders ?</t> <t w="45">wonders</t> <t w="42">wonders.</t> </item> ``` So maybe the best approach here is to just take the first `t` element within each `item`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Better handling of OCR data 718949182 | |
777827396 | https://github.com/dogsheep/evernote-to-sqlite/issues/7#issuecomment-777827396 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7 | MDEyOklzc3VlQ29tbWVudDc3NzgyNzM5Ng== | simonw 9599 | 2021-02-11T22:13:14Z | 2021-02-11T22:13:14Z | MEMBER | My best guess is that you have an older version of `sqlite-utils` installed here - the `replace=True` argument was added in version 2.0. I've bumped the dependency in `setup.py`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace' 743297582 | |
777821383 | https://github.com/dogsheep/evernote-to-sqlite/issues/9#issuecomment-777821383 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9 | MDEyOklzc3VlQ29tbWVudDc3NzgyMTM4Mw== | simonw 9599 | 2021-02-11T22:01:28Z | 2021-02-11T22:01:28Z | MEMBER | Aha! I think I've figured out what's going on here. The CData blocks containing the notes look like this: `<![CDATA[<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd"><en-note><div>This note includes two images.</div><div><br /></div>...` The DTD at http://xml.evernote.com/pub/enml2.dtd includes some entities: ``` <!--=========== External character mnemonic entities ===================--> <!ENTITY % HTMLlat1 PUBLIC "-//W3C//ENTITIES Latin 1 for XHTML//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent"> %HTMLlat1; <!ENTITY % HTMLsymbol PUBLIC "-//W3C//ENTITIES Symbols for XHTML//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent"> %HTMLsymbol; <!ENTITY % HTMLspecial PUBLIC "-//W3C//ENTITIES Special for XHTML//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent"> %HTMLspecial; ``` So I need to be able to handle all of those different entities. I think I can do that using `html.entities.entitydefs` from the Python standard library, which looks a bit like this: ```python {'Aacute': 'Á', 'aacute': 'á', 'Aacute;': 'Á', 'aacute;': 'á', 'Abreve;': 'Ă', 'abreve;': 'ă', 'ac;': '∾', 'acd;': '∿', # ... } ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ParseError: undefined entity š 748372469 | |
544646516 | https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544646516 | https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDU0NDY0NjUxNg== | simonw 9599 | 2019-10-21T18:30:14Z | 2019-10-21T18:30:14Z | MEMBER | Thanks to help from Dr. Laura Cantino at Science Hack Day San Francisco I've been able to pull together this query: ```sql select rsid, genotype, case genotype when 'AA' then 'brown eye color, 80% of the time' when 'AG' then 'brown eye color' when 'GG' then 'blue eye color, 99% of the time' end as interpretation from genome where rsid = 'rs12913832' ``` See also https://www.snpedia.com/index.php/Rs12913832 - in particular this table: <img width="321" alt="rs12913832_-_SNPedia" src="https://user-images.githubusercontent.com/9599/67232392-216ff300-f3f6-11e9-8e14-b5f50c0c0d16.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out some interesting example SQL queries 496415321 | |
544648863 | https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544648863 | https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDU0NDY0ODg2Mw== | simonw 9599 | 2019-10-21T18:36:03Z | 2019-10-21T18:36:03Z | MEMBER | <img width="1418" alt="natalie__select_rsid__genotype__case_genotype_when__AA__then__brown_eye_color__80__of_the_time__when__AG__then__brown_eye_color__when__GG__then__blue_eye_color__99__of_the_time__end_as_interpretation_from_genome_where_rsid____rs12913832__an" src="https://user-images.githubusercontent.com/9599/67232810-f4701000-f3f6-11e9-90e2-8fe2cca1d98d.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out some interesting example SQL queries 496415321 | |
765495861 | https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765495861 | https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDc2NTQ5NTg2MQ== | cobiadigital 25372415 | 2021-01-22T15:44:00Z | 2021-01-22T15:44:00Z | NONE | Risk of autoimmune disorders: https://www.snpedia.com/index.php/Genotype ``` select rsid, genotype, case genotype when 'AA' then '2x risk of rheumatoid arthritis and other autoimmune diseases' when 'GG' then 'Normal risk for autoimmune disorders' end as interpretation from genome where rsid = 'rs2476601' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out some interesting example SQL queries 496415321 | |
765498984 | https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765498984 | https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDc2NTQ5ODk4NA== | cobiadigital 25372415 | 2021-01-22T15:48:25Z | 2021-01-22T15:49:33Z | NONE | The "Warrior Gene" https://www.snpedia.com/index.php/Rs4680 ``` select rsid, genotype, case genotype when 'AA' then '(worrier) advantage in memory and attention tasks' when 'AG' then 'Intermediate dopamine levels, other effects' when 'GG' then '(warrior) multiple associations, see details' end as interpretation from genome where rsid = 'rs4680' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out some interesting example SQL queries 496415321 | |
765502845 | https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765502845 | https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDc2NTUwMjg0NQ== | cobiadigital 25372415 | 2021-01-22T15:53:19Z | 2021-01-22T15:53:19Z | NONE | rs7903146 Influences risk of Type-2 diabetes https://www.snpedia.com/index.php/Rs7903146 ``` select rsid, genotype, case genotype when 'CC' then 'Normal (lower) risk of Type 2 Diabetes and Gestational Diabetes.' when 'CT' then '1.4x increased risk for diabetes (and perhaps colon cancer).' when 'TT' then '2x increased risk for Type-2 diabetes' end as interpretation from genome where rsid = 'rs7903146' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out some interesting example SQL queries 496415321 | |
765506901 | https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765506901 | https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDc2NTUwNjkwMQ== | cobiadigital 25372415 | 2021-01-22T15:58:41Z | 2021-01-22T15:58:58Z | NONE | Both rs10757274 and rs2383206 can both indicate higher risks of heart disease https://www.snpedia.com/index.php/Rs2383206 ``` select rsid, genotype, case genotype when 'AA' then 'Normal' when 'AG' then '~1.2x increased risk for heart disease' when 'GG' then '~1.3x increased risk for heart disease' end as interpretation from genome where rsid = 'rs10757274' ``` ``` select rsid, genotype, case genotype when 'AA' then 'Normal' when 'AG' then '1.4x increased risk for heart disease' when 'GG' then '1.7x increased risk for heart disease' end as interpretation from genome where rsid = 'rs2383206' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out some interesting example SQL queries 496415321 | |
765523517 | https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765523517 | https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDc2NTUyMzUxNw== | cobiadigital 25372415 | 2021-01-22T16:20:25Z | 2021-01-22T16:20:25Z | NONE | rs53576: the oxytocin receptor (OXTR) gene ``` select rsid, genotype, case genotype when 'AA' then 'Lack of empathy?' when 'AG' then 'Lack of empathy?' when 'GG' then 'Optimistic and empathetic; handle stress well' end as interpretation from genome where rsid = 'rs53576' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out some interesting example SQL queries 496415321 | |
765525338 | https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765525338 | https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDc2NTUyNTMzOA== | cobiadigital 25372415 | 2021-01-22T16:22:44Z | 2021-01-22T16:22:44Z | NONE | rs1333049 associated with coronary artery disease https://www.snpedia.com/index.php/Rs1333049 ``` select rsid, genotype, case genotype when 'CC' then '1.9x increased risk for coronary artery disease' when 'CG' then '1.5x increased risk for CAD' when 'GG' then 'normal' end as interpretation from genome where rsid = 'rs1333049' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out some interesting example SQL queries 496415321 | |
831004775 | https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-831004775 | https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDgzMTAwNDc3NQ== | cobiadigital 25372415 | 2021-05-03T03:46:23Z | 2021-05-03T03:46:23Z | NONE | RS1800955 is related to novelty seeking and ADHD https://www.snpedia.com/index.php/Rs1800955 `select rsid, genotype, case genotype when 'CC' then 'increased susceptibility to novelty seeking' when 'CT' then 'increased susceptibility to novelty seeking' when 'TT' then 'normal' end as interpretation from genome where rsid = 'rs1800955'` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out some interesting example SQL queries 496415321 | |
549230337 | https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-549230337 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU0OTIzMDMzNw== | simonw 9599 | 2019-11-04T05:47:18Z | 2019-11-04T05:47:18Z | MEMBER | This definition isn't quite right - it's not pulling the identity of the user who starred the repo (`users.login` ends up being the owner login instead). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this repos_starred view 516967682 | |
622461122 | https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622461122 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTEyMg== | simonw 9599 | 2020-05-01T16:34:39Z | 2020-05-01T16:34:39Z | MEMBER | Blocked on #37 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this repos_starred view 516967682 | |
622980203 | https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622980203 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDYyMjk4MDIwMw== | simonw 9599 | 2020-05-02T16:34:29Z | 2020-05-02T16:34:29Z | MEMBER | Fixed definition: ```sql select stars.starred_at, starring_user.login as starred_by, repos.* from repos join stars on repos.id = stars.repo join users as starring_user on stars.user = starring_user.id join users on repos.owner = users.id order by starred_at desc; ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this repos_starred view 516967682 | |
594151327 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594151327 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDU5NDE1MTMyNw== | simonw 9599 | 2020-03-03T20:26:15Z | 2020-03-03T20:32:23Z | MEMBER | Better version (since this also includes JSON array of repository topics): ```sql CREATE VIEW recent_releases AS select repos.rowid as rowid, json_object("label", repos.full_name, "href", repos.html_url) as repo, json_object( "href", releases.html_url, "label", releases.name ) as release, substr(releases.published_at, 0, 11) as date, releases.body as body_markdown, releases.published_at, coalesce(repos.topics, '[]') as topics from releases join repos on repos.id = releases.repo order by releases.published_at desc ``` That `repos.rowid as rowid` bit is necessary because otherwise clicking on a link in facet-by-topic doesn't return any results. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
594155249 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594155249 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDU5NDE1NTI0OQ== | simonw 9599 | 2020-03-03T20:35:17Z | 2020-03-03T20:35:17Z | MEMBER | `swarm-to-sqlite` has an example of adding views here: https://github.com/dogsheep/swarm-to-sqlite/blob/f2c89dd613fb8a7f14e5267ccc2145463b996190/swarm_to_sqlite/utils.py#L141 I think that approach can be approved by first checking if the view exists, then dropping it, then recreating it. Could even try to see if the view exists and matches what we were going to set it to and do nothing if that is the case. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
622461223 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw== | simonw 9599 | 2020-05-01T16:34:52Z | 2020-05-01T16:34:52Z | MEMBER | Blocked on #37 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
622989874 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA== | simonw 9599 | 2020-05-02T17:46:14Z | 2020-05-02T17:46:14Z | MEMBER | Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
622998813 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw== | simonw 9599 | 2020-05-02T18:58:17Z | 2020-05-02T18:58:17Z | MEMBER | Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
602861730 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602861730 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjg2MTczMA== | simonw 9599 | 2020-03-23T21:19:16Z | 2020-03-23T21:19:42Z | MEMBER | I'm going to do this with a scheduled GitHub Action in this repo. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602862236 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862236 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjg2MjIzNg== | simonw 9599 | 2020-03-23T21:20:26Z | 2020-03-23T21:20:26Z | MEMBER | I'll run the `commits` and `issues` and `issue-comments` commands in addition to the `releases` command. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602862967 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862967 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjg2Mjk2Nw== | simonw 9599 | 2020-03-23T21:22:04Z | 2020-03-23T21:22:04Z | MEMBER | Following these instructions: https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602895896 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602895896 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjg5NTg5Ng== | simonw 9599 | 2020-03-23T22:42:25Z | 2020-03-23T22:42:25Z | MEMBER | Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602916947 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602916947 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjkxNjk0Nw== | simonw 9599 | 2020-03-23T23:38:06Z | 2020-03-23T23:38:06Z | MEMBER | Woohoo! https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602917713 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602917713 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjkxNzcxMw== | simonw 9599 | 2020-03-23T23:40:29Z | 2020-03-23T23:40:29Z | MEMBER | Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602918689 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602918689 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjkxODY4OQ== | simonw 9599 | 2020-03-23T23:43:39Z | 2020-03-23T23:47:50Z | MEMBER | I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602919058 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602919058 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjkxOTA1OA== | simonw 9599 | 2020-03-23T23:44:48Z | 2020-03-23T23:44:48Z | MEMBER | Next step: use a `metadata.json` file to add some extras. And add the `datasette-render-markdown` plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
602924714 | https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602924714 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDYwMjkyNDcxNA== | simonw 9599 | 2020-03-24T00:03:25Z | 2020-03-24T00:03:25Z | MEMBER | This is good enough for the 1.0 release. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a live demo Datasette instance 521275281 | |
559883311 | https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559883311 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14 | MDEyOklzc3VlQ29tbWVudDU1OTg4MzMxMQ== | simonw 9599 | 2019-11-29T21:30:37Z | 2019-11-29T21:30:37Z | MEMBER | I should build the command to persist ETags and obey their polling guidelines: > Events are optimized for polling with the "ETag" header. If no new events have been triggered, you will see a "304 Not Modified" response, and your current rate limit will be untouched. There is also an "X-Poll-Interval" header that specifies how often (in seconds) you are allowed to poll. In times of high server load, the time may increase. Please obey the header. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing events 530491074 | |
559902818 | https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559902818 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14 | MDEyOklzc3VlQ29tbWVudDU1OTkwMjgxOA== | simonw 9599 | 2019-11-30T01:32:38Z | 2019-11-30T01:32:38Z | MEMBER | Prototype: ``` pip install sqlite-utils paginate-json paginate-json "https://api.github.com/users/simonw/events" | sqlite-utils insert /tmp/events.db events - --pk=id ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing events 530491074 | |
613641947 | https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-613641947 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14 | MDEyOklzc3VlQ29tbWVudDYxMzY0MTk0Nw== | simonw 9599 | 2020-04-14T19:38:24Z | 2020-04-14T19:38:34Z | MEMBER | Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing events 530491074 | |
602309825 | https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602309825 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15 | MDEyOklzc3VlQ29tbWVudDYwMjMwOTgyNQ== | simonw 9599 | 2020-03-23T00:52:38Z | 2020-03-23T00:52:38Z | MEMBER | Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the `releases` command. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Assets table with downloads 544571092 | |
602311701 | https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602311701 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15 | MDEyOklzc3VlQ29tbWVudDYwMjMxMTcwMQ== | simonw 9599 | 2020-03-23T01:05:15Z | 2020-03-23T01:06:14Z | MEMBER | None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0 ``` github-to-sqlite releases releases.db instrumenta/conftest ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Assets table with downloads 544571092 | |
602312054 | https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602312054 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15 | MDEyOklzc3VlQ29tbWVudDYwMjMxMjA1NA== | simonw 9599 | 2020-03-23T01:07:10Z | 2020-03-23T01:07:10Z | MEMBER | Each asset looks like this: ```json { "url": "https://api.github.com/repos/instrumenta/conftest/releases/assets/11811946", "id": 11811946, "node_id": "MDEyOlJlbGVhc2VBc3NldDExODExOTQ2", "name": "checksums.txt", "label": "", "uploader": { "login": "garethr", "id": 2029, "node_id": "MDQ6VXNlcjIwMjk=", "avatar_url": "https://avatars2.githubusercontent.com/u/2029?v=4", "gravatar_id": "", "url": "https://api.github.com/users/garethr", "html_url": "https://github.com/garethr", "followers_url": "https://api.github.com/users/garethr/followers", "following_url": "https://api.github.com/users/garethr/following{/other_user}", "gists_url": "https://api.github.com/users/garethr/gists{/gist_id}", "starred_url": "https://api.github.com/users/garethr/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/garethr/subscriptions", "organizations_url": "https://api.github.com/users/garethr/orgs", "repos_url": "https://api.github.com/users/garethr/repos", "events_url": "https://api.github.com/users/garethr/events{/privacy}", "received_events_url": "https://api.github.com/users/garethr/received_events", "type": "User", "site_admin": false }, "content_type": "text/plain; charset=utf-8", "state": "uploaded", "size": 600, "download_count": 2, "created_at": "2019-03-30T16:56:44Z", "updated_at": "2019-03-30T16:56:44Z", "browser_download_url": "https://github.com/instrumenta/conftest/releases/download/v0.1.0/checksums.txt" } ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Assets table with downloads 544571092 | |
605439685 | https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-605439685 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15 | MDEyOklzc3VlQ29tbWVudDYwNTQzOTY4NQ== | garethr 2029 | 2020-03-28T12:17:01Z | 2020-03-28T12:17:01Z | NONE | That looks great, thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Assets table with downloads 544571092 | |
571412923 | https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-571412923 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16 | MDEyOklzc3VlQ29tbWVudDU3MTQxMjkyMw== | jayvdb 15092 | 2020-01-07T03:06:46Z | 2020-01-07T03:06:46Z | NONE | I re-tried after doing `auth`, and I get the same result. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Exception running first command: IndexError: list index out of range 546051181 | |
601979709 | https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-601979709 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16 | MDEyOklzc3VlQ29tbWVudDYwMTk3OTcwOQ== | simonw 9599 | 2020-03-21T02:20:08Z | 2020-03-21T02:20:08Z | MEMBER | I'm pretty sure this means I need to upgrade the underlying `sqlite-utils` dependency. I saw the same bug in `twitter-to-sqlite`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Exception running first command: IndexError: list index out of range 546051181 | |
602136481 | https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-602136481 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16 | MDEyOklzc3VlQ29tbWVudDYwMjEzNjQ4MQ== | jayvdb 15092 | 2020-03-22T02:08:57Z | 2020-03-22T02:08:57Z | NONE | I'd love to be using your library as a better cached gh layer for a new library I have built, replacing large parts of the very ugly https://github.com/jayvdb/pypidb/blob/master/pypidb/_github.py , and then probably being able to rebuild the setuppy chunk as a feature here at a later stage. I would also need tokenless and netrc support, but I would be happy to add those bits. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Exception running first command: IndexError: list index out of range 546051181 | |
613611455 | https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-613611455 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16 | MDEyOklzc3VlQ29tbWVudDYxMzYxMTQ1NQ== | simonw 9599 | 2020-04-14T18:37:21Z | 2020-04-14T18:37:21Z | MEMBER | This should have been fixed by #20 and #23 @jayvdb I'm definitely interested in this tool working as a library - it's purely designed as a CLI tool at the moment, but cleaning it up to work better as a dependency is totally in-scope for the project. https://sqlite-utils.readthedocs.io/ is an example of a tool I've built that works for both. Feel free to open a new issue here with some notes on what you would need for this to work as a library for your project! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Exception running first command: IndexError: list index out of range 546051181 | |
597354514 | https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597354514 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17 | MDEyOklzc3VlQ29tbWVudDU5NzM1NDUxNA== | simonw 9599 | 2020-03-10T22:37:45Z | 2020-03-10T22:37:45Z | MEMBER | I should add an option to stop the moment you see a commit you have fetched before. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing commits 578883725 | |
597358364 | https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597358364 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17 | MDEyOklzc3VlQ29tbWVudDU5NzM1ODM2NA== | simonw 9599 | 2020-03-10T22:50:20Z | 2020-03-11T01:18:36Z | MEMBER | By default it will stop when it sees a commit that has already been stored. You will be able to over-ride that behaviour using `--all`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing commits 578883725 | |
601980724 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980724 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMTk4MDcyNA== | simonw 9599 | 2020-03-21T02:28:05Z | 2020-03-21T02:28:05Z | MEMBER | This suggests that `commit["author"]` can be `None` in some cases? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
601980957 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980957 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMTk4MDk1Nw== | simonw 9599 | 2020-03-21T02:30:03Z | 2020-03-21T02:30:03Z | MEMBER | https://github.community/t5/GitHub-API-Development-and/Request-for-commits-quot-author-quot-null-and-quot-committer/m-p/35842/highlight/true#M3372 > Commits aren't always associated with a GitHub user. For example, perhaps a friend of mine and I were working on a project together. I have a GitHub account and my friend doesn't. If we both add commits to the repository using our own email addresses and names and then I push the repository to GitHub, my commits will be associated with my GitHub user account but my friends' commits will show up with `author` and `committer` as `null`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602807178 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602807178 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMjgwNzE3OA== | simonw 9599 | 2020-03-23T19:24:43Z | 2020-03-23T19:24:43Z | MEMBER | I need to find an example before I work on this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602811468 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811468 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMjgxMTQ2OA== | simonw 9599 | 2020-03-23T19:33:14Z | 2020-03-23T19:33:14Z | MEMBER | Found one: https://api.github.com/repos/simonw/simonw.github.com/commits | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602811785 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811785 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMjgxMTc4NQ== | simonw 9599 | 2020-03-23T19:33:56Z | 2020-03-23T19:34:06Z | MEMBER | ```json [ { "sha": "a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "node_id": "MDY6Q29tbWl0OTMyMDk6YThkYzkxNDA4OWQzOTlkOWI1MjJlYmI1MWI2N2Y5YWMyZThhYTZiMA==", "commit": { "author": { "name": "Simon Willison", "email": "simon@...", "date": "2008-12-18T23:17:12Z" }, "committer": { "name": "Simon Willison", "email": "simon@...", "date": "2008-12-18T23:17:12Z" }, "message": "First commit", "tree": { "sha": "ac2dfb75e2592c59165c2880f3f7a16dafd452a1", "url": "https://api.github.com/repos/simonw/simonw.github.com/git/trees/ac2dfb75e2592c59165c2880f3f7a16dafd452a1" }, "url": "https://api.github.com/repos/simonw/simonw.github.com/git/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "comment_count": 0, "verification": { "verified": false, "reason": "unsigned", "signature": null, "payload": null } }, "url": "https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "html_url": "https://github.com/simonw/simonw.github.com/commit/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0", "comments_url": "https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0/comments", "author": null, "committer": null, "parents": [] } ] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602813731 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602813731 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMjgxMzczMQ== | simonw 9599 | 2020-03-23T19:38:03Z | 2020-03-23T20:44:49Z | MEMBER | So it turns out `"author"` and `"committer"` on the commit are `null` if the email address in the nested `"commit"` doesn't match an existing GitHub user. Maybe I should be storing the nested data somewhere as well? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602815120 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602815120 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMjgxNTEyMA== | simonw 9599 | 2020-03-23T19:40:55Z | 2020-03-23T19:43:19Z | MEMBER | I could pull a pk-hashed version of the name/email into separate `raw_author` and `raw_committer` columns perhaps - against a `commit_authors` table. Could be interesting. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602846293 | https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602846293 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDYwMjg0NjI5Mw== | simonw 9599 | 2020-03-23T20:44:40Z | 2020-03-23T20:44:40Z | MEMBER | I implemented the `raw_authors` idea. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commits in GitHub API can have null author 585411547 | |
602304531 | https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602304531 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19 | MDEyOklzc3VlQ29tbWVudDYwMjMwNDUzMQ== | simonw 9599 | 2020-03-23T00:20:28Z | 2020-03-23T00:20:28Z | MEMBER | Current FTS code: https://github.com/dogsheep/github-to-sqlite/blob/e35eec4343aa560c58c1634cc228d0d46c442304/github_to_sqlite/utils.py#L136-L143 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Enable full-text search for more stuff (like commits, issues and issue_comments) 585850715 | |
602782437 | https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602782437 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19 | MDEyOklzc3VlQ29tbWVudDYwMjc4MjQzNw== | simonw 9599 | 2020-03-23T18:36:12Z | 2020-03-23T18:51:23Z | MEMBER | * `commits`: `message` * `issue_comments`: `body` * `issues`: `title`, `body` * `labels`: `name`, `description` * `licenses`: `name` * `milestones`: `title`, `description` * `releases`: `name`, `body` * `repos`: `name`, `description` * `users`: `login`, `name` (maybe `company`, `bio`) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Enable full-text search for more stuff (like commits, issues and issue_comments) 585850715 | |
602896434 | https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602896434 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21 | MDEyOklzc3VlQ29tbWVudDYwMjg5NjQzNA== | simonw 9599 | 2020-03-23T22:43:37Z | 2020-03-23T22:43:37Z | MEMBER | I'm going to do this now to help figure out the latest error in #13: ``` Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite", line 11, in <module> load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')() File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 237, in commits utils.save_commits(db, commits, repo_full["id"]) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 345, in save_commits for commit in commits: File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 207, in fetch_commits if stop_when(commit): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 224, in stop_when db["commits"].get(commit["sha"]) TypeError: string indices must be integers ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Turn GitHub API errors into exceptions 586561727 | |
602920163 | https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602920163 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21 | MDEyOklzc3VlQ29tbWVudDYwMjkyMDE2Mw== | simonw 9599 | 2020-03-23T23:48:22Z | 2020-03-23T23:48:22Z | MEMBER | I'm happy with this pattern: https://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Turn GitHub API errors into exceptions 586561727 | |
602928533 | https://github.com/dogsheep/github-to-sqlite/issues/23#issuecomment-602928533 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23 | MDEyOklzc3VlQ29tbWVudDYwMjkyODUzMw== | simonw 9599 | 2020-03-24T00:15:49Z | 2020-03-24T00:15:49Z | MEMBER | https://github.com/dogsheep/github-to-sqlite/releases/tag/1.0 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Release 1.0 586595839 | |
614810417 | https://github.com/dogsheep/github-to-sqlite/issues/25#issuecomment-614810417 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25 | MDEyOklzc3VlQ29tbWVudDYxNDgxMDQxNw== | simonw 9599 | 2020-04-16T18:07:11Z | 2020-04-16T18:07:11Z | MEMBER | Turns out the main problem was #26 - now fixed. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Improvements to demo instance 601265023 | |
614794739 | https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614794739 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26 | MDEyOklzc3VlQ29tbWVudDYxNDc5NDczOQ== | simonw 9599 | 2020-04-16T17:38:28Z | 2020-04-16T17:38:28Z | MEMBER | I'm already doing this here: https://github.com/dogsheep/github-to-sqlite/blob/c4aaa50e167cfa9021c7c94260bc3e89e10947bf/github_to_sqlite/utils.py#L246-L250 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Topics are missing from repositories 601271612 | |
614795712 | https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614795712 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26 | MDEyOklzc3VlQ29tbWVudDYxNDc5NTcxMg== | simonw 9599 | 2020-04-16T17:40:27Z | 2020-04-16T17:40:27Z | MEMBER | Aha! it was missing from the `fetch_repo()` function. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Topics are missing from repositories 601271612 | |
614831451 | https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831451 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDYxNDgzMTQ1MQ== | simonw 9599 | 2020-04-16T18:47:25Z | 2020-04-16T18:47:25Z | MEMBER | Is it possible for a repo to have an `owner` that differs from its `organization`? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Repos have a big blob of JSON in the organization column 601330277 | |
614831842 | https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831842 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDYxNDgzMTg0Mg== | simonw 9599 | 2020-04-16T18:48:18Z | 2020-04-16T18:48:18Z | MEMBER | I'm going to make `organization` another foreign key to the `users` table just in case it IS possible (maybe with GitHub Enterprise or similar?) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Repos have a big blob of JSON in the organization column 601330277 | |
614843406 | https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDYxNDg0MzQwNg== | simonw 9599 | 2020-04-16T19:11:53Z | 2020-04-16T19:20:23Z | MEMBER | This didn't quite work: the column type is incorrect, so the foreign key relationship isn't sticking: https://github-to-sqlite.dogsheep.net/github/repos?organization=53015001 `[organization] TEXT REFERENCES [users]([id])` - should be `INTEGER`. The problem is that if the first repo inserted has no organization it's set to `null`, which `sqlite-utils` derives as a `TEXT` column. One solution would be to create the column explicitly with a type, but this could get messy. I think I want a new sqlite-utils feature for this instead. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Repos have a big blob of JSON in the organization column 601330277 | |
615518606 | https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615518606 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDYxNTUxODYwNg== | simonw 9599 | 2020-04-18T00:14:32Z | 2020-04-18T00:14:32Z | MEMBER | https://github.com/simonw/sqlite-utils/issues/100 is done and released in sqlite-utils 2.7. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Repos have a big blob of JSON in the organization column 601330277 | |
615519409 | https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615519409 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDYxNTUxOTQwOQ== | simonw 9599 | 2020-04-18T00:19:16Z | 2020-04-18T00:19:16Z | MEMBER | ``` $ github-to-sqlite repos b.db dogsheep $ sqlite3 b.db '.schema repos' CREATE TABLE [repos] ( [id] INTEGER PRIMARY KEY, ... [permissions] TEXT, [organization] INTEGER REFERENCES [users]([id]), FOREIGN KEY(license) REFERENCES licenses(key) ); ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Repos have a big blob of JSON in the organization column 601330277 | |
615883040 | https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883040 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28 | MDEyOklzc3VlQ29tbWVudDYxNTg4MzA0MA== | simonw 9599 | 2020-04-18T14:45:38Z | 2020-04-18T14:45:38Z | MEMBER | ``` File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 219, in contributors utils.save_contributors(db, contributors, repo_full["id"]) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 354, in save_contributors for contributor in contributors: File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 228, in fetch_contributors for contributors in paginate(url, headers): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 286, in paginate data = response.json() File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/requests/models.py", line 898, in json return complexjson.loads(self.text, **kwargs) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/__init__.py", line 357, in loads return _default_decoder.decode(s) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from N… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pull repository contributors 601333634 | |
615883687 | https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883687 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28 | MDEyOklzc3VlQ29tbWVudDYxNTg4MzY4Nw== | simonw 9599 | 2020-04-18T14:49:58Z | 2020-04-18T14:49:58Z | MEMBER | That happened trying to pull contributors for `dogsheep/beta` - an empty repository. Turns out it was returning a `204 no content`: ``` ~ $ curl -i 'https://api.github.com/repos/dogsheep/beta/contributors' HTTP/1.1 204 No Content ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pull repository contributors 601333634 | |
615886206 | https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615886206 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28 | MDEyOklzc3VlQ29tbWVudDYxNTg4NjIwNg== | simonw 9599 | 2020-04-18T15:04:59Z | 2020-04-18T15:04:59Z | MEMBER | Demo: https://github-to-sqlite.dogsheep.net/github/contributors Documentation: https://github.com/dogsheep/github-to-sqlite/blob/13f8868fb5efa01c263b24f6dd91c617e6e938e1/README.md#fetching-contributors-to-a-repository | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pull repository contributors 601333634 | |
616883275 | https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29 | MDEyOklzc3VlQ29tbWVudDYxNjg4MzI3NQ== | simonw 9599 | 2020-04-21T00:43:28Z | 2020-04-21T00:43:28Z | MEMBER | I'm copying repo from issue, which surprisingly is a string, not an integer ID. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Milestones should have foreign key to creator and repo 603617013 | |
531516956 | https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531516956 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDUzMTUxNjk1Ng== | simonw 9599 | 2019-09-14T21:56:31Z | 2019-09-14T21:56:31Z | MEMBER | https://api.github.com/users/simonw/repos It would be useful to be able to fetch stargazers, forks etc as well. Not sure if that should be a separate command or a `--stargazers` option to this command. Probably a separate command since `issues` is a separate command already. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch all repos belonging to a user or organization 493670426 | |
531517083 | https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531517083 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDUzMTUxNzA4Mw== | simonw 9599 | 2019-09-14T21:58:42Z | 2019-09-14T21:58:42Z | MEMBER | Split stargazers into #4 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch all repos belonging to a user or organization 493670426 | |
616879753 | https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616879753 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30 | MDEyOklzc3VlQ29tbWVudDYxNjg3OTc1Mw== | simonw 9599 | 2020-04-21T00:29:29Z | 2020-04-21T00:29:29Z | MEMBER | `assignee` looks like it's the incorrect type too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Issues milestone column is the wrong type 603618244 | |
616883726 | https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616883726 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30 | MDEyOklzc3VlQ29tbWVudDYxNjg4MzcyNg== | simonw 9599 | 2020-04-21T00:45:23Z | 2020-04-21T00:45:23Z | MEMBER | Demo of fix: https://github-to-sqlite.dogsheep.net/github/issues?assignee__notblank=1&milestone__notblank=1 ![github__issues__4_rows_where_where_assignee_is_not_blank_and_milestone_is_not_blank_sorted_by_updated_at_descending](https://user-images.githubusercontent.com/9599/79812758-b49ace80-832e-11ea-81db-bdf993b872cc.png) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Issues milestone column is the wrong type 603618244 | |
616884647 | https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-616884647 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31 | MDEyOklzc3VlQ29tbWVudDYxNjg4NDY0Nw== | simonw 9599 | 2020-04-21T00:49:16Z | 2020-04-21T00:50:20Z | MEMBER | The API just gives us the `repository_url`: https://api.github.com/repos/simonw/datasette/issues ![Mozilla_Firefox_and_Topic__Week_2__Discussion__Submit_your_six_story_points_here](https://user-images.githubusercontent.com/9599/79812950-283cdb80-832f-11ea-8759-9633087d1e7e.png) We currently turn that into a `simonw/datasette` string here: https://github.com/dogsheep/github-to-sqlite/blob/e0e8d8caa9657b04bfb8a2cf16c9b580f38b1805/github_to_sqlite/utils.py#L43-L46 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Issue and milestone should have foreign key to repo 603624862 | |
617348174 | https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617348174 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31 | MDEyOklzc3VlQ29tbWVudDYxNzM0ODE3NA== | simonw 9599 | 2020-04-21T18:50:29Z | 2020-04-21T18:50:29Z | MEMBER | Since this represents a breaking schema change for anyone running SQL queries against these tables, I'm going to do a major version bump to 2.0 when I release this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Issue and milestone should have foreign key to repo 603624862 | |
617491607 | https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617491607 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31 | MDEyOklzc3VlQ29tbWVudDYxNzQ5MTYwNw== | simonw 9599 | 2020-04-22T01:20:19Z | 2020-04-22T01:20:19Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/milestones now link to repo: <img width="1095" alt="Screen Shot 2020-04-21 at 6 19 03 PM" src="https://user-images.githubusercontent.com/9599/79930145-956a7280-83fc-11ea-9b47-b0c5674a0d4f.png"> And so do issues: https://github-to-sqlite.dogsheep.net/github/issues <img width="1095" alt="Screen Shot 2020-04-21 at 6 19 53 PM" src="https://user-images.githubusercontent.com/9599/79930200-bd59d600-83fc-11ea-8fb8-b40772140409.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Issue and milestone should have foreign key to repo 603624862 | |
617364956 | https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617364956 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32 | MDEyOklzc3VlQ29tbWVudDYxNzM2NDk1Ng== | simonw 9599 | 2020-04-21T19:24:45Z | 2020-04-21T19:24:45Z | MEMBER | That's because I just broke this code: https://github.com/dogsheep/github-to-sqlite/blob/2cf75a0a036719eb7e57fdc7c5c2ea0f4c26978a/github_to_sqlite/utils.py#L131-L139 It expects the `repo` column to be `simonw/datasette` but it's now an ID instead. I should add a test for this as part of the fix. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Issue comments don't appear to populate issues foreign key 604222295 | |
617369247 | https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617369247 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32 | MDEyOklzc3VlQ29tbWVudDYxNzM2OTI0Nw== | simonw 9599 | 2020-04-21T19:33:03Z | 2020-04-21T19:33:03Z | MEMBER | Caused by #31. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Issue comments don't appear to populate issues foreign key 604222295 | |
617490914 | https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617490914 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32 | MDEyOklzc3VlQ29tbWVudDYxNzQ5MDkxNA== | simonw 9599 | 2020-04-22T01:17:44Z | 2020-04-22T01:17:44Z | MEMBER | https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101 now shows issues. And https://github-to-sqlite.dogsheep.net/github/issue_comments links to them: <img width="395" alt="Screen Shot 2020-04-21 at 6 17 33 PM" src="https://user-images.githubusercontent.com/9599/79930066-6227e380-83fc-11ea-8416-8b9d55660cb5.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Issue comments don't appear to populate issues foreign key 604222295 | |
622169728 | https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622169728 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33 | MDEyOklzc3VlQ29tbWVudDYyMjE2OTcyOA== | simonw 9599 | 2020-04-30T23:18:51Z | 2020-04-30T23:18:51Z | MEMBER | Sure, that sounds fine to me. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fall back to authentication via ENV 609950090 | |
622171097 | https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622171097 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33 | MDEyOklzc3VlQ29tbWVudDYyMjE3MTA5Nw== | simonw 9599 | 2020-04-30T23:22:45Z | 2020-04-30T23:23:57Z | MEMBER | The `auth.json` mechanism this uses is standard across all of the other Dogsheep tools - it's actually designed so you can have one `auth.json` with a bunch of different credentials for different tools: ```json { "goodreads_personal_token": "...", "goodreads_user_id": "...", "github_personal_token": "...", "pocket_consumer_key": "...", "pocket_username": "...", "pocket_access_token": "..." } ``` But... `github-to-sqlite` does feel like it deserves a special case here, since it's such a good fit for running inside of GitHub Actions - which even provide a `GITHUB_TOKEN` for you to use! So I don't think it will harm the family of tools too much if this has an environment variable alternative to the `-a` file. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fall back to authentication via ENV 609950090 | |
622279374 | https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622279374 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33 | MDEyOklzc3VlQ29tbWVudDYyMjI3OTM3NA== | garethr 2029 | 2020-05-01T07:12:47Z | 2020-05-01T07:12:47Z | NONE | I also go it working with: ```yaml run: echo ${{ secrets.github_token }} | github-to-sqlite auth ``` | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fall back to authentication via ENV 609950090 | |
622997410 | https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622997410 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33 | MDEyOklzc3VlQ29tbWVudDYyMjk5NzQxMA== | simonw 9599 | 2020-05-02T18:46:10Z | 2020-05-02T18:46:10Z | MEMBER | Documented here: https://github.com/dogsheep/github-to-sqlite/blob/10fb34de41aaa35681f08b5991540d65bfcf2e2e/README.md#authentication | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fall back to authentication via ENV 609950090 | |
622133298 | https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133298 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 | MDEyOklzc3VlQ29tbWVudDYyMjEzMzI5OA== | simonw 9599 | 2020-04-30T21:48:24Z | 2020-04-30T21:48:24Z | MEMBER | Unfortunately it's not available through any GitHub API - I managed to figure out how to get dependencies, but I need dependents. https://github.com/simonw/til/blob/master/github/dependencies-graphql-api.md | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for retrieving dependents for a repo 610408908 | |
622133422 | https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133422 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 | MDEyOklzc3VlQ29tbWVudDYyMjEzMzQyMg== | simonw 9599 | 2020-04-30T21:48:39Z | 2020-04-30T21:48:39Z | MEMBER | It looks like the only option is to scrape them. I'll do that and then replace with an API as soon as one becomes available. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for retrieving dependents for a repo 610408908 | |
622133775 | https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133775 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 | MDEyOklzc3VlQ29tbWVudDYyMjEzMzc3NQ== | simonw 9599 | 2020-04-30T21:49:27Z | 2020-04-30T21:49:27Z | MEMBER | Proposed command: github-to-sqlite scrape-dependents github.db simonw/datasette I'll pull full details of the scraped repos from the regular API. I'll also record when they were "first seen" by the command. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for retrieving dependents for a repo 610408908 | |
622135654 | https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622135654 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 | MDEyOklzc3VlQ29tbWVudDYyMjEzNTY1NA== | simonw 9599 | 2020-04-30T21:53:44Z | 2020-04-30T21:56:06Z | MEMBER | I think this is the neatest scraping pattern: ```python [a["href"].lstrip("/") for a in soup.select("a[data-hovercard-type=repository]")] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for retrieving dependents for a repo 610408908 | |
622136585 | https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622136585 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 | MDEyOklzc3VlQ29tbWVudDYyMjEzNjU4NQ== | simonw 9599 | 2020-04-30T21:55:51Z | 2020-04-30T21:55:51Z | MEMBER | And to find the "Next" pagination link: ```python soup.select(".paginate-container")[0].find("a", text="Next") ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for retrieving dependents for a repo 610408908 | |
622162835 | https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622162835 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 | MDEyOklzc3VlQ29tbWVudDYyMjE2MjgzNQ== | simonw 9599 | 2020-04-30T22:59:26Z | 2020-04-30T22:59:26Z | MEMBER | Documentation: https://github.com/dogsheep/github-to-sqlite/blob/c9f48404481882e8b3af06f35e4801a80ac79ed6/README.md#scraping-dependents-for-a-repository | {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 2, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for retrieving dependents for a repo 610408908 | |
622213950 | https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622213950 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYyMjIxMzk1MA== | simonw 9599 | 2020-05-01T02:09:04Z | 2020-05-01T02:09:04Z | MEMBER | It sped up this query a lot - 2.5s down to 300ms: ```sql select repos.full_name, json_object( 'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number, 'label', '#' || issues.number ) as issue, issues.title, users.login, users.id, issues.state, issues.locked, issues.assignee, issues.milestone, issues.comments, issues.created_at, issues.updated_at, issues.closed_at, issues.author_association, issues.pull_request, issues.repo, issues.type from issues join repos on repos.id = issues.repo join users on issues.user = users.id where issues.state = 'open' and issues.user not in (9599, 27856297) and not exists ( select id from issue_comments where issue_comments.user = 9599 and issues.id = issue_comments.issue ) order by issues.updated_at desc; ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create index on issue_comments(user) and other foreign keys 610511450 | |
622214262 | https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622214262 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYyMjIxNDI2Mg== | simonw 9599 | 2020-05-01T02:10:32Z | 2020-05-01T02:11:19Z | MEMBER | This sped that query up even more - down to 4ms. ```sql create index issue_comments_issue on issue_comments(issue); ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create index on issue_comments(user) and other foreign keys 610511450 | |
622982667 | https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622982667 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYyMjk4MjY2Nw== | simonw 9599 | 2020-05-02T16:52:53Z | 2020-05-02T16:52:53Z | MEMBER | Easiest option: use `db.index_foreign_keys()`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#adding-indexes-for-all-foreign-keys | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create index on issue_comments(user) and other foreign keys 610511450 | |
622461025 | https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622461025 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTAyNQ== | simonw 9599 | 2020-05-01T16:34:24Z | 2020-05-01T16:34:24Z | MEMBER | Blocked on #37 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add view for better display of dependent repos 610842926 | |
622982346 | https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622982346 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36 | MDEyOklzc3VlQ29tbWVudDYyMjk4MjM0Ng== | simonw 9599 | 2020-05-02T16:50:31Z | 2020-05-02T16:50:31Z | MEMBER | Demo: https://github-to-sqlite.dogsheep.net/github/dependent_repos | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add view for better display of dependent repos 610842926 | |
622461537 | https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461537 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTUzNw== | simonw 9599 | 2020-05-01T16:35:40Z | 2020-05-01T16:35:40Z | MEMBER | This will check if the view exists and has the exact same matching definition as the one we want. If it doesn't, we will drop it (if it exists) and recreate it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for creating views if they don't yet exist 610843136 | |
622461948 | https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461948 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37 | MDEyOklzc3VlQ29tbWVudDYyMjQ2MTk0OA== | simonw 9599 | 2020-05-01T16:36:42Z | 2020-05-01T16:36:42Z | MEMBER | It should only create views if the underlying tables exist. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for creating views if they don't yet exist 610843136 | |
622978173 | https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622978173 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37 | MDEyOklzc3VlQ29tbWVudDYyMjk3ODE3Mw== | simonw 9599 | 2020-05-02T16:19:31Z | 2020-05-02T16:19:47Z | MEMBER | I can use the new `.create_view(..., replace=True)` parameter in `sqlite-utils` 2.7.2 for this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for creating views if they don't yet exist 610843136 | |
623027889 | https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623027889 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38 | MDEyOklzc3VlQ29tbWVudDYyMzAyNzg4OQ== | simonw 9599 | 2020-05-02T23:15:11Z | 2020-05-02T23:15:11Z | MEMBER | This is one of the use-cases for the `repos_starred` view: it allows you to easily run this kid of query without having to construct the SQL by hand. Here's a demo: https://github-to-sqlite.dogsheep.net/github/repos_starred?name__contains=twitter My philosophy here is to keep the raw tables (like `stars`) as normalized as possible, then use SQL views which expose the data in a form that's easier to query. | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | [Feature Request] Support Repo Name in Search 🥺 611284481 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
updated_at (date) >30 ✖