issue_comments
525 rows where author_association = "MEMBER" sorted by node_id
This data as json, CSV (advanced)
id | html_url | issue_url | node_id ▼ | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
549095217 | https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095217 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDU0OTA5NTIxNw== | simonw 9599 | 2019-11-03T01:06:25Z | 2019-11-03T01:06:25Z | MEMBER | Wow, that `retweets_of_me` endpoint is almost completely useless: ``` $ twitter-to-sqlite fetch https://api.twitter.com/1.1/statuses/retweets_of_me.json ``` It returns my own tweets that have been retweeted, but with no indication at all of who retweeted them. It looks like this needs to be combined with this API - https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-retweets-id - to fetch the details of up to 100 recent users who actually DID retweet an individual status. But that has a one-every-12-seconds rate limit on it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | retweets-of-me command 514459062 | |
549095317 | https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095317 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDU0OTA5NTMxNw== | simonw 9599 | 2019-11-03T01:08:10Z | 2019-11-03T01:08:10Z | MEMBER | Hmm... one thing that could be useful is that `retweets_of_me` can support a `--since` parameter - so if run frequently it should hopefully let us know which tweets we would need to run `statuses/retweets/:id.json` against. I'm not sure if the `--since` parameter would show me a tweet that was previously retweeted but has now been retweeted again. I'll have a bit of a test and see. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | retweets-of-me command 514459062 | |
549095463 | https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095463 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDU0OTA5NTQ2Mw== | simonw 9599 | 2019-11-03T01:10:52Z | 2019-11-03T01:10:52Z | MEMBER | I imagine it won't, since the data I would be recording and then passing to `since_id` would be the highest ID of my own tweets that have been retweeted at least once. So it won't be able to spot if I should check for fresh retweets of a given tweet. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | retweets-of-me command 514459062 | |
549095641 | https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095641 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDU0OTA5NTY0MQ== | simonw 9599 | 2019-11-03T01:12:58Z | 2019-11-03T01:12:58Z | MEMBER | It looks like Twitter really want you to subscribe to a premium API for this kind of thing and consume retweets via webhooks: https://developer.twitter.com/en/docs/accounts-and-users/subscribe-account-activity/api-reference I'm going to give up on this for the moment. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | retweets-of-me command 514459062 | |
549096321 | https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549096321 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDU0OTA5NjMyMQ== | simonw 9599 | 2019-11-03T01:27:55Z | 2019-11-03T01:28:17Z | MEMBER | It would be neat if this could support `--since`, with that argument automatically finding the maximum tweet ID from a previous search that used the same exact arguments (using the `search_runs` table). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for running a search and saving tweets for that search 488833975 | |
549226399 | https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549226399 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDU0OTIyNjM5OQ== | simonw 9599 | 2019-11-04T05:11:57Z | 2019-11-04T05:11:57Z | MEMBER | I'm going to add a `hash` column to `search_runs` to support that. It's going to be the sha1 hash of the key-ordered JSON of the search arguments used by that run. Then `--since` can look for an identical hash and use it to identify the highest last fetched tweet to use in `since_id`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for running a search and saving tweets for that search 488833975 | |
549228535 | https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549228535 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDU0OTIyODUzNQ== | simonw 9599 | 2019-11-04T05:31:55Z | 2019-11-04T05:31:55Z | MEMBER | Documented here: https://github.com/dogsheep/twitter-to-sqlite/blob/801c0c2daf17d8abce9dcb5d8d610410e7e25dbe/README.md#running-searches | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for running a search and saving tweets for that search 488833975 | |
549230337 | https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-549230337 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU0OTIzMDMzNw== | simonw 9599 | 2019-11-04T05:47:18Z | 2019-11-04T05:47:18Z | MEMBER | This definition isn't quite right - it's not pulling the identity of the user who starred the repo (`users.login` ends up being the owner login instead). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this repos_starred view 516967682 | |
549230583 | https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549230583 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDU0OTIzMDU4Mw== | simonw 9599 | 2019-11-04T05:49:26Z | 2019-11-04T05:49:26Z | MEMBER | Adding the view from #10 would be useful here too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | stargazers command, refs #4 516763727 | |
549233778 | https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549233778 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDU0OTIzMzc3OA== | simonw 9599 | 2019-11-04T06:14:40Z | 2019-11-04T06:14:40Z | MEMBER | Spotted a tricky problem: running `github-to-sqlite starred stargazers.db` results in an incomplete `simonw` record. It creates a proper record for me thanks to this bit: https://github.com/dogsheep/github-to-sqlite/blob/ea07274667a08c67907e8bfbbccb6f0fb95ce817/github_to_sqlite/cli.py#L120-L126 But then... when it gets to the `datasette` repository which I have starred it over-writes my full user record with one that's missing most of the details, thanks to this bit: https://github.com/dogsheep/github-to-sqlite/blob/ea07274667a08c67907e8bfbbccb6f0fb95ce817/github_to_sqlite/utils.py#L117-L124 I need to find a way of NOT over-writing a good record with a thinner one. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | stargazers command, refs #4 516763727 | |
550388354 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-550388354 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDU1MDM4ODM1NA== | simonw 9599 | 2019-11-06T16:26:55Z | 2019-11-06T16:26:55Z | MEMBER | Here's a query I figured out using a window function that shows cumulative stargazers over time: ```sql select yyyymmdd, sum(n) over ( order by yyyymmdd rows unbounded preceding ) as cumulative_count from ( select substr(starred_at, 0, 11) as yyyymmdd, count(*) as n from stars group by yyyymmdd ) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
550783316 | https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550783316 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU1MDc4MzMxNg== | simonw 9599 | 2019-11-07T05:16:56Z | 2019-11-07T05:34:29Z | MEMBER | It looks like Apple changed the location of these in iOS 13 - they are now in separate `.gpx` files: ![2FF70E95-CDEE-4241-A5C5-EE95A862E519](https://user-images.githubusercontent.com/9599/68362042-be12e000-00da-11ea-8925-7397410332d8.png) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Failed to import workout points 519038979 | |
550806302 | https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550806302 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU1MDgwNjMwMg== | simonw 9599 | 2019-11-07T05:33:31Z | 2019-11-07T05:33:31Z | MEMBER | The XML now includes references to these new files: ![CBBA54FC-51FB-4BB3-927C-C2CA99237B04](https://user-images.githubusercontent.com/9599/68362716-121ec400-00dd-11ea-9846-387c7cd64c8b.jpeg) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Failed to import workout points 519038979 | |
550824838 | https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550824838 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU1MDgyNDgzOA== | simonw 9599 | 2019-11-07T05:47:07Z | 2019-11-07T05:47:07Z | MEMBER | Relevant code: https://github.com/dogsheep/healthkit-to-sqlite/blob/d16f45f06fbae6ec8a78cc9ca7b5b7db0413f139/healthkit_to_sqlite/utils.py#L58-L64 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Failed to import workout points 519038979 | |
550828084 | https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550828084 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU1MDgyODA4NA== | simonw 9599 | 2019-11-07T05:49:24Z | 2019-11-07T05:49:24Z | MEMBER | So the fix there is going to be to detect the new `FileReference` element and load the corresponding points data from it. This will be a little tricky because that function will need access to the zip file. It probably won't work at all for the mode where the `export.xml` file is passed directly using the `--xml` option. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Failed to import workout points 519038979 | |
552129686 | https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129686 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29 | MDEyOklzc3VlQ29tbWVudDU1MjEyOTY4Ng== | simonw 9599 | 2019-11-09T19:27:39Z | 2019-11-09T19:27:39Z | MEMBER | I think this is fixed by the latest version of `sqlite-utils` - https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-12-1 - I'll bump the dependency. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `import` command fails on empty files 518725064 | |
552129921 | https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129921 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29 | MDEyOklzc3VlQ29tbWVudDU1MjEyOTkyMQ== | simonw 9599 | 2019-11-09T19:30:42Z | 2019-11-09T19:30:42Z | MEMBER | Confirmed, that seems to fix it: ``` (twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js Traceback (most recent call last): File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite", line 11, in <module> load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')() File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py", line 633, in import_ archive.import_from_file(db, path.name, open(path, "rb").read()) File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/archive.py", line 224, in import_from_file db[table_name].upsert_all(rows, hash_id="pk") File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py", line 1094, in upsert_all extracts=extracts, File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `import` command fails on empty files 518725064 | |
552131798 | https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552131798 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30 | MDEyOklzc3VlQ29tbWVudDU1MjEzMTc5OA== | simonw 9599 | 2019-11-09T19:54:45Z | 2019-11-09T19:54:45Z | MEMBER | Good catch - not sure how that bug crept in. Removing line 116 looks like the right fix to me. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `followers` fails because `transform_user` is called twice 518739697 | |
552133449 | https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552133449 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29 | MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ0OQ== | simonw 9599 | 2019-11-09T20:15:15Z | 2019-11-09T20:15:15Z | MEMBER | Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `import` command fails on empty files 518725064 | |
552133468 | https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552133468 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30 | MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ2OA== | simonw 9599 | 2019-11-09T20:15:27Z | 2019-11-09T20:15:27Z | MEMBER | Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `followers` fails because `transform_user` is called twice 518739697 | |
552133488 | https://github.com/dogsheep/twitter-to-sqlite/issues/28#issuecomment-552133488 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/28 | MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ4OA== | simonw 9599 | 2019-11-09T20:15:42Z | 2019-11-09T20:15:42Z | MEMBER | Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add indexes to followers table 515658861 | |
552135263 | https://github.com/dogsheep/twitter-to-sqlite/issues/31#issuecomment-552135263 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31 | MDEyOklzc3VlQ29tbWVudDU1MjEzNTI2Mw== | simonw 9599 | 2019-11-09T20:38:35Z | 2019-11-09T20:38:35Z | MEMBER | Command still needs documentation and a bit more testing. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | "friends" command (similar to "followers") 520508502 | |
559883311 | https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559883311 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14 | MDEyOklzc3VlQ29tbWVudDU1OTg4MzMxMQ== | simonw 9599 | 2019-11-29T21:30:37Z | 2019-11-29T21:30:37Z | MEMBER | I should build the command to persist ETags and obey their polling guidelines: > Events are optimized for polling with the "ETag" header. If no new events have been triggered, you will see a "304 Not Modified" response, and your current rate limit will be untouched. There is also an "X-Poll-Interval" header that specifies how often (in seconds) you are allowed to poll. In times of high server load, the time may increase. Please obey the header. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing events 530491074 | |
559902818 | https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559902818 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14 | MDEyOklzc3VlQ29tbWVudDU1OTkwMjgxOA== | simonw 9599 | 2019-11-30T01:32:38Z | 2019-11-30T01:32:38Z | MEMBER | Prototype: ``` pip install sqlite-utils paginate-json paginate-json "https://api.github.com/users/simonw/events" | sqlite-utils insert /tmp/events.db events - --pk=id ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing events 530491074 | |
594151327 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594151327 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDU5NDE1MTMyNw== | simonw 9599 | 2020-03-03T20:26:15Z | 2020-03-03T20:32:23Z | MEMBER | Better version (since this also includes JSON array of repository topics): ```sql CREATE VIEW recent_releases AS select repos.rowid as rowid, json_object("label", repos.full_name, "href", repos.html_url) as repo, json_object( "href", releases.html_url, "label", releases.name ) as release, substr(releases.published_at, 0, 11) as date, releases.body as body_markdown, releases.published_at, coalesce(repos.topics, '[]') as topics from releases join repos on repos.id = releases.repo order by releases.published_at desc ``` That `repos.rowid as rowid` bit is necessary because otherwise clicking on a link in facet-by-topic doesn't return any results. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
594154644 | https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-594154644 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDU5NDE1NDY0NA== | simonw 9599 | 2020-03-03T20:33:57Z | 2020-03-03T20:33:57Z | MEMBER | `sqlite-utils` supports proper upserts now so this problem should be easy to fix. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | stargazers command, refs #4 516763727 | |
594155249 | https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594155249 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDU5NDE1NTI0OQ== | simonw 9599 | 2020-03-03T20:35:17Z | 2020-03-03T20:35:17Z | MEMBER | `swarm-to-sqlite` has an example of adding views here: https://github.com/dogsheep/swarm-to-sqlite/blob/f2c89dd613fb8a7f14e5267ccc2145463b996190/swarm_to_sqlite/utils.py#L141 I think that approach can be approved by first checking if the view exists, then dropping it, then recreating it. Could even try to see if the view exists and matches what we were going to set it to and do nothing if that is the case. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add this view for seeing new releases 520756546 | |
597354514 | https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597354514 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17 | MDEyOklzc3VlQ29tbWVudDU5NzM1NDUxNA== | simonw 9599 | 2020-03-10T22:37:45Z | 2020-03-10T22:37:45Z | MEMBER | I should add an option to stop the moment you see a commit you have fetched before. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing commits 578883725 | |
597358364 | https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597358364 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17 | MDEyOklzc3VlQ29tbWVudDU5NzM1ODM2NA== | simonw 9599 | 2020-03-10T22:50:20Z | 2020-03-11T01:18:36Z | MEMBER | By default it will stop when it sees a commit that has already been stored. You will be able to over-ride that behaviour using `--all`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing commits 578883725 | |
513440090 | https://github.com/dogsheep/healthkit-to-sqlite/issues/4#issuecomment-513440090 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDUxMzQ0MDA5MA== | simonw 9599 | 2019-07-20T06:11:50Z | 2019-07-20T06:11:50Z | MEMBER | Some examples: https://github.com/dogsheep/healthkit-to-sqlite/blob/d016e70c31cf84ba0f5ec3102546db54a51aaffb/tests/export.xml#L4-L13 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import Records 470640505 | |
513437463 | https://github.com/dogsheep/healthkit-to-sqlite/issues/1#issuecomment-513437463 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDUxMzQzNzQ2Mw== | simonw 9599 | 2019-07-20T05:19:59Z | 2019-07-20T05:19:59Z | MEMBER | I ran xml_analyser against the XML HealthKit `export.xml` file and got the following results: ```python { 'ActivitySummary': {'attr_counts': {'activeEnergyBurned': 980, 'activeEnergyBurnedGoal': 980, 'activeEnergyBurnedUnit': 980, 'appleExerciseTime': 980, 'appleExerciseTimeGoal': 980, 'appleStandHours': 980, 'appleStandHoursGoal': 980, 'dateComponents': 980}, 'child_counts': {}, 'count': 980, 'parent_counts': {'HealthData': 980}}, 'Correlation': {'attr_counts': {'creationDate': 1, 'endDate': 1, 'sourceName': 1, 'sourceVersion': 1, 'startDate': 1, 'type': 1}, 'child_counts': {'MetadataEntry': 1, 'Record': 2}, 'count': 1, 'parent_counts': {'HealthData': 1}}, 'ExportDate': {'attr_counts': {'value': 1}, 'child_counts': {}, 'count': 1, 'parent_counts': {'HealthData': 1}}, 'HealthData': {'attr_counts': {'locale': 1}, 'child_counts': {'ActivitySummary': 980, 'Correlation': 1, 'ExportDate': 1, 'Me': 1, 'Record': 2672231, 'Workout': 663}, 'count': 1, 'parent_counts': {}}, 'HeartRateVariabilityMetadataList': {'attr_counts': {}, 'child_counts': {'InstantaneousBeatsPerMinute': 93653}, 'count': 2318, … | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Use XML Analyser to figure out the structure of the export XML 470637068 | |
513439411 | https://github.com/dogsheep/healthkit-to-sqlite/issues/2#issuecomment-513439411 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDUxMzQzOTQxMQ== | simonw 9599 | 2019-07-20T05:58:57Z | 2019-07-20T05:58:57Z | MEMBER | ```python 'Workout': {'attr_counts': {'creationDate': 663, 'device': 230, 'duration': 663, 'durationUnit': 663, 'endDate': 663, 'sourceName': 663, 'sourceVersion': 663, 'startDate': 663, 'totalDistance': 663, 'totalDistanceUnit': 663, 'totalEnergyBurned': 663, 'totalEnergyBurnedUnit': 663, 'workoutActivityType': 663}, 'child_counts': {'MetadataEntry': 1928, 'WorkoutEvent': 2094, 'WorkoutRoute': 340}, 'count': 663, 'parent_counts': {'HealthData': 663}}, 'WorkoutEvent': {'attr_counts': {'date': 2094, 'duration': 837, 'durationUnit': 837, 'type': 2094}, 'child_counts': {}, 'count': 2094, 'parent_counts': {'Workout': 2094}}, 'WorkoutRoute': {'attr_counts': {'creationDate': 340, 'endDate': 340, 'sourceName': 340, 'sourceVersion': 340, 'startDate': 340}, 'child_counts': {'Location': 398683, 'MetadataEntry': 546}, 'count': 340, 'parent_counts': {'Workout': 340}}} ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Import workouts 470637152 | |
513514978 | https://github.com/dogsheep/healthkit-to-sqlite/issues/5#issuecomment-513514978 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDUxMzUxNDk3OA== | simonw 9599 | 2019-07-21T02:55:12Z | 2019-07-21T02:55:12Z | MEMBER | I'm going to show this by default. Users can pass `-s` or `--silent` to disable the progress bar. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add progress bar 470691622 | |
513625406 | https://github.com/dogsheep/healthkit-to-sqlite/issues/5#issuecomment-513625406 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDUxMzYyNTQwNg== | simonw 9599 | 2019-07-22T03:20:16Z | 2019-07-22T03:20:16Z | MEMBER | It now renders like this: ``` Importing from HealthKit [#-----------------------------------] 5% 00:01:33 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add progress bar 470691622 | |
513626742 | https://github.com/dogsheep/healthkit-to-sqlite/issues/6#issuecomment-513626742 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDUxMzYyNjc0Mg== | simonw 9599 | 2019-07-22T03:28:55Z | 2019-07-22T03:28:55Z | MEMBER | Here's what it looks like now as separate tables: <img width="1050" alt="hello9_and_Populate__endpoint__key_in_ASGI_scope_·_Issue__537_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/61604322-eba05000-abf5-11e9-8f8b-d0204e9a705a.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Break up records into different tables for each type 470856782 | |
514496725 | https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514496725 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7 | MDEyOklzc3VlQ29tbWVudDUxNDQ5NjcyNQ== | simonw 9599 | 2019-07-24T06:20:59Z | 2019-07-24T06:20:59Z | MEMBER | I'm using https://pypi.org/project/memory-profiler/ to explore this in more detail: ``` $ pip install memory-profiler matplotlib ``` Then: ``` $ mprof run healthkit-to-sqlite ~/Downloads/healthkit-export.zip healthkit.db $ mprof plot ``` <img width="1128" alt="Screen Shot 2019-07-24 at 8 17 06 AM" src="https://user-images.githubusercontent.com/9599/61769849-f3294a00-adeb-11e9-9026-02c24209530f.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Script uses a lot of RAM 472097220 | |
514498221 | https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514498221 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7 | MDEyOklzc3VlQ29tbWVudDUxNDQ5ODIyMQ== | simonw 9599 | 2019-07-24T06:26:49Z | 2019-07-24T06:26:49Z | MEMBER | Adding `el.clear()` got me a huge improvement: <img width="1128" alt="Screen Shot 2019-07-24 at 8 23 26 AM" src="https://user-images.githubusercontent.com/9599/61770130-c3c70d00-adec-11e9-886a-93a2cea5bf4b.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Script uses a lot of RAM 472097220 | |
514500253 | https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514500253 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7 | MDEyOklzc3VlQ29tbWVudDUxNDUwMDI1Mw== | simonw 9599 | 2019-07-24T06:34:28Z | 2019-07-24T06:34:28Z | MEMBER | Clearing the root element each time saved even more: <img width="1128" alt="Screen Shot 2019-07-24 at 8 30 38 AM" src="https://user-images.githubusercontent.com/9599/61770555-d3932100-aded-11e9-8ffe-bebd682f94ed.png"> | {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 2, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Script uses a lot of RAM 472097220 | |
515226724 | https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515226724 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9 | MDEyOklzc3VlQ29tbWVudDUxNTIyNjcyNA== | simonw 9599 | 2019-07-25T21:46:01Z | 2019-07-25T21:46:01Z | MEMBER | I can work around this here (prior to the fix in sqlite-utils) by setting the batch size to something a bit lower here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Too many SQL variables 472429048 | |
515322294 | https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515322294 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9 | MDEyOklzc3VlQ29tbWVudDUxNTMyMjI5NA== | simonw 9599 | 2019-07-26T06:07:12Z | 2019-07-26T06:07:12Z | MEMBER | @tholo this should be fixed in just-released version 0.3.2 - could you run a `pip install -U healthkit-to-sqlite` and let me know if it works for you now? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Too many SQL variables 472429048 | |
526701674 | https://github.com/dogsheep/swarm-to-sqlite/issues/2#issuecomment-526701674 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDUyNjcwMTY3NA== | simonw 9599 | 2019-08-30T18:24:26Z | 2019-08-30T18:24:26Z | MEMBER | I renamed `--file` to `--load` in 0e5b6025c6f9823ff81aa8aae1cbff5c45e57baf | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --save option to dump checkins to a JSON file on disk 487598468 | |
526853542 | https://github.com/dogsheep/swarm-to-sqlite/issues/4#issuecomment-526853542 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDUyNjg1MzU0Mg== | simonw 9599 | 2019-08-31T18:06:32Z | 2019-08-31T18:06:32Z | MEMBER | https://your-foursquare-oauth-token.glitch.me/ Source code: https://glitch.com/~your-foursquare-oauth-token | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Online tool for getting a Foursquare OAuth token 487601121 | |
527200332 | https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-527200332 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDUyNzIwMDMzMg== | simonw 9599 | 2019-09-02T16:32:20Z | 2019-09-02T16:32:39Z | MEMBER | Also needed: an option for "fetch all checkins created within the last X days". This should help provide support for that Swarm feature where you can retroactively checkin to places in the past. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Option to fetch only checkins more recent than the current max checkin 487600595 | |
527682713 | https://github.com/dogsheep/twitter-to-sqlite/issues/4#issuecomment-527682713 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDUyNzY4MjcxMw== | simonw 9599 | 2019-09-03T23:48:57Z | 2019-09-03T23:48:57Z | MEMBER | One interesting challenge here is that the JSON format for tweets in the archive is subtly different from the JSON format currently returned by the API. If we want to keep the tweets in the same database table (which feels like the right thing to me) we'll need to handle this. One thing we can do is have a column for `from_archive` which is set to 1 for tweets that were recovered from the archive. We can also ensure that tweets from the API always over-write the version that came from the archive (using `.upsert()`) while tweets from the archive use `.insert(..., ignore=True)` to avoid over-writing a better version that came from the API. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for importing data from a Twitter Export file 488835586 | |
527684202 | https://github.com/dogsheep/twitter-to-sqlite/issues/5#issuecomment-527684202 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDUyNzY4NDIwMg== | simonw 9599 | 2019-09-03T23:56:28Z | 2019-09-03T23:56:28Z | MEMBER | I previously used betamax here: https://github.com/simonw/github-contents/blob/master/test_github_contents.py | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Write tests that simulate the Twitter API 488874815 | |
527954898 | https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527954898 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDUyNzk1NDg5OA== | simonw 9599 | 2019-09-04T15:31:46Z | 2019-09-04T15:31:46Z | MEMBER | I'm going to call this `twitter-to-sqlite user-timeline` to reflect the language used to describe the API endpoint: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | "twitter-to-sqlite user-timeline" command for pulling tweets by a specific user 488833698 | |
527955302 | https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527955302 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDUyNzk1NTMwMg== | simonw 9599 | 2019-09-04T15:32:39Z | 2019-09-04T15:32:39Z | MEMBER | Rate limit is 900 / 15 minutes which is 1 call per second. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | "twitter-to-sqlite user-timeline" command for pulling tweets by a specific user 488833698 | |
527990908 | https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527990908 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDUyNzk5MDkwOA== | simonw 9599 | 2019-09-04T16:57:24Z | 2019-09-04T16:57:24Z | MEMBER | I just tried this using `max_id=` pagination as described in [Working with timelines](https://developer.twitter.com/en/docs/tweets/timelines/guides/working-with-timelines) and I got back all 17,759 of my tweets. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | "twitter-to-sqlite user-timeline" command for pulling tweets by a specific user 488833698 | |
529240286 | https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-529240286 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDUyOTI0MDI4Ng== | simonw 9599 | 2019-09-08T20:48:33Z | 2019-09-08T20:48:33Z | MEMBER | ```ATTACH DATABASE "file:blah.db?mode=ro" AS foo``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --sql and --attach options for feeding commands from SQL queries 490803176 | |
529239307 | https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-529239307 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDUyOTIzOTMwNw== | simonw 9599 | 2019-09-08T20:36:49Z | 2019-09-08T20:36:49Z | MEMBER | `--attach` can optionally take a name for the database connection alias like this: $ twitter-to-sqlite users-lookup users.db --attach foo:attending.db ... If you omit the `alias:` bit the stem of the database (without the file extension) will be used. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --sql and --attach options for feeding commands from SQL queries 490803176 | |
530028567 | https://github.com/dogsheep/twitter-to-sqlite/issues/9#issuecomment-530028567 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/9 | MDEyOklzc3VlQ29tbWVudDUzMDAyODU2Nw== | simonw 9599 | 2019-09-10T16:59:25Z | 2019-09-10T16:59:25Z | MEMBER | By default in SQLite foreign key constraints are not enforced (you need to run `PRAGMA foreign_keys = ON;` to enforce them). We will take advantage of this - even though the `following` table has foreign keys against user we will allow IDs to populate that table without a corresponding user record. In the future we may add a command that can backfill missing user records. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | followers-ids and friends-ids subcommands 491791152 | |
530417631 | https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-530417631 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDUzMDQxNzYzMQ== | simonw 9599 | 2019-09-11T14:52:44Z | 2019-09-14T19:09:22Z | MEMBER | - [x] This needs documentation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --sql and --attach options for feeding commands from SQL queries 490803176 | |
531404891 | https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-531404891 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDUzMTQwNDg5MQ== | simonw 9599 | 2019-09-13T22:01:57Z | 2019-09-13T22:01:57Z | MEMBER | I also wrote about this in https://simonwillison.net/2019/Sep/13/weeknotestwitter-sqlite-datasette-rure/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | --sql and --attach options for feeding commands from SQL queries 490803176 | |
531516956 | https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531516956 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDUzMTUxNjk1Ng== | simonw 9599 | 2019-09-14T21:56:31Z | 2019-09-14T21:56:31Z | MEMBER | https://api.github.com/users/simonw/repos It would be useful to be able to fetch stargazers, forks etc as well. Not sure if that should be a separate command or a `--stargazers` option to this command. Probably a separate command since `issues` is a separate command already. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch all repos belonging to a user or organization 493670426 | |
531517083 | https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531517083 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDUzMTUxNzA4Mw== | simonw 9599 | 2019-09-14T21:58:42Z | 2019-09-14T21:58:42Z | MEMBER | Split stargazers into #4 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch all repos belonging to a user or organization 493670426 | |
531517138 | https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-531517138 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDUzMTUxNzEzOA== | simonw 9599 | 2019-09-14T21:59:59Z | 2019-09-14T21:59:59Z | MEMBER | Paginate through https://api.github.com/repos/simonw/datasette/stargazers Send `Accept: application/vnd.github.v3.star+json` to get the `starred_at` dates. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command to fetch stargazers for one or more repos 493670730 | |
538711918 | https://github.com/dogsheep/twitter-to-sqlite/issues/11#issuecomment-538711918 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDUzODcxMTkxOA== | simonw 9599 | 2019-10-06T04:54:17Z | 2019-10-06T04:54:17Z | MEMBER | Shipped in 0.6. Here's the documentation: https://github.com/dogsheep/twitter-to-sqlite#capturing-tweets-in-real-time-with-track-and-follow | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for recording real-time tweets from the streaming API 503045221 | |
538847446 | https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-538847446 | https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDUzODg0NzQ0Ng== | simonw 9599 | 2019-10-07T05:41:17Z | 2019-10-07T05:41:17Z | MEMBER | Prototype code: ```python offset = 0 fetched = [] size = 500 while True: page = requests.get("https://getpocket.com/v3/get", { "consumer_key": consumer_key, "access_token": access_token, "sort": "oldest", "detailType": "complete", "count": size, "offset": offset, }).json() print(offset) fetched.append(page) offset += size if not len(page["list"]): break ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Use better pagination (and implement progress bar) 503233021 | |
538847796 | https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-538847796 | https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDUzODg0Nzc5Ng== | simonw 9599 | 2019-10-07T05:43:30Z | 2019-10-07T05:43:30Z | MEMBER | We can persist the `since` value in its own single-row table. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Track and use the 'since' value 503234169 | |
538804815 | https://github.com/dogsheep/twitter-to-sqlite/issues/13#issuecomment-538804815 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDUzODgwNDgxNQ== | simonw 9599 | 2019-10-07T00:33:49Z | 2019-10-07T00:33:49Z | MEMBER | Documentation: https://github.com/dogsheep/twitter-to-sqlite#retrieve-tweets-in-bulk | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | statuses-lookup command 503085013 | |
643393506 | https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643393506 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDY0MzM5MzUwNg== | simonw 9599 | 2020-06-12T17:21:14Z | 2020-06-12T17:21:14Z | MEMBER | I only install SQLite for this: https://github.com/dogsheep/github-to-sqlite/blob/c0d54e0260468be38152293df5abd775c068495d/.github/workflows/deploy-demo.yml#L77-L78 I'm going to remove the need to install sqlite3 by making this possible with sqlite-utils: https://github.com/simonw/sqlite-utils/issues/115 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo deploy is broken 637899539 | |
643414646 | https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643414646 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDY0MzQxNDY0Ng== | simonw 9599 | 2020-06-12T18:06:48Z | 2020-06-12T18:06:48Z | MEMBER | That fixed it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo deploy is broken 637899539 | |
645599881 | https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645599881 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47 | MDEyOklzc3VlQ29tbWVudDY0NTU5OTg4MQ== | simonw 9599 | 2020-06-17T20:13:48Z | 2020-06-17T20:13:48Z | MEMBER | I've now figured out how to compile specific SQLite versions to help replicate this problem: https://github.com/simonw/til/blob/master/sqlite/ld-preload.md Next step: replicate the problem! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fall back to FTS4 if FTS5 is not available 639542974 | |
645512127 | https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47 | MDEyOklzc3VlQ29tbWVudDY0NTUxMjEyNw== | simonw 9599 | 2020-06-17T17:24:22Z | 2020-06-17T17:24:22Z | MEMBER | That means your version of SQLite is old enough that it doesn't support the FTS5 extension. Could you share what operating system you're running, and what the output is that you get from running this? python -c 'import sqlite3; print(sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0])' I can teach this tool to fall back on FTS4 if FTS5 isn't available. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fall back to FTS4 if FTS5 is not available 639542974 | |
653947916 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653947916 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk0NzkxNg== | simonw 9599 | 2020-07-05T22:40:47Z | 2020-07-05T22:40:47Z | MEMBER | Might be that it's not got enough RAM. I'll try deploying to a larger instance. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653960989 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653960989 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MDk4OQ== | simonw 9599 | 2020-07-06T00:32:34Z | 2020-07-06T00:32:34Z | MEMBER | Same error. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653962418 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962418 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjQxOA== | simonw 9599 | 2020-07-06T00:41:38Z | 2020-07-06T00:41:38Z | MEMBER | https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/logs?project=datasette-222320 has some clues. <img width="818" alt="github-to-sqlite_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/86545887-92086280-bee6-11ea-8787-b3b85a7bb4b9.png"> <img width="1451" alt="github-to-sqlite_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/86545917-c11ed400-bee6-11ea-9cf8-957911ca9f6a.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653962530 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962530 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjUzMA== | simonw 9599 | 2020-07-06T00:42:13Z | 2020-07-06T00:42:13Z | MEMBER | So it looks like it's the ASGI lifespan change I made in https://github.com/simonw/datasette/commit/16f592247a2a0e140ada487e9972645406dcae69 - It must be incompatible with one of the plugins. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653962669 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962669 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjY2OQ== | simonw 9599 | 2020-07-06T00:42:57Z | 2020-07-06T00:42:57Z | MEMBER | https://github-to-sqlite.dogsheep.net/-/plugins ```json [ { "name": "datasette-json-html", "static": false, "templates": false, "version": "0.6", "hooks": [ "prepare_connection", "render_cell" ] }, { "name": "datasette-render-markdown", "static": false, "templates": false, "version": "1.1.2", "hooks": [ "extra_template_vars", "render_cell" ] }, { "name": "datasette-pretty-json", "static": false, "templates": false, "version": "0.2", "hooks": [ "render_cell" ] }, { "name": "datasette-search-all", "static": false, "templates": true, "version": "0.2.1", "hooks": [ "asgi_wrapper", "extra_template_vars" ] }, { "name": "datasette-vega", "static": true, "templates": false, "version": "0.6.2", "hooks": [ "extra_css_urls", "extra_js_urls" ] } ] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653962708 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962708 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2MjcwOA== | simonw 9599 | 2020-07-06T00:43:10Z | 2020-07-06T00:43:10Z | MEMBER | I bet it's datasette-search-all. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
653966670 | https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653966670 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 | MDEyOklzc3VlQ29tbWVudDY1Mzk2NjY3MA== | simonw 9599 | 2020-07-06T01:07:02Z | 2020-07-06T01:07:02Z | MEMBER | OK that fix worked.https://github.com/dogsheep/github-to-sqlite/runs/839764768?check_suite_focus=true | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Demo is failing to deploy 651159727 | |
660547502 | https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660547502 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDY2MDU0NzUwMg== | simonw 9599 | 2020-07-18T21:50:37Z | 2020-07-18T21:50:37Z | MEMBER | ``` $ github-to-sqlite tags tags.db simonw/datasette dogsheep/github-to-sqlite $ sqlite-utils tables tags.db --counts [{"table": "users", "count": 2}, {"table": "licenses", "count": 1}, {"table": "repos", "count": 2}, {"table": "tags", "count": 76}, {"table": "licenses_fts", "count": 1}, {"table": "licenses_fts_data", "count": 3}, {"table": "licenses_fts_idx", "count": 1}, {"table": "licenses_fts_docsize", "count": 1}, {"table": "licenses_fts_config", "count": 1}, {"table": "repos_fts", "count": 2}, {"table": "repos_fts_data", "count": 3}, {"table": "repos_fts_idx", "count": 1}, {"table": "repos_fts_docsize", "count": 2}, {"table": "repos_fts_config", "count": 1}, {"table": "users_fts", "count": 2}, {"table": "users_fts_data", "count": 3}, {"table": "users_fts_idx", "count": 1}, {"table": "users_fts_docsize", "count": 2}, {"table": "users_fts_config", "count": 1}] $ sqlite-utils rows tags.db tags [{"repo_id": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"}, {"repo_id": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"}, {"repo_id": 107914493, "name": "0.45a4", "sha": "265483173bc8341dc02c8b782b9b59d2ce8bbedb"}, {"repo_id": 107914493, "name": "0.45a3", "sha": "1f55a4a2b68fa65e56a28baeb7f44122fdeca7e7"}, {"repo_id": 107914493, "name": "0.45a2", "sha": "1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f"}, {"repo_id": 107914493, "name": "0.45a1", "sha": "b59b92b1b0517cf18fa748ff9d0a0bf86298dd43"}, {"repo_id": 107914493, "name": "0.45a0", "sha": "dda932d818b34ccab11730a76554f0a3748d8348"}, {"repo_id": 107914493, "name": "0.44", "sha": "b906030235efbdff536405d66078f4868ce0d3bd"}, {"repo_id": 107914493, "name": "0.43", "sha": "d56f402822df102f9cf1a9a056449d01a15e3aae"}, {"repo_id": 107914493, "name": "0.42", "sha": "af6c6c5d6f929f951c0e63bfd1c82e37a071b50f"}, {"repo_id": 107914493, "name": "0.41", "sha": "182e5c8745c94576718315f7596ccc81e5e2417b"}, {"repo_id": 107914493, "name": "0.40", "sha": "8da108193b08ab… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite tags command for fetching tags 660355904 | |
660548780 | https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660548780 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDY2MDU0ODc4MA== | simonw 9599 | 2020-07-18T22:02:37Z | 2020-07-18T23:05:56Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/tags?_facet=repo | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite tags command for fetching tags 660355904 | |
660551397 | https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660551397 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDY2MDU1MTM5Nw== | simonw 9599 | 2020-07-18T22:27:32Z | 2020-07-18T23:05:45Z | MEMBER | ```sql with most_recent_releases as ( with ranked as ( select repo, tag_name, published_at, row_number() OVER ( partition BY repo ORDER BY published_at DESC ) rank FROM releases ) select * from ranked where rank = 1 ) select repos.full_name as repo, most_recent_releases.tag_name as release, commits.committer_date as release_commit_date, ( select count(*) from commits c2 where c2.repo = repos.id and c2.committer_date > commits.committer_date ) as commits_since_release, 'https://github.com/' || repos.full_name || '/compare/' || most_recent_releases.tag_name || '...' || repos.default_branch as view_commits from most_recent_releases join repos on most_recent_releases.repo = repos.id join tags on tags.repo = repos.id and tags.name = most_recent_releases.tag_name join commits on tags.sha = commits.sha order by commits_since_release desc ``` repo | release | release_commit_date | commits_since_release | view_commits -- | -- | -- | -- | -- simonw/datasette | 0.45 | 2020-07-01T21:43:07Z | 9 | https://github.com/simonw/datasette/compare/0.45...master dogsheep/twitter-to-sqlite | 0.21.1 | 2020-04-30T18:20:43Z | 2 | https://github.com/dogsheep/twitter-to-sqlite/compare/0.21.1...master dogsheep/github-to-sqlite | 2.3 | 2020-07-09T23:26:34Z | 2 | https://github.com/dogsheep/github-to-sqlite/compare/2.3...master dogsheep/dogsheep-photos | 0.4.1 | 2020-05-25T20:11:20Z | 2 | https://github.com/dogsheep/dogsheep-photos/compare/0.4.1...master dogsheep/swarm-to-sqlite | 0.3.1 | 2020-03-28T02:29:41Z | 1 | https://github.com/dogsheep/swarm-to-sqlite/compare/0.3.1...master dogsheep/hacker-news-to-sqlite | 0.3.1 | 2020-03-21T22:39:34Z | 1 | https://github.com/dogsheep/hacker-news-to-sqlite/compare/0.3.1...master simonw/sqlite-utils | 2.11 | 2020-07-08T17:36:07Z | 0 | https://github.com/simonw/sqlite-utils/compare/2.11...master dog… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite tags command for fetching tags 660355904 | |
660553646 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553646 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1MzY0Ng== | simonw 9599 | 2020-07-18T22:51:41Z | 2020-07-18T22:51:41Z | MEMBER | I could fix this by putting `REFRESH_DB` in a commit message: https://github.com/dogsheep/github-to-sqlite/blob/4ae4aa6f172344b19ff3513707195ee6d2654bd4/.github/workflows/deploy-demo.yml#L41-L46 But... doing so would lose the data I've collected in https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc concerning the first time each dependent repo was spotted. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660553711 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553711 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1MzcxMQ== | simonw 9599 | 2020-07-18T22:52:16Z | 2020-07-18T22:52:16Z | MEMBER | I think the best fix is to download the `github.db` database, manually fix it and then manually deploy it to Cloud Run from my laptop. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660554162 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554162 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1NDE2Mg== | simonw 9599 | 2020-07-18T22:56:58Z | 2020-07-18T22:56:58Z | MEMBER | Manually fixing the database: ``` $ wget 'https://github-to-sqlite.dogsheep.net/github.db' --2020-07-18 15:52:33-- https://github-to-sqlite.dogsheep.net/github.db Resolving github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)... 172.217.5.115 Connecting to github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)|172.217.5.115|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 14626816 (14M) [application/octet-stream] Saving to: ‘github.db’ github.db 100%[============================================================================>] 13.95M 1.22MB/s in 18s 2020-07-18 15:52:53 (773 KB/s) - ‘github.db’ saved [14626816/14626816] $ sqlite3 github.db SQLite version 3.28.0 2019-04-15 14:49:49 Enter ".help" for usage hints. sqlite> drop table tags; sqlite> ^D $ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils dogsheep/healthkit-to-sqlite dogsheep/swarm-to-sqlite dogsheep/twitter-to-sqlite dogsheep/inaturalist-to-sqlite dogsheep/google-takeout-to-sqlite dogsheep/github-to-sqlite dogsheep/genome-to-sqlite dogsheep/pocket-to-sqlite dogsheep/hacker-news-to-sqlite dogsheep/dogsheep-photos $ sqlite-utils tables github.db --counts [{"table": "users", "count": 4048}, {"table": "repos", "count": 210}, ... {"table": "stars", "count": 4140}, {"table": "tags", "count": 188}] $ sqlite-utils rows github.db tags [{"repo": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"}, {"repo": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"}, ... ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660554299 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554299 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1NDI5OQ== | simonw 9599 | 2020-07-18T22:58:24Z | 2020-07-18T23:02:52Z | MEMBER | Deploying the fixed version like this: ``` $ gcloud config set run/region us-central1 $ gcloud config set project datasette-222320 $ datasette publish cloudrun /tmp/github.db \ -m demo-metadata.json \ --service github-to-sqlite \ --install=py-gfm \ --install='datasette-search-all>=0.3' \ --install='datasette-render-markdown>=1.1.2' \ --install=datasette-pretty-json \ --install=datasette-json-html \ --install=datasette-vega ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660554811 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554811 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1NDgxMQ== | simonw 9599 | 2020-07-18T23:03:13Z | 2020-07-18T23:03:13Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/tags now shows a `repo` column instead of a `repo_id` column. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660536265 | https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660536265 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDY2MDUzNjI2NQ== | simonw 9599 | 2020-07-18T20:15:12Z | 2020-07-18T20:15:12Z | MEMBER | I want to create a SQL query which shows me all of my repositories that have commits that are NOT in the most recent release. The releases table doesn't have enough information for this because it doesn't tell you the commit hash associated with each release, just the tag: https://github-to-sqlite.dogsheep.net/github/releases | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite tags command for fetching tags 660355904 | |
662626901 | https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662626901 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48 | MDEyOklzc3VlQ29tbWVudDY2MjYyNjkwMQ== | simonw 9599 | 2020-07-22T18:54:53Z | 2020-07-22T18:54:53Z | MEMBER | I'm going to use a GitHub Action to run `npx markdown-toc README.md -i` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add a table of contents to the README 663976976 | |
662630868 | https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662630868 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48 | MDEyOklzc3VlQ29tbWVudDY2MjYzMDg2OA== | simonw 9599 | 2020-07-22T19:03:02Z | 2020-07-22T19:03:02Z | MEMBER | Done! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add a table of contents to the README 663976976 | |
663143160 | https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-663143160 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48 | MDEyOklzc3VlQ29tbWVudDY2MzE0MzE2MA== | simonw 9599 | 2020-07-23T17:46:07Z | 2020-07-23T17:46:07Z | MEMBER | Frustratingly, these links don't work on PyPI: https://pypi.org/project/twitter-to-sqlite/ There's an issue about that here: https://github.com/pypa/readme_renderer/issues/169 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add a table of contents to the README 663976976 | |
669241341 | https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-669241341 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDY2OTI0MTM0MQ== | simonw 9599 | 2020-08-05T14:55:14Z | 2020-08-05T14:55:14Z | MEMBER | Looks like there's a column that's not consistently there, so the table got created without it. Easiest fix is to add `alter=True` to this line: https://github.com/dogsheep/swarm-to-sqlite/blob/f4a82633da927cde672c9d9af92930bfca2e3ddf/swarm_to_sqlite/utils.py#L94 That will cause `sqlite-utils` to notice if there's a missing column and add it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Error thrown: table photos has no column named hasSticker 648245071 | |
675259273 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-675259273 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDY3NTI1OTI3Mw== | simonw 9599 | 2020-08-18T05:28:32Z | 2020-08-18T05:28:32Z | MEMBER | Oh that's interesting - i didn't realize "reviews" were a separate concept. I'd definitely accept a pull request adding those! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: pull request reviews and comments 664485022 | |
675509550 | https://github.com/dogsheep/github-to-sqlite/issues/39#issuecomment-675509550 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDY3NTUwOTU1MA== | simonw 9599 | 2020-08-18T14:23:56Z | 2020-08-18T14:23:56Z | MEMBER | I think this is fixed: https://github-to-sqlite.dogsheep.net/github/issues?_facet=repo | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | issues foreign key to repo isn't working 613777056 | |
675523053 | https://github.com/dogsheep/github-to-sqlite/issues/47#issuecomment-675523053 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/47 | MDEyOklzc3VlQ29tbWVudDY3NTUyMzA1Mw== | simonw 9599 | 2020-08-18T14:45:53Z | 2020-08-18T14:45:53Z | MEMBER | ``` % github-to-sqlite emojis emojis.db --fetch [########----------------------------] 397/1682 23% 00:03:43 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | emojis command 681086659 | |
684250044 | https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-684250044 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3 | MDEyOklzc3VlQ29tbWVudDY4NDI1MDA0NA== | simonw 9599 | 2020-09-01T05:01:09Z | 2020-09-01T05:01:23Z | MEMBER | Maybe this starts out as a custom templated canned query. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Datasette plugin to provide custom page for running faceted, ranked searches 689810340 | |
684395444 | https://github.com/dogsheep/dogsheep-beta/issues/4#issuecomment-684395444 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/4 | MDEyOklzc3VlQ29tbWVudDY4NDM5NTQ0NA== | simonw 9599 | 2020-09-01T06:00:03Z | 2020-09-01T06:00:03Z | MEMBER | I ran `sqlite-utils optimize beta.db` against my test DB and the size reduced from 183M to 176M - and a 450ms search ran in 359ms. So not a huge improvement but still worthwhile. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Optimize the FTS table 689839399 | |
684424396 | https://github.com/dogsheep/pocket-to-sqlite/issues/3#issuecomment-684424396 | https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDY4NDQyNDM5Ng== | simonw 9599 | 2020-09-01T06:17:45Z | 2020-09-01T06:17:45Z | MEMBER | It looks like I could ignore the `image` column and synthesize a unique key from the data in the `images` column using `$item_id/$image_id`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Extract images into separate tables 503243784 | |
684425714 | https://github.com/dogsheep/pocket-to-sqlite/issues/5#issuecomment-684425714 | https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDY4NDQyNTcxNA== | simonw 9599 | 2020-09-01T06:18:32Z | 2020-09-01T06:18:32Z | MEMBER | Good suggestion, I'll setup a demo somewhere. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a demo 629473827 | |
685115519 | https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685115519 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2 | MDEyOklzc3VlQ29tbWVudDY4NTExNTUxOQ== | simonw 9599 | 2020-09-01T20:31:57Z | 2020-09-01T20:31:57Z | MEMBER | Actually this doesn't work: you can't turn on stemming for specific tables, because all of the content goes into a single `search_index` table which is configured the same way. So stemming needs to be a global option. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Apply porter stemming 689809225 | |
685121074 | https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685121074 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2 | MDEyOklzc3VlQ29tbWVudDY4NTEyMTA3NA== | simonw 9599 | 2020-09-01T20:42:00Z | 2020-09-01T20:42:00Z | MEMBER | Documentation at the bottom of the Usage section here: https://github.com/dogsheep/dogsheep-beta/blob/0.2/README.md#usage | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Apply porter stemming 689809225 | |
685895540 | https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685895540 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7 | MDEyOklzc3VlQ29tbWVudDY4NTg5NTU0MA== | simonw 9599 | 2020-09-02T17:46:44Z | 2020-09-02T17:46:44Z | MEMBER | Some opet questions about this: - Should I restrict to two exclusive categories here, or should I have a generic category mechanism that can be expanded to more than two? - Should an item be able to exist in more than one category? Do I want to be able to mark an indexed item as both by-me and liked-by-me for example? This question is more interesting if the number of categories is greater than two. - How should this be modeled? Single column, multiple boolean columns, JSON array, m2m against separate table? - What's the best way to make this performant | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for differentiating between "by me" and "liked by me" 691265198 | |
685960072 | https://github.com/dogsheep/dogsheep-beta/issues/8#issuecomment-685960072 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/8 | MDEyOklzc3VlQ29tbWVudDY4NTk2MDA3Mg== | simonw 9599 | 2020-09-02T19:50:47Z | 2020-09-02T19:50:47Z | MEMBER | This doesn't actually help, because the Datasette table view page doesn't then support adding the `where search_index_fts match :query` bit. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create a view for running faceted searches 691369691 | |
685961809 | https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-685961809 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3 | MDEyOklzc3VlQ29tbWVudDY4NTk2MTgwOQ== | simonw 9599 | 2020-09-02T19:54:24Z | 2020-09-02T19:54:24Z | MEMBER | This should implement search highlighting too, as seen on https://til.simonwillison.net/til/search?q=cloud <img width="1029" alt="TIL_search__cloud" src="https://user-images.githubusercontent.com/9599/92029959-32c6a300-ed1b-11ea-8b5e-971950980c38.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Datasette plugin to provide custom page for running faceted, ranked searches 689810340 | |
685962280 | https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685962280 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7 | MDEyOklzc3VlQ29tbWVudDY4NTk2MjI4MA== | simonw 9599 | 2020-09-02T19:55:26Z | 2020-09-02T19:59:58Z | MEMBER | Relevant: https://charlesleifer.com/blog/a-tour-of-tagging-schemas-many-to-many-bitmaps-and-more/ SQLite supports bitwise operators Binary AND (&) and Binary OR (|) - I could try those. Not sure how they interact with indexes though. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for differentiating between "by me" and "liked by me" 691265198 | |
685965516 | https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685965516 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7 | MDEyOklzc3VlQ29tbWVudDY4NTk2NTUxNg== | simonw 9599 | 2020-09-02T20:01:54Z | 2020-09-02T20:01:54Z | MEMBER | Relevant post: https://sqlite.org/forum/forumpost/9f06fedaa5 - drh says: > Indexes are one-to-one. There is one entry in the index for each row in the table. > > You are asking for an index that is many-to-one - multiple index entries for each table row. > > A Full-Text Index is basically a many-to-one index. So if all of your array entries really are words, you could probably get this to work using a Full-Text Index. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for differentiating between "by me" and "liked by me" 691265198 | |
685966361 | https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966361 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7 | MDEyOklzc3VlQ29tbWVudDY4NTk2NjM2MQ== | simonw 9599 | 2020-09-02T20:03:29Z | 2020-09-02T20:03:41Z | MEMBER | I'm going to implement the first version of this as an indexed integer `category` column which has 1 for "about me" and 2 for "liked by me" - and space for other category numerals in the future, albeit a row can only belong to one category. I'll think about a full tagging system separately. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for differentiating between "by me" and "liked by me" 691265198 | |
685966707 | https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966707 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7 | MDEyOklzc3VlQ29tbWVudDY4NTk2NjcwNw== | simonw 9599 | 2020-09-02T20:04:08Z | 2020-09-02T20:04:08Z | MEMBER | I'll make `category` a foreign key to a `categories` table so Datasette can automatically show the `name` column. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for differentiating between "by me" and "liked by me" 691265198 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
updated_at (date) >30 ✖