{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/9#issuecomment-530028567", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/9", "id": 530028567, "node_id": "MDEyOklzc3VlQ29tbWVudDUzMDAyODU2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-10T16:59:25Z", "updated_at": "2019-09-10T16:59:25Z", "author_association": "MEMBER", "body": "By default in SQLite foreign key constraints are not enforced (you need to run `PRAGMA foreign_keys = ON;` to enforce them).\r\n\r\nWe will take advantage of this - even though the `following` table has foreign keys against user we will allow IDs to populate that table without a corresponding user record.\r\n\r\nIn the future we may add a command that can backfill missing user records.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 491791152, "label": "followers-ids and friends-ids subcommands"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-529239307", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8", "id": 529239307, "node_id": "MDEyOklzc3VlQ29tbWVudDUyOTIzOTMwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-08T20:36:49Z", "updated_at": "2019-09-08T20:36:49Z", "author_association": "MEMBER", "body": "`--attach` can optionally take a name for the database connection alias like this:\r\n\r\n $ twitter-to-sqlite users-lookup users.db --attach foo:attending.db ...\r\n\r\nIf you omit the `alias:` bit the stem of the database (without the file extension) will be used.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 490803176, "label": "--sql and --attach options for feeding commands from SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-529240286", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8", "id": 529240286, "node_id": "MDEyOklzc3VlQ29tbWVudDUyOTI0MDI4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-08T20:48:33Z", "updated_at": "2019-09-08T20:48:33Z", "author_association": "MEMBER", "body": "```ATTACH DATABASE \"file:blah.db?mode=ro\" AS foo```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 490803176, "label": "--sql and --attach options for feeding commands from SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-530417631", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8", "id": 530417631, "node_id": "MDEyOklzc3VlQ29tbWVudDUzMDQxNzYzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-11T14:52:44Z", "updated_at": "2019-09-14T19:09:22Z", "author_association": "MEMBER", "body": "- [x] This needs documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 490803176, "label": "--sql and --attach options for feeding commands from SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-531404891", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8", "id": 531404891, "node_id": "MDEyOklzc3VlQ29tbWVudDUzMTQwNDg5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-13T22:01:57Z", "updated_at": "2019-09-13T22:01:57Z", "author_association": "MEMBER", "body": "I also wrote about this in https://simonwillison.net/2019/Sep/13/weeknotestwitter-sqlite-datasette-rure/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 490803176, "label": "--sql and --attach options for feeding commands from SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/pull/59#issuecomment-924209583", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/59", "id": 924209583, "node_id": "IC_kwDODEm0Qs43FlGv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-09-21T17:37:34Z", "updated_at": "2021-09-21T17:37:34Z", "author_association": "MEMBER", "body": "Thanks for this!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 984942782, "label": "Fix for since_id bug, closes #58"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902327457", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902327457, "node_id": "IC_kwDODEm0Qs41yGyh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:53:25Z", "updated_at": "2021-08-19T23:53:25Z", "author_association": "MEMBER", "body": "I'm getting this too. Looking into it now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328369", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902328369, "node_id": "IC_kwDODEm0Qs41yHAx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:56:26Z", "updated_at": "2021-08-19T23:56:26Z", "author_association": "MEMBER", "body": "https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-user_timeline says the API has been replaced by the new v2 one, but it should still work - and the `since_id` parameter is still documented on that page.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328760", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902328760, "node_id": "IC_kwDODEm0Qs41yHG4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:57:41Z", "updated_at": "2021-08-19T23:57:41Z", "author_association": "MEMBER", "body": "Weird, added debug code and got this: `{'screen_name': 'simonw', 'count': 200, 'since_id': 'False', 'tweet_mode': 'extended'}` - so maybe it's a `twitter-to-sqlite` bug where somehow the string `False` is being passed somewhere.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329455", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902329455, "node_id": "IC_kwDODEm0Qs41yHRv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:59:56Z", "updated_at": "2021-08-19T23:59:56Z", "author_association": "MEMBER", "body": "This looks like the bug to me:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/197e69cec40052c423a5ed071feb5f7cccea41b9/twitter_to_sqlite/cli.py#L239-L241\r\n\r\n`type=str, default=False`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329884", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902329884, "node_id": "IC_kwDODEm0Qs41yHYc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T00:01:05Z", "updated_at": "2021-08-20T00:01:05Z", "author_association": "MEMBER", "body": "Maybe Click changed something which meant that this broke things when it didn't used to?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769957751", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56", "id": 769957751, "node_id": "MDEyOklzc3VlQ29tbWVudDc2OTk1Nzc1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-29T17:59:40Z", "updated_at": "2021-01-29T17:59:40Z", "author_association": "MEMBER", "body": "This is interesting - how did you create that initial table? Was this using the `twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip` command, or something else?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 796736607, "label": "Not all quoted statuses get fetched?"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/5#issuecomment-527684202", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/5", "id": 527684202, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzY4NDIwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-03T23:56:28Z", "updated_at": "2019-09-03T23:56:28Z", "author_association": "MEMBER", "body": "I previously used betamax here: https://github.com/simonw/github-contents/blob/master/test_github_contents.py", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488874815, "label": "Write tests that simulate the Twitter API"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/pull/49#issuecomment-902330301", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49", "id": 902330301, "node_id": "IC_kwDODEm0Qs41yHe9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T00:01:56Z", "updated_at": "2021-08-20T00:01:56Z", "author_association": "MEMBER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681575714, "label": "Document the use of --stop_after with favorites, refs #20"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662626901", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48", "id": 662626901, "node_id": "MDEyOklzc3VlQ29tbWVudDY2MjYyNjkwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-22T18:54:53Z", "updated_at": "2020-07-22T18:54:53Z", "author_association": "MEMBER", "body": "I'm going to use a GitHub Action to run `npx markdown-toc README.md -i`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 663976976, "label": "Add a table of contents to the README"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662630868", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48", "id": 662630868, "node_id": "MDEyOklzc3VlQ29tbWVudDY2MjYzMDg2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-22T19:03:02Z", "updated_at": "2020-07-22T19:03:02Z", "author_association": "MEMBER", "body": "Done!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 663976976, "label": "Add a table of contents to the README"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-663143160", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48", "id": 663143160, "node_id": "MDEyOklzc3VlQ29tbWVudDY2MzE0MzE2MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-23T17:46:07Z", "updated_at": "2020-07-23T17:46:07Z", "author_association": "MEMBER", "body": "Frustratingly, these links don't work on PyPI: https://pypi.org/project/twitter-to-sqlite/\r\n\r\nThere's an issue about that here: https://github.com/pypa/readme_renderer/issues/169", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 663976976, "label": "Add a table of contents to the README"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47", "id": 645512127, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NTUxMjEyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-17T17:24:22Z", "updated_at": "2020-06-17T17:24:22Z", "author_association": "MEMBER", "body": "That means your version of SQLite is old enough that it doesn't support the FTS5 extension.\r\n\r\nCould you share what operating system you're running, and what the output is that you get from running this?\r\n\r\n python -c 'import sqlite3; print(sqlite3.connect(\":memory:\").execute(\"select sqlite_version()\").fetchone()[0])'\r\n\r\nI can teach this tool to fall back on FTS4 if FTS5 isn't available.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 639542974, "label": "Fall back to FTS4 if FTS5 is not available"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645599881", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47", "id": 645599881, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NTU5OTg4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-17T20:13:48Z", "updated_at": "2020-06-17T20:13:48Z", "author_association": "MEMBER", "body": "I've now figured out how to compile specific SQLite versions to help replicate this problem: https://github.com/simonw/til/blob/master/sqlite/ld-preload.md\r\n\r\nNext step: replicate the problem!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 639542974, "label": "Fall back to FTS4 if FTS5 is not available"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/45#issuecomment-616029262", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45", "id": 616029262, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNjAyOTI2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-19T04:39:21Z", "updated_at": "2020-04-19T04:39:21Z", "author_association": "MEMBER", "body": "![44714E00-8CC5-46CD-9E48-1F4DD148FCC8](https://user-images.githubusercontent.com/9599/79679696-09b6d300-81bd-11ea-80e4-0653d92e4f58.jpeg)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602619330, "label": "Use raise_for_status() everywhere"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/43#issuecomment-615513491", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/43", "id": 615513491, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxMzQ5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:48:28Z", "updated_at": "2020-04-17T23:48:28Z", "author_association": "MEMBER", "body": "Released in 0.21.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602176870, "label": "\"twitter-to-sqlite lists\" command for retrieving a user's owned lists"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307019", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 606307019, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNzAxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:34:27Z", "updated_at": "2020-03-30T23:34:27Z", "author_association": "MEMBER", "body": "The count properties available for a user are:\r\n\r\n* followers_count\r\n* friends_count\r\n* listed_count\r\n* favourites_count\r\n* statuses_count\r\n\r\nMay as well track history for all of them? Should be pretty cheap to store.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307376", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 606307376, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNzM3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:35:40Z", "updated_at": "2020-03-30T23:39:15Z", "author_association": "MEMBER", "body": "I think five separate tables:\r\n\r\n* followers_count_history\r\n* friends_count_history\r\n* listed_count_history\r\n* favourites_count_history\r\n* statuses_count_history\r\n\r\nEach with the following structure:\r\n\r\n* datetime (ISO UTC)\r\n* user (ID, foreign key to users)\r\n* count (integer)\r\n\r\nI'm tempted to have a compound primary key here - user, datetime\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011421", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 607011421, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMTQyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:47:37Z", "updated_at": "2020-04-01T03:55:08Z", "author_association": "MEMBER", "body": "Actually a single table with a `type` integer ID referencing a `count_history_types` table would better match the way I implemented the `since_ids` table:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/4b6c8d8c1cc6fefdb566ec8506157133f47c569a/twitter_to_sqlite/utils.py#L331-L341\r\n\r\nIn which case the compound primary key would be `type`, `user`, `datetime`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011972", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 607011972, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMTk3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:49:02Z", "updated_at": "2020-04-01T03:50:01Z", "author_association": "MEMBER", "body": "I want the datetime value to look like `2020-04-01T03:34:58+00:00` (the format returned by the Twitter API which I am storing in other tables at the moment).\r\n```\r\n>>> datetime.utcnow().isoformat().split('.')[0] + '+00:00'\r\n'2020-04-01T03:49:52+00:00'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607019151", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40", "id": 607019151, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxOTE1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T04:11:10Z", "updated_at": "2020-04-01T04:11:10Z", "author_association": "MEMBER", "body": "In testing this collects a LOT of data. I'm going to skip tracking favourites_count and statuses_count and just track followers, friends and listed instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590669793, "label": "Feature: record history of follower counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/4#issuecomment-527682713", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4", "id": 527682713, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzY4MjcxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-03T23:48:57Z", "updated_at": "2019-09-03T23:48:57Z", "author_association": "MEMBER", "body": "One interesting challenge here is that the JSON format for tweets in the archive is subtly different from the JSON format currently returned by the API.\r\n\r\nIf we want to keep the tweets in the same database table (which feels like the right thing to me) we'll need to handle this.\r\n\r\nOne thing we can do is have a column for `from_archive` which is set to 1 for tweets that were recovered from the archive.\r\n\r\nWe can also ensure that tweets from the API always over-write the version that came from the archive (using `.upsert()`) while tweets from the archive use `.insert(..., ignore=True)` to avoid over-writing a better version that came from the API.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488835586, "label": "Command for importing data from a Twitter Export file"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/4#issuecomment-540879620", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4", "id": 540879620, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MDg3OTYyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T02:59:16Z", "updated_at": "2019-10-11T02:59:16Z", "author_association": "MEMBER", "body": "Also import ad preferences and all that other junk.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488835586, "label": "Command for importing data from a Twitter Export file"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606304837", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606304837, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNDgzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:27:50Z", "updated_at": "2020-03-30T23:29:31Z", "author_association": "MEMBER", "body": "One option would be something like this:\r\n```sql\r\nselect max(id) from tweets\r\nwhere user = ?\r\nand not exists (select id from tweets where retweeted_status = id)\r\nand not exists (select id from tweets where quoted_status = id)\r\nand not exists (select id from tweets where in_reply_to_status_id = id)\r\n```\r\nMight be a good idea to index those columns (after confirming that doing so would indeed speed up the query).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606305701", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606305701, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNTcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:30:27Z", "updated_at": "2020-03-30T23:30:27Z", "author_association": "MEMBER", "body": "A better alternative would be to maintain a separate table with the last seen since value for when we ran `user-timeline` for any specific user.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606309165", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606309165, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwOTE2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:41:31Z", "updated_at": "2020-03-30T23:41:31Z", "author_association": "MEMBER", "body": "I like the separate `user_timeline_since` table solution.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606824992", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606824992, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjgyNDk5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T19:24:23Z", "updated_at": "2020-03-31T19:24:23Z", "author_association": "MEMBER", "body": "The `--since` option is actually used by four commands:\r\n\r\n* `user-timeline`\r\n* `home-timeline`\r\n* `mentions-timeline`\r\n* `search`\r\n\r\nAll of them use the same `fetch_timeline()` utility function under the hood. I should move the logic that looks up the last `since_id` into that shared function.\r\n\r\nQuestion: should I have a table for each of those four methods or a single table that is used by them all? I'm leaning towards four separate tables.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606843224", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606843224, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg0MzIyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T19:59:11Z", "updated_at": "2020-03-31T20:06:32Z", "author_association": "MEMBER", "body": "Or... have a single `since_ids` table to track since values, and have its primary key be a string that looks something like this:\r\n\r\n`user:123145`\r\n`home:23441`\r\n`mentions:23425`\r\n`search:99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d`\r\n\r\nThat last example would use the hash generated here:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/cli.py#L792-L808", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606844521", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606844521, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg0NDUyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:01:39Z", "updated_at": "2020-03-31T20:01:39Z", "author_association": "MEMBER", "body": "I think `utils.fetch_timeline()` grows a new argument, `since_key`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850008", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606850008, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg1MDAwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:13:59Z", "updated_at": "2020-04-01T00:23:00Z", "author_association": "MEMBER", "body": "Table design for `since_ids` table:\r\n\r\ntype | key | since_id\r\n--- | --- | ---\r\n1 | 124324 | 2347239847293\r\n2 | 99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d | 2125947927344\r\n\r\nPrimary compound key on `(category, key)`\r\n\r\n`type` is also a foreign key to a `since_id_types` table with `id` and `name` columns (probably created using https://sqlite-utils.readthedocs.io/en/stable/python-api.html#working-with-lookup-tables )", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850453", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606850453, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg1MDQ1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:14:58Z", "updated_at": "2020-04-01T03:03:50Z", "author_association": "MEMBER", "body": "Actually I'll hard-code the population of `since_id_types` to get known ID constants.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606998669", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606998669, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjk5ODY2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T02:57:36Z", "updated_at": "2020-04-01T02:57:36Z", "author_association": "MEMBER", "body": "The tricky thing here is thinking about the interaction between the recorded since_id and a desire to run the initial import.\r\n\r\nThe first time you run `twitter-to-sqlite user-timeline db.db username` we want to fetch as many tweets from that user as possible - probably around 3,200 before the API limitations cut us off.\r\n\r\nWe need to record the maximum ID from those as the `since_id` - which we will see on the very first page we paginate through. That way next time we run the command with `--since` we will only fetch new tweets.\r\n\r\nBut what happens if our initial import is cancelled after only a few tweets? We risk never pulling in the rest of the tweets.\r\n\r\nNot sure if I need to solve this at all or if I should instead trust users to run the command a second time without `--since` if they think they didn't retrieve anything the first time.\r\n\r\nI had considered letting `--stop_after=` over-ride `--since` but that doesn't actually make sense - if you send a since_id to the Twitter API you'll never get back more tweets than exist after that ID, so the `--stop_after` would not make a meaningful difference.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607003655", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 607003655, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAwMzY1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:18:00Z", "updated_at": "2020-04-01T03:18:00Z", "author_association": "MEMBER", "body": "I've got this working for the `user-timeline` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607010634", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 607010634, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMDYzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:45:16Z", "updated_at": "2020-04-01T03:45:16Z", "author_association": "MEMBER", "body": "OK, fix is applied to everything now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/38#issuecomment-601939656", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/38", "id": 601939656, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTkzOTY1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T22:37:04Z", "updated_at": "2020-03-20T22:37:04Z", "author_association": "MEMBER", "body": "New formatting:\r\n```\r\n@Alaska_DHSS [####################################] 3205\r\n@BWhippleKS [####################################] 1\r\n@CAPublicHealth [####################################] 2\r\n@CDPHE [####################################] 67\r\n@CHFSKy [####################################] 3216\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585359363, "label": "Screen name display for user-timeline is uneven"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509578", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37", "id": 615509578, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUwOTU3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:36:00Z", "updated_at": "2020-04-17T23:36:00Z", "author_association": "MEMBER", "body": "```\r\n$ twitter-to-sqlite user-timeline doggo.db doggoenthuonetuh\r\nTraceback (most recent call last):\r\n...\r\n File \"/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py\", line 272, in transform_user\r\n user[\"created_at\"] = parser.parse(user[\"created_at\"])\r\nKeyError: 'created_at'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585353598, "label": "Handle \"User not found\" error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509803", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37", "id": 615509803, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUwOTgwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:36:40Z", "updated_at": "2020-04-17T23:36:40Z", "author_association": "MEMBER", "body": "I'm going to print a warning to stderr, skip and continue - because if you have 100 screen names and only one of them is invalid you should still execute for the other 99.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585353598, "label": "Handle \"User not found\" error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615510361", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37", "id": 615510361, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTUxMDM2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-17T23:38:27Z", "updated_at": "2020-04-17T23:38:27Z", "author_association": "MEMBER", "body": "That's a bit tricky since I'd have to rewrite the internals of a bunch of other commands. For the moment I'll exit the script with an error but at least it will be a decent error!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585353598, "label": "Handle \"User not found\" error"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875405", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601875405, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3NTQwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:30:09Z", "updated_at": "2020-03-20T19:32:33Z", "author_association": "MEMBER", "body": "I'm going to redesign the command a little. Right now it defaults to fetching the authenticated user's timeline and allows `--screen_name` to be used to fetch for one other user instead.\r\n\r\nI think the design of `users-lookup` is much better. I'm going to copy that:\r\n\r\n```\r\ntwitter-to-sqlite user-timeline users.db simonw cleopaws\r\ntwitter-to-sqlite user-timeline users.db 12497 3166449535 --ids\r\n```\r\nIt will also accept `--since` and `--sql` and `--alter` and `--since_id`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875524", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601875524, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3NTUyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:30:27Z", "updated_at": "2020-03-20T19:30:27Z", "author_association": "MEMBER", "body": "I can give it a snazzier progress bar to, as requested by #10.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601876051", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601876051, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3NjA1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:31:43Z", "updated_at": "2020-03-20T19:31:43Z", "author_association": "MEMBER", "body": "I'm going to keep `--screen_name` working for backwards compatibility. Using `hidden=True` on it should hide it from the help output.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601890795", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601890795, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg5MDc5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T20:13:08Z", "updated_at": "2020-03-20T20:13:08Z", "author_association": "MEMBER", "body": "Needs documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601892338", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35", "id": 601892338, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg5MjMzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T20:17:00Z", "updated_at": "2020-03-20T20:17:00Z", "author_association": "MEMBER", "body": "Released as 0.18.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585282212, "label": "twitter-to-sqlite user-timeline [screen_names] --sql / --attach"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601861908", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34", "id": 601861908, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg2MTkwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T18:56:44Z", "updated_at": "2020-03-20T18:56:44Z", "author_association": "MEMBER", "body": "Could this be a bug in `sqlite-utils`? This table has a primary key, so why is it running a query on `rowid = ?`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585266763, "label": "IndexError running user-timeline command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601871579", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34", "id": 601871579, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3MTU3OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:20:52Z", "updated_at": "2020-03-20T19:20:52Z", "author_association": "MEMBER", "body": "Cause was the redefining of how `upsert()` works in sqlite-utils 2.x.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 585266763, "label": "IndexError running user-timeline command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/pull/33#issuecomment-601871930", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/33", "id": 601871930, "node_id": "MDEyOklzc3VlQ29tbWVudDYwMTg3MTkzMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-20T19:21:41Z", "updated_at": "2020-03-20T19:21:41Z", "author_association": "MEMBER", "body": "Made obsolete by https://github.com/dogsheep/twitter-to-sqlite/commit/d3887d1535c3edd59d3ebf6aea4495bacd82e59b", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 561469252, "label": "Upgrade to sqlite-utils 2.2.1"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/31#issuecomment-552135263", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31", "id": 552135263, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEzNTI2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T20:38:35Z", "updated_at": "2019-11-09T20:38:35Z", "author_association": "MEMBER", "body": "Command still needs documentation and a bit more testing.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520508502, "label": "\"friends\" command (similar to \"followers\")"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552131798", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30", "id": 552131798, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEzMTc5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T19:54:45Z", "updated_at": "2019-11-09T19:54:45Z", "author_association": "MEMBER", "body": "Good catch - not sure how that bug crept in. Removing line 116 looks like the right fix to me.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 518739697, "label": "`followers` fails because `transform_user` is called twice"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552133468", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30", "id": 552133468, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T20:15:27Z", "updated_at": "2019-11-09T20:15:27Z", "author_association": "MEMBER", "body": "Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 518739697, "label": "`followers` fails because `transform_user` is called twice"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-541493242", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3", "id": 541493242, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTQ5MzI0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-14T03:35:36Z", "updated_at": "2019-10-14T03:35:36Z", "author_association": "MEMBER", "body": "https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488833975, "label": "Command for running a search and saving tweets for that search"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543273540", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3", "id": 543273540, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI3MzU0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T17:12:51Z", "updated_at": "2019-10-17T17:12:51Z", "author_association": "MEMBER", "body": "Just importing tweets here isn't enough - how are we supposed to know which tweets were imported by which search?\r\n\r\nSo I think the right thing to do here is to also create a `search_runs` table, which records each individual run of this tool (with a timestamp and the search terms used). Then have a `search_runs_tweets` m2m table which shows which Tweets were found by that search.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488833975, "label": "Command for running a search and saving tweets for that search"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543290744", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3", "id": 543290744, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI5MDc0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T17:57:14Z", "updated_at": "2019-10-17T17:57:14Z", "author_association": "MEMBER", "body": "I have a working command now. I'm going to ship it early because it could do with some other people trying it out.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488833975, "label": "Command for running a search and saving tweets for that search"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549096321", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3", "id": 549096321, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTA5NjMyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-03T01:27:55Z", "updated_at": "2019-11-03T01:28:17Z", "author_association": "MEMBER", "body": "It would be neat if this could support `--since`, with that argument automatically finding the maximum tweet ID from a previous search that used the same exact arguments (using the `search_runs` table).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488833975, "label": "Command for running a search and saving tweets for that search"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549226399", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3", "id": 549226399, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTIyNjM5OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-04T05:11:57Z", "updated_at": "2019-11-04T05:11:57Z", "author_association": "MEMBER", "body": "I'm going to add a `hash` column to `search_runs` to support that. It's going to be the sha1 hash of the key-ordered JSON of the search arguments used by that run. Then `--since` can look for an identical hash and use it to identify the highest last fetched tweet to use in `since_id`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488833975, "label": "Command for running a search and saving tweets for that search"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549228535", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3", "id": 549228535, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTIyODUzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-04T05:31:55Z", "updated_at": "2019-11-04T05:31:55Z", "author_association": "MEMBER", "body": "Documented here: https://github.com/dogsheep/twitter-to-sqlite/blob/801c0c2daf17d8abce9dcb5d8d610410e7e25dbe/README.md#running-searches", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488833975, "label": "Command for running a search and saving tweets for that search"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129686", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29", "id": 552129686, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEyOTY4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T19:27:39Z", "updated_at": "2019-11-09T19:27:39Z", "author_association": "MEMBER", "body": "I think this is fixed by the latest version of `sqlite-utils` - https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-12-1 - I'll bump the dependency.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 518725064, "label": "`import` command fails on empty files"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129921", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29", "id": 552129921, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEyOTkyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T19:30:42Z", "updated_at": "2019-11-09T19:30:42Z", "author_association": "MEMBER", "body": "Confirmed, that seems to fix it:\r\n```\r\n(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js \r\nTraceback (most recent call last):\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite\", line 11, in \r\n load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')()\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 764, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 717, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 1137, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 956, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 555, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py\", line 633, in import_\r\n archive.import_from_file(db, path.name, open(path, \"rb\").read())\r\n File \"/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/archive.py\", line 224, in import_from_file\r\n db[table_name].upsert_all(rows, hash_id=\"pk\")\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1094, in upsert_all\r\n extracts=extracts,\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py\", line 961, in insert_all\r\n first_record = next(records)\r\nStopIteration\r\n(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ pip install -U sqlite-utils\r\nCollecting sqlite-utils\r\n Using cached https://files.pythonhosted.org/packages/ee/a2/1b135010c7ac8e2d7545f659e9e6c6ede0f406f20b52e08d5817e1e31a9a/sqlite_utils-1.12.1-py3-none-any.whl\r\nRequirement already satisfied, skipping upgrade: click in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (7.0)\r\nRequirement already satisfied, skipping upgrade: tabulate in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (0.8.5)\r\nRequirement already satisfied, skipping upgrade: click-default-group in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (1.2.2)\r\nInstalling collected packages: sqlite-utils\r\n Found existing installation: sqlite-utils 1.11\r\n Uninstalling sqlite-utils-1.11:\r\n Successfully uninstalled sqlite-utils-1.11\r\nSuccessfully installed sqlite-utils-1.12.1\r\n(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js \r\n(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ \r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 518725064, "label": "`import` command fails on empty files"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552133449", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29", "id": 552133449, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T20:15:15Z", "updated_at": "2019-11-09T20:15:15Z", "author_association": "MEMBER", "body": "Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 518725064, "label": "`import` command fails on empty files"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/28#issuecomment-552133488", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/28", "id": 552133488, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-09T20:15:42Z", "updated_at": "2019-11-09T20:15:42Z", "author_association": "MEMBER", "body": "Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 515658861, "label": "Add indexes to followers table"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095217", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27", "id": 549095217, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTA5NTIxNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-03T01:06:25Z", "updated_at": "2019-11-03T01:06:25Z", "author_association": "MEMBER", "body": "Wow, that `retweets_of_me` endpoint is almost completely useless:\r\n```\r\n$ twitter-to-sqlite fetch https://api.twitter.com/1.1/statuses/retweets_of_me.json\r\n```\r\nIt returns my own tweets that have been retweeted, but with no indication at all of who retweeted them.\r\n\r\nIt looks like this needs to be combined with this API - https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-retweets-id - to fetch the details of up to 100 recent users who actually DID retweet an individual status. But that has a one-every-12-seconds rate limit on it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 514459062, "label": "retweets-of-me command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095317", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27", "id": 549095317, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTA5NTMxNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-03T01:08:10Z", "updated_at": "2019-11-03T01:08:10Z", "author_association": "MEMBER", "body": "Hmm... one thing that could be useful is that `retweets_of_me` can support a `--since` parameter - so if run frequently it should hopefully let us know which tweets we would need to run `statuses/retweets/:id.json` against.\r\n\r\nI'm not sure if the `--since` parameter would show me a tweet that was previously retweeted but has now been retweeted again. I'll have a bit of a test and see.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 514459062, "label": "retweets-of-me command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095463", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27", "id": 549095463, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTA5NTQ2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-03T01:10:52Z", "updated_at": "2019-11-03T01:10:52Z", "author_association": "MEMBER", "body": "I imagine it won't, since the data I would be recording and then passing to `since_id` would be the highest ID of my own tweets that have been retweeted at least once. So it won't be able to spot if I should check for fresh retweets of a given tweet.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 514459062, "label": "retweets-of-me command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095641", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27", "id": 549095641, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTA5NTY0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-03T01:12:58Z", "updated_at": "2019-11-03T01:12:58Z", "author_association": "MEMBER", "body": "It looks like Twitter really want you to subscribe to a premium API for this kind of thing and consume retweets via webhooks: https://developer.twitter.com/en/docs/accounts-and-users/subscribe-account-activity/api-reference\r\n\r\nI'm going to give up on this for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 514459062, "label": "retweets-of-me command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/26#issuecomment-547713287", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/26", "id": 547713287, "node_id": "MDEyOklzc3VlQ29tbWVudDU0NzcxMzI4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-30T02:36:13Z", "updated_at": "2019-10-30T02:36:13Z", "author_association": "MEMBER", "body": "Shipped this in 0.13: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.13\r\n\r\nSee also this Twitter thread: https://twitter.com/simonw/status/1189369677509623809", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 513074501, "label": "Command for importing mentions timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543265058", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25", "id": 543265058, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI2NTA1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T16:51:12Z", "updated_at": "2019-10-17T16:51:12Z", "author_association": "MEMBER", "body": "This migration function only runs if there is a table called `tweets` and the migration has not run before.\r\n\r\nI think this can happen if the database has just been freshly created (by a command that fetches the user's user timeline for example) and is then run a SECOND time.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 508578780, "label": "Ensure migrations don't accidentally create foreign key twice"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543266947", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25", "id": 543266947, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI2Njk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T16:56:06Z", "updated_at": "2019-10-17T16:56:06Z", "author_association": "MEMBER", "body": "I wrote a test that proves that this is a problem. Should be an easy fix though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 508578780, "label": "Ensure migrations don't accidentally create foreign key twice"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543217890", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23", "id": 543217890, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzIxNzg5MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T15:03:10Z", "updated_at": "2019-10-17T15:03:10Z", "author_association": "MEMBER", "body": "Thinking about this further: the concept of migrations may end up being in direct conflict with the `sqlite-utils` concept of creating tables on demand the first time they are used - and of creating table schemas automatically to fit the shape of the JSON that is being inserted into them.\r\n\r\nI'm going to forge ahead anyway and build this because I think it will be an interesting exploration, but it's very likely this turns out to be a bad idea in the long run!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 508190730, "label": "Extremely simple migration system"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543222239", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23", "id": 543222239, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzIyMjIzOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T15:12:33Z", "updated_at": "2019-10-17T15:12:33Z", "author_association": "MEMBER", "body": "Migrations will run only if you open a database that previously existed (as opposed to opening a brand new empty database).\r\n\r\nThis means that the first time you run a command against a fresh database, migrations will not run and the `migrations` table will not be created. The _second_ time you run any command against that database the migrations will execute and populate the `migrations` table.\r\n\r\nThis also means that each migration needs to be able to sanity check the database to see if it should run or not. If it should NOT run, it will do nothing but still be marked as having executed by adding to the `migrations` table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 508190730, "label": "Extremely simple migration system"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/21#issuecomment-542333836", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/21", "id": 542333836, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MjMzMzgzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-15T18:00:48Z", "updated_at": "2019-10-15T18:00:48Z", "author_association": "MEMBER", "body": "I'll use `html.unescape()` for this: https://docs.python.org/3/library/html.html#html.unescape", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506432572, "label": "Fix & escapes in tweet text"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-541388038", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20", "id": 541388038, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTM4ODAzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-13T05:31:58Z", "updated_at": "2019-10-13T05:31:58Z", "author_association": "MEMBER", "body": "For favourites a `--stop_after=200` option is probably good enough.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506268945, "label": "--since support for various commands for refresh-by-cron"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-542854749", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20", "id": 542854749, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg1NDc0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T19:26:01Z", "updated_at": "2019-10-16T19:26:01Z", "author_association": "MEMBER", "body": "I'm not going to do this for \"accounts that have followed me\" and \"new accounts that I have followed\" - instead I will recommend running the `friend_ids` and `followers_ids` commands on a daily basis since that data doesn't really change much by the hour.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506268945, "label": "--since support for various commands for refresh-by-cron"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-544335363", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20", "id": 544335363, "node_id": "MDEyOklzc3VlQ29tbWVudDU0NDMzNTM2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-21T03:32:04Z", "updated_at": "2019-10-21T03:32:04Z", "author_association": "MEMBER", "body": "In case anyone is interested, here's an extract from the crontab I'm running these under at the moment:\r\n```\r\n1,11,21,31,41,51 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite user-timeline /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --since\r\n2,7,12,17,22,27,32,37,42,47,52,57 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite home-timeline /home/ubuntu/timeline.db -a /home/ubuntu/auth.json --since\r\n6,16,26,36,46,56 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite favorites /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --stop_after=50\r\n```", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506268945, "label": "--since support for various commands for refresh-by-cron"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527954898", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2", "id": 527954898, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzk1NDg5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-04T15:31:46Z", "updated_at": "2019-09-04T15:31:46Z", "author_association": "MEMBER", "body": "I'm going to call this `twitter-to-sqlite user-timeline` to reflect the language used to describe the API endpoint: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488833698, "label": "\"twitter-to-sqlite user-timeline\" command for pulling tweets by a specific user"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527955302", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2", "id": 527955302, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzk1NTMwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-04T15:32:39Z", "updated_at": "2019-09-04T15:32:39Z", "author_association": "MEMBER", "body": "Rate limit is 900 / 15 minutes which is 1 call per second.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488833698, "label": "\"twitter-to-sqlite user-timeline\" command for pulling tweets by a specific user"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527990908", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2", "id": 527990908, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzk5MDkwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-04T16:57:24Z", "updated_at": "2019-09-04T16:57:24Z", "author_association": "MEMBER", "body": "I just tried this using `max_id=` pagination as described in [Working with timelines](https://developer.twitter.com/en/docs/tweets/timelines/guides/working-with-timelines) and I got back all 17,759 of my tweets.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488833698, "label": "\"twitter-to-sqlite user-timeline\" command for pulling tweets by a specific user"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-541248629", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19", "id": 541248629, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTI0ODYyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T22:48:56Z", "updated_at": "2019-10-11T22:48:56Z", "author_association": "MEMBER", "body": "`since_id` documented here: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-home_timeline\r\n\r\n> Returns results with an ID greater than (that is, more recent than) the specified ID. There are limits to the number of Tweets which can be accessed through the API. If the limit of Tweets has occured since the since_id, the since_id will be forced to the oldest ID available.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506087267, "label": "since_id support for home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542832952", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19", "id": 542832952, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MjgzMjk1Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T18:30:11Z", "updated_at": "2019-10-16T18:30:11Z", "author_association": "MEMBER", "body": "The `--since` option will derive the `since_id` from the max ID in the `timeline_tweets` table:\r\n\r\n $ twitter-to-sqlite home-timeline --since\r\n\r\nThe `--since_id=xxx` option lets you specify that ID directly.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506087267, "label": "since_id support for home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542849963", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19", "id": 542849963, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg0OTk2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T19:13:06Z", "updated_at": "2019-10-16T19:13:06Z", "author_association": "MEMBER", "body": "Updated documentation: https://github.com/dogsheep/twitter-to-sqlite/blob/fced2a9b67d2cbdf9817f1eb75f7c28e413c963b/README.md#retrieving-tweets-from-your-home-timeline", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506087267, "label": "since_id support for home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541118773", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18", "id": 541118773, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTExODc3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T15:48:31Z", "updated_at": "2019-10-11T15:48:31Z", "author_association": "MEMBER", "body": "https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-home_timeline", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 505928530, "label": "Command to import home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541118934", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18", "id": 541118934, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTExODkzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T15:48:54Z", "updated_at": "2019-10-11T15:48:54Z", "author_association": "MEMBER", "body": "Rate limit is tight: 15 requests every 15 mins!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 505928530, "label": "Command to import home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541119834", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18", "id": 541119834, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTExOTgzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T15:51:22Z", "updated_at": "2019-10-11T16:51:33Z", "author_association": "MEMBER", "body": "In order to support multiple user timelines being saved in the same database, I'm going to import the tweets into the `tweets` table AND add a new `timeline_tweets` table recording that a specific tweet showed up in a specific user's timeline.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 505928530, "label": "Command to import home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541141169", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18", "id": 541141169, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTE0MTE2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T16:51:29Z", "updated_at": "2019-10-11T16:51:29Z", "author_association": "MEMBER", "body": "Documented here: https://github.com/dogsheep/twitter-to-sqlite/blob/master/README.md#retrieving-tweets-from-your-home-timeline", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 505928530, "label": "Command to import home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112108", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17", "id": 541112108, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTExMjEwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T15:30:15Z", "updated_at": "2019-10-11T15:30:15Z", "author_association": "MEMBER", "body": "It should delete the tables entirely. That way it will work even if the table schema has changed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 505674949, "label": "import command should empty all archive-* tables first"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112588", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17", "id": 541112588, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTExMjU4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T15:31:30Z", "updated_at": "2019-10-11T15:31:30Z", "author_association": "MEMBER", "body": "No need for an option:\r\n\r\n> This command will delete and recreate all of your `archive-*` tables every time you run it. If this is not what you want, run the command against a fresh SQLite database rather than running it again one that already exists.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 505674949, "label": "import command should empty all archive-* tables first"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/13#issuecomment-538804815", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/13", "id": 538804815, "node_id": "MDEyOklzc3VlQ29tbWVudDUzODgwNDgxNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-07T00:33:49Z", "updated_at": "2019-10-07T00:33:49Z", "author_association": "MEMBER", "body": "Documentation: https://github.com/dogsheep/twitter-to-sqlite#retrieve-tweets-in-bulk", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503085013, "label": "statuses-lookup command"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855081", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12", "id": 542855081, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg1NTA4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T19:26:56Z", "updated_at": "2019-10-16T19:26:56Z", "author_association": "MEMBER", "body": "This may be the first case where I want to be able to repair existing databases rather than discarding their contents.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503053800, "label": "Extract \"source\" into a separate lookup table"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855427", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12", "id": 542855427, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg1NTQyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T19:27:55Z", "updated_at": "2019-10-16T19:27:55Z", "author_association": "MEMBER", "body": "I can do that by keeping `source` as a `TEXT` column but turning it into a non-enforced foreign key against a new `sources` table. Then I can run code that scans that column for any values beginning with a `<` and converts them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503053800, "label": "Extract \"source\" into a separate lookup table"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542858025", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12", "id": 542858025, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg1ODAyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T19:35:31Z", "updated_at": "2019-10-16T19:36:09Z", "author_association": "MEMBER", "body": "Maybe this means I need an `upgrade` command to apply these kinds of migrations? Total feature creep!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503053800, "label": "Extract \"source\" into a separate lookup table"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/11#issuecomment-538711918", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/11", "id": 538711918, "node_id": "MDEyOklzc3VlQ29tbWVudDUzODcxMTkxOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-06T04:54:17Z", "updated_at": "2019-10-06T04:54:17Z", "author_association": "MEMBER", "body": "Shipped in 0.6. Here's the documentation: https://github.com/dogsheep/twitter-to-sqlite#capturing-tweets-in-real-time-with-track-and-follow", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503045221, "label": "Commands for recording real-time tweets from the streaming API"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-541748580", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10", "id": 541748580, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTc0ODU4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-14T15:30:44Z", "updated_at": "2019-10-14T15:30:44Z", "author_association": "MEMBER", "body": "Had several recommendations for https://github.com/tqdm/tqdm which is what goodreads-to-sqlite uses.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 492297930, "label": "Rethink progress bars for various commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543269396", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10", "id": 543269396, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI2OTM5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T17:02:07Z", "updated_at": "2019-10-17T17:02:07Z", "author_association": "MEMBER", "body": "A neat trick that Click does is detecting if an interactive terminal is attached and NOT showing a progress bar if there isn't one. Need to figure out how to do that with tqdm.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 492297930, "label": "Rethink progress bars for various commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543270714", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10", "id": 543270714, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI3MDcxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T17:05:16Z", "updated_at": "2019-10-17T17:05:16Z", "author_association": "MEMBER", "body": "https://github.com/pallets/click/blob/716a5be90f56ce6cd506bb53d5739d09374b1636/click/_termui_impl.py#L93 is how Click does this:\r\n```\r\n self.is_hidden = not isatty(self.file)\r\n```\r\nWhere `isatty` is a Click utility function: `from ._compat import isatty`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 492297930, "label": "Rethink progress bars for various commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543271000", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10", "id": 543271000, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI3MTAwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T17:05:59Z", "updated_at": "2019-10-17T17:05:59Z", "author_association": "MEMBER", "body": "Looks like tqdm already does a TTY check here: https://github.com/tqdm/tqdm/blob/89b73bdc30c099c5b53725806e7edf3a121c9b3a/tqdm/std.py#L889-L890", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 492297930, "label": "Rethink progress bars for various commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-607010791", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10", "id": 607010791, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMDc5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:45:48Z", "updated_at": "2020-04-01T03:45:48Z", "author_association": "MEMBER", "body": "I'm happy with the recent work I did on this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 492297930, "label": "Rethink progress bars for various commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-669241341", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8", "id": 669241341, "node_id": "MDEyOklzc3VlQ29tbWVudDY2OTI0MTM0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-05T14:55:14Z", "updated_at": "2020-08-05T14:55:14Z", "author_association": "MEMBER", "body": "Looks like there's a column that's not consistently there, so the table got created without it.\r\n\r\nEasiest fix is to add `alter=True` to this line:\r\n\r\nhttps://github.com/dogsheep/swarm-to-sqlite/blob/f4a82633da927cde672c9d9af92930bfca2e3ddf/swarm_to_sqlite/utils.py#L94\r\n\r\nThat will cause `sqlite-utils` to notice if there's a missing column and add it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648245071, "label": "Error thrown: table photos has no column named hasSticker"}, "performed_via_github_app": null}