{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/50#issuecomment-690860653", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50", "id": 690860653, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MDg2MDY1Mw==", "user": {"value": 370930, "label": "mikepqr"}, "created_at": "2020-09-11T04:04:08Z", "updated_at": "2020-09-11T04:04:08Z", "author_association": "CONTRIBUTOR", "body": "There's probably a nicer way of doing (hence this is a comment rather than a PR), but this appears to fix it:\r\n```diff\r\n--- a/twitter_to_sqlite/utils.py\r\n+++ b/twitter_to_sqlite/utils.py\r\n@@ -181,6 +181,7 @@ def fetch_timeline(\r\n     args[\"tweet_mode\"] = \"extended\"\r\n     min_seen_id = None\r\n     num_rate_limit_errors = 0\r\n+    seen_count = 0\r\n     while True:\r\n         if min_seen_id is not None:\r\n             args[\"max_id\"] = min_seen_id - 1\r\n@@ -208,6 +209,7 @@ def fetch_timeline(\r\n             yield tweet\r\n         min_seen_id = min(t[\"id\"] for t in tweets)\r\n         max_seen_id = max(t[\"id\"] for t in tweets)\r\n+        seen_count += len(tweets)\r\n         if last_since_id is not None:\r\n             max_seen_id = max((last_since_id, max_seen_id))\r\n             last_since_id = max_seen_id\r\n@@ -217,7 +219,9 @@ def fetch_timeline(\r\n                 replace=True,\r\n             )\r\n         if stop_after is not None:\r\n-            break\r\n+            if seen_count >= stop_after:\r\n+                break\r\n+            args[\"count\"] = min(args[\"count\"], stop_after - seen_count)\r\n         time.sleep(sleep)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 698791218, "label": "favorites --stop_after=N stops after min(N, 200)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/962#issuecomment-691250299", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/962", "id": 691250299, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MTI1MDI5OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-11T18:33:50Z", "updated_at": "2020-09-11T18:33:50Z", "author_association": "OWNER", "body": "Since this is purely a debugging option I'm going to allow myself not to write a unit test for it!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 699622046, "label": "datasette --pdb option for debugging errors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/947#issuecomment-691318133", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/947", "id": 691318133, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MTMxODEzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-11T21:23:40Z", "updated_at": "2020-09-11T21:23:40Z", "author_association": "OWNER", "body": "I'm going to use exit code 1 for any errors, be they 500 or 404.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 684111953, "label": "datasette --get exit code should reflect HTTP errors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-691323302", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 691323302, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MTMyMzMwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-11T21:38:27Z", "updated_at": "2020-09-11T21:40:04Z", "author_association": "OWNER", "body": "Another idea: the default output could be the list of dicts:\r\n```json\r\n[\r\n  {\r\n    \"pk1\": \"a\",\r\n    \"pk2\": \"a\",\r\n    \"pk3\": \"a\",\r\n    \"content\": \"a-a-a\"\r\n  },\r\n  ...\r\n]\r\n```\r\nBUT... I could include pagination information in the HTTP headers - as seen in the WordPress REST API or the GitHub API:\r\n\r\n```\r\n~ % curl -s -i 'https://api.github.com/repos/simonw/datasette/commits' | head -n 40\r\nHTTP/1.1 200 OK\r\nserver: GitHub.com\r\ndate: Fri, 11 Sep 2020 21:37:46 GMT\r\ncontent-type: application/json; charset=utf-8\r\nstatus: 200 OK\r\ncache-control: public, max-age=60, s-maxage=60\r\nvary: Accept, Accept-Encoding, Accept, X-Requested-With\r\netag: W/\"71c99379743513394e880c6306b66bf9\"\r\nlast-modified: Fri, 11 Sep 2020 21:32:54 GMT\r\nx-github-media-type: github.v3; format=json\r\nlink: <https://api.github.com/repositories/107914493/commits?page=2>; rel=\"next\", <https://api.github.com/repositories/107914493/commits?page=44>; rel=\"last\"\r\naccess-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset\r\naccess-control-allow-origin: *\r\nstrict-transport-security: max-age=31536000; includeSubdomains; preload\r\nx-frame-options: deny\r\nx-content-type-options: nosniff\r\nx-xss-protection: 1; mode=block\r\nreferrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin\r\ncontent-security-policy: default-src 'none'\r\nX-Ratelimit-Limit: 60\r\nX-Ratelimit-Remaining: 55\r\nX-Ratelimit-Reset: 1599863850\r\nX-Ratelimit-Used: 5\r\nAccept-Ranges: bytes\r\nContent-Length: 118240\r\nX-GitHub-Request-Id: EC76:0EAD:313F40:5291A4:5F5BEE37\r\n\r\n[\r\n  {\r\n    \"sha\": \"d02f6151dae073135a22d0123e8abdc6cbef7c50\",\r\n    \"node_id\": \"MDY6Q29tbWl0MTA3OTE0NDkzOmQwMmY2MTUxZGFlMDczMTM1YTIyZDAxMjNlOGFiZGM2Y2JlZjdjNTA=\",\r\n    \"commit\": {\r\n```\r\nAlternative shapes would provide the pagination information (and other extensions) in the JSON, e.g.:\r\n\r\n`/squirrels/squirrels.json?_shape=paginated`\r\n```json\r\n{\r\n  \"rows\": [\r\n  {\r\n    \"pk1\": \"a\",\r\n    \"pk2\": \"a\",\r\n    \"pk3\": \"a\",\r\n    \"content\": \"a-a-a\"\r\n  }\r\n  ],\r\n  \"pagination\": {\r\n    \"next\": \"234\",\r\n    \"count\": 442\r\n  }\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}