{"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135562", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2", "id": 886135562, "node_id": "IC_kwDODtX3eM400VsK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T02:01:11Z", "updated_at": "2021-07-25T02:01:11Z", "author_association": "MEMBER", "body": "That page doesn't have an API but does look easy to scrape.\r\n\r\nThe other option here is the HN Search API powered by Algolia, documented at https://hn.algolia.com/api", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952179830, "label": "Command for fetching Hacker News threads from the search API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135922", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2", "id": 886135922, "node_id": "IC_kwDODtX3eM400Vxy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T02:06:20Z", "updated_at": "2021-07-25T02:06:20Z", "author_association": "MEMBER", "body": "https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url looks like it does what I want.\r\n\r\nhttps://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000 - returns 1000 at once.\r\n\r\nOtherwise you have to paginate using `&page=2` etc - up to `nbPages` pages.\r\n\r\nhttps://www.algolia.com/doc/api-reference/api-parameters/hitsPerPage/ says 1000 is the maximum.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952179830, "label": "Command for fetching Hacker News threads from the search API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886136224", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2", "id": 886136224, "node_id": "IC_kwDODtX3eM400V2g", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T02:08:29Z", "updated_at": "2021-07-25T02:08:29Z", "author_association": "MEMBER", "body": "Prototype:\r\n\r\n    curl \"https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000\" | \\\r\n      jq .hits | sqlite-utils insert hn.db items - --pk objectID --alter", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952179830, "label": "Command for fetching Hacker News threads from the search API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886140431", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2", "id": 886140431, "node_id": "IC_kwDODtX3eM400W4P", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T03:12:57Z", "updated_at": "2021-07-25T03:12:57Z", "author_association": "MEMBER", "body": "I'm going to build a general-purpose `hacker-new-to-sqlite search ...` command, where one of the options is to search within the URL.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952179830, "label": "Command for fetching Hacker News threads from the search API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886142671", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3", "id": 886142671, "node_id": "IC_kwDODtX3eM400XbP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T03:51:05Z", "updated_at": "2021-07-25T03:51:05Z", "author_association": "MEMBER", "body": "Prototype:\r\n\r\n    curl 'https://hn.algolia.com/api/v1/items/27941108' \\\r\n      | jq '[recurse(.children[]) | del(.children)]' \\\r\n      | sqlite-utils insert hn.db items - --pk id\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952189173, "label": "Use HN algolia endpoint to retrieve trees"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886237834", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3", "id": 886237834, "node_id": "IC_kwDODtX3eM400uqK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T18:05:32Z", "updated_at": "2021-07-25T18:05:32Z", "author_association": "MEMBER", "body": "If you hit the endpoint for a comment that's part of a thread you get that comment and its recursive children: https://hn.algolia.com/api/v1/items/27941552\r\n\r\nYou can tell that it's not the top-level because the `parent_id` isn't `null`. You can use `story_id` to figure out what the top-level item is.\r\n\r\n```json\r\n{\r\n  \"id\": 27941552,\r\n  \"created_at\": \"2021-07-24T15:08:39.000Z\",\r\n  \"created_at_i\": 1627139319,\r\n  \"type\": \"comment\",\r\n  \"author\": \"nine_k\",\r\n  \"title\": null,\r\n  \"url\": null,\r\n  \"text\": \"<p>I wish ...\",\r\n  \"points\": null,\r\n  \"parent_id\": 27941108,\r\n  \"story_id\": 27941108\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952189173, "label": "Use HN algolia endpoint to retrieve trees"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886241674", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3", "id": 886241674, "node_id": "IC_kwDODtX3eM400vmK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T18:41:17Z", "updated_at": "2021-07-25T18:41:17Z", "author_association": "MEMBER", "body": "Got a TIL out of this: https://til.simonwillison.net/jq/extracting-objects-recursively", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952189173, "label": "Use HN algolia endpoint to retrieve trees"}, "performed_via_github_app": null}