{"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902356871", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20", "id": 902356871, "node_id": "IC_kwDOC8tyDs41yN-H", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T01:12:48Z", "updated_at": "2021-08-20T01:12:48Z", "author_association": "MEMBER", "body": "Also on `workout_points.workout_id` to speed up queries to show all points in a specific workout.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 975166271, "label": "Add index on workout_points.date"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902355471", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20", "id": 902355471, "node_id": "IC_kwDOC8tyDs41yNoP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T01:09:07Z", "updated_at": "2021-08-20T01:09:07Z", "author_association": "MEMBER", "body": "Workaround:\r\n\r\n sqlite-utils create-index healthkit.db workout_points -- -date\r\n\r\nSee https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 975166271, "label": "Add index on workout_points.date"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502543165", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13", "id": 1502543165, "node_id": "IC_kwDODD6af85Zjv09", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-11T01:10:36Z", "updated_at": "2023-04-11T01:11:47Z", "author_association": "MEMBER", "body": "I just had that error myself on macOS while running the tests:\r\n```\r\nERROR tests/test_save_checkin.py::test_tables - sqlite3.OperationalError: table sqlite_master may not be modified\r\nERROR tests/test_save_checkin.py::test_venue - sqlite3.OperationalError: table sqlite_master may not be modified\r\nERROR tests/test_save_checkin.py::test_event - sqlite3.OperationalError: table sqlite_master may not be modified\r\nERROR tests/test_save_checkin.py::test_sticker - sqlite3.OperationalError: table sqlite_master may not be modified\r\nERROR tests/test_save_checkin.py::test_likes - sqlite3.OperationalError: table sqlite_master may not be modified\r\nERROR tests/test_save_checkin.py::test_with_ - sqlite3.OperationalError: table sqlite_master may not be modified\r\nERROR tests/test_save_checkin.py::test_users - sqlite3.OperationalError: table sqlite_master may not be modified\r\nERROR tests/test_save_checkin.py::test_photos - sqlite3.OperationalError: table sqlite_master may not be modified\r\nERROR tests/test_save_checkin.py::test_posts - sqlite3.OperationalError: table sqlite_master may not be modified\r\nERROR tests/test_save_checkin.py::test_view - sqlite3.OperationalError: table sqlite_master may not be modified\r\n```\r\n`pytest --pdb` shows it happening in the bit that adds foreign keys:\r\n```\r\n> /Users/simon/.local/share/virtualenvs/swarm-to-sqlite-daPW7yIJ/lib/python3.9/site-packages/sqlite_utils/db.py(1096)add_foreign_keys()\r\n-> cursor.execute(\r\n(Pdb) list\r\n1096 >>\t cursor.execute(\r\n1097 \t \"UPDATE sqlite_master SET sql = ? WHERE name = ?\",\r\n1098 \t (new_sql, table_name),\r\n1099 \t )\r\n1100 \t cursor.execute(\"PRAGMA schema_version = %d\" % (schema_version + 1))\r\n1101 ->\t cursor.execute(\"PRAGMA writable_schema = 0\")\r\n1102 \t # Have to VACUUM outside the transaction to ensure .foreign_keys property\r\n1103 \t # can see the newly created foreign key.\r\n1104 \t self.vacuum()\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373210675, "label": "fails before generating views. ERR: table sqlite_master may not be modified"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502546045", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13", "id": 1502546045, "node_id": "IC_kwDODD6af85Zjwh9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-11T01:14:50Z", "updated_at": "2023-04-11T01:14:50Z", "author_association": "MEMBER", "body": "Related:\r\n- https://github.com/simonw/sqlite-utils/issues/235", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373210675, "label": "fails before generating views. ERR: table sqlite_master may not be modified"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502629219", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13", "id": 1502629219, "node_id": "IC_kwDODD6af85ZkE1j", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-11T03:15:26Z", "updated_at": "2023-04-11T03:15:26Z", "author_association": "MEMBER", "body": "OK, I figured this out. Unfortunately it's an error that occurs on Python versions that have defensive mode turned on, and it doesn't look like there's a way to turn that mode off. See notes above.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373210675, "label": "fails before generating views. ERR: table sqlite_master may not be modified"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502629404", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13", "id": 1502629404, "node_id": "IC_kwDODD6af85ZkE4c", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-11T03:15:47Z", "updated_at": "2023-04-11T03:46:17Z", "author_association": "MEMBER", "body": "I think `swarm-to-sqlite` needs to avoid this error, maybe by setting up foreign keys in another way - or even by skipping foreign keys entirely on databases that don't support this kind of operation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373210675, "label": "fails before generating views. ERR: table sqlite_master may not be modified"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902327457", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902327457, "node_id": "IC_kwDODEm0Qs41yGyh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:53:25Z", "updated_at": "2021-08-19T23:53:25Z", "author_association": "MEMBER", "body": "I'm getting this too. Looking into it now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328369", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902328369, "node_id": "IC_kwDODEm0Qs41yHAx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:56:26Z", "updated_at": "2021-08-19T23:56:26Z", "author_association": "MEMBER", "body": "https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-user_timeline says the API has been replaced by the new v2 one, but it should still work - and the `since_id` parameter is still documented on that page.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328760", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902328760, "node_id": "IC_kwDODEm0Qs41yHG4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:57:41Z", "updated_at": "2021-08-19T23:57:41Z", "author_association": "MEMBER", "body": "Weird, added debug code and got this: `{'screen_name': 'simonw', 'count': 200, 'since_id': 'False', 'tweet_mode': 'extended'}` - so maybe it's a `twitter-to-sqlite` bug where somehow the string `False` is being passed somewhere.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329455", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902329455, "node_id": "IC_kwDODEm0Qs41yHRv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-19T23:59:56Z", "updated_at": "2021-08-19T23:59:56Z", "author_association": "MEMBER", "body": "This looks like the bug to me:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/197e69cec40052c423a5ed071feb5f7cccea41b9/twitter_to_sqlite/cli.py#L239-L241\r\n\r\n`type=str, default=False`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329884", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57", "id": 902329884, "node_id": "IC_kwDODEm0Qs41yHYc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T00:01:05Z", "updated_at": "2021-08-20T00:01:05Z", "author_association": "MEMBER", "body": "Maybe Click changed something which meant that this broke things when it didn't used to?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 907645813, "label": "Error: Use either --since or --since_id, not both"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/pull/49#issuecomment-902330301", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49", "id": 902330301, "node_id": "IC_kwDODEm0Qs41yHe9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-20T00:01:56Z", "updated_at": "2021-08-20T00:01:56Z", "author_association": "MEMBER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681575714, "label": "Document the use of --stop_after with favorites, refs #20"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/pull/59#issuecomment-924209583", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/59", "id": 924209583, "node_id": "IC_kwDODEm0Qs43FlGv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-09-21T17:37:34Z", "updated_at": "2021-09-21T17:37:34Z", "author_association": "MEMBER", "body": "Thanks for this!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 984942782, "label": "Fix for since_id bug, closes #58"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/69#issuecomment-985928838", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69", "id": 985928838, "node_id": "IC_kwDODFdgUs46xBSG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-04T00:34:52Z", "updated_at": "2021-12-04T00:34:52Z", "author_association": "MEMBER", "body": "First attempt at this:\r\n```sql\r\nselect 'issues' as \"table\", id, node_id, title, user, created_at, body, repo\r\nfrom issues\r\nunion all\r\nselect 'issue_comments' as \"table\", issue_comments.id, issue_comments.node_id, '' as title, issue_comments.user, issue_comments.created_at, issue_comments.body, issues.repo\r\nfrom issue_comments join issues on issues.id = issue_comments.issue\r\norder by created_at desc\r\n```\r\nhttps://github-to-sqlite.dogsheep.net/github?sql=select+%27issues%27+as+%22table%22%2C+id%2C+node_id%2C+title%2C+user%2C+created_at%2C+body%2C+repo%0D%0Afrom+issues%0D%0Aunion+all%0D%0Aselect+%27issue_comments%27+as+%22table%22%2C+issue_comments.id%2C+issue_comments.node_id%2C+%27%27+as+title%2C+issue_comments.user%2C+issue_comments.created_at%2C+issue_comments.body%2C+issues.repo%0D%0Afrom+issue_comments+join+issues+on+issues.id+%3D+issue_comments.issue%0D%0Aorder+by+created_at+desc", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1071071397, "label": "View that combines issues and issue comments"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/72#issuecomment-1105474232", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/72", "id": 1105474232, "node_id": "IC_kwDODFdgUs5B5DK4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-21T17:02:15Z", "updated_at": "2022-04-21T17:02:15Z", "author_association": "MEMBER", "body": "That's interesting - yeah it looks like the number of pages can be derived from the `Link` header, which is enough information to show a progress bar, probably using Click just to avoid adding another dependency.\r\n\r\nhttps://docs.github.com/en/rest/guides/traversing-with-pagination", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1211283427, "label": "feature: display progress bar when downloading multi-page responses"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/pull/73#issuecomment-1188223299", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/73", "id": 1188223299, "node_id": "IC_kwDODFdgUs5G0tlD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-18T19:40:06Z", "updated_at": "2022-07-18T19:40:06Z", "author_association": "MEMBER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1261884917, "label": "Fixing 'NoneType' object has no attribute 'items'"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188223933", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74", "id": 1188223933, "node_id": "IC_kwDODFdgUs5G0tu9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-18T19:40:50Z", "updated_at": "2022-07-18T19:42:41Z", "author_association": "MEMBER", "body": "Here's how the demo is deployed: https://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/.github/workflows/deploy-demo.yml#L103-L119\r\n\r\nI'm suspicious of `py-gfm`, which is used like this:\r\n\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/demo-metadata.json#L49-L51", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1308461063, "label": "500 error in github-to-sqlite demo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188228964", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74", "id": 1188228964, "node_id": "IC_kwDODFdgUs5G0u9k", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-18T19:45:30Z", "updated_at": "2022-07-18T19:47:35Z", "author_association": "MEMBER", "body": "`pycmarkgfm` doesn't implement the Markdown plugin extension I was using.\r\n\r\nI'm going to drop the GFM rendering from the demo, and just treat it as regular markdown.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1308461063, "label": "500 error in github-to-sqlite demo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188225625", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74", "id": 1188225625, "node_id": "IC_kwDODFdgUs5G0uJZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-18T19:41:52Z", "updated_at": "2022-07-18T19:41:52Z", "author_association": "MEMBER", "body": "https://github.com/Zopieux/py-gfm says that library is no longer maintained, and suggests https://github.com/Zopieux/pycmarkgfm as an alternative.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1308461063, "label": "500 error in github-to-sqlite demo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188233729", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74", "id": 1188233729, "node_id": "IC_kwDODFdgUs5G0wIB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-18T19:51:02Z", "updated_at": "2022-07-18T19:51:02Z", "author_association": "MEMBER", "body": "Takes 30m to deploy the demo!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1308461063, "label": "500 error in github-to-sqlite demo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188317682", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74", "id": 1188317682, "node_id": "IC_kwDODFdgUs5G1Eny", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-18T21:14:22Z", "updated_at": "2022-07-18T21:14:22Z", "author_association": "MEMBER", "body": "That fixed it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1308461063, "label": "500 error in github-to-sqlite demo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/79#issuecomment-1498167714", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79", "id": 1498167714, "node_id": "IC_kwDODFdgUs5ZTDmi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-05T21:12:55Z", "updated_at": "2023-04-05T21:12:55Z", "author_association": "MEMBER", "body": "It's a rate limiting problem: https://github.com/dogsheep/github-to-sqlite/actions/runs/4622674009/jobs/8175633155\r\n\r\n```\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py\", line 407, in commits\r\n utils.save_commits(db, commits, repo_full[\"id\"])\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 635, in save_commits\r\n for commit in commits:\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 417, in fetch_commits\r\n for commits in paginate(url, headers):\r\n File \"/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py\", line 478, in paginate\r\n raise GitHubError.from_response(response)\r\ngithub_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403)\r\nError: Process completed with exit code 1.\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1570375808, "label": "Deploy demo job is failing due to rate limit"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1221623052", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10", "id": 1221623052, "node_id": "IC_kwDODLZ_YM5I0H0M", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-21T21:20:33Z", "updated_at": "2022-08-21T21:20:33Z", "author_association": "MEMBER", "body": "That was clearly the intention from the description of this issue:\r\n- #4", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1246826792, "label": "When running `auth` command, don't overwrite an existing auth.json file"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221621466", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11", "id": 1221621466, "node_id": "IC_kwDODLZ_YM5I0Hba", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-21T21:09:47Z", "updated_at": "2022-08-21T21:09:47Z", "author_association": "MEMBER", "body": "Great catch, thanks.\r\n\r\nI'm going to use it to mean `--auth` - since other tools in the Dogsheep family have the same convention.\r\n\r\n `--all` will be the only way to specify all.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1345452427, "label": "-a option is used for \"--auth\" and for \"--all\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221621529", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11", "id": 1221621529, "node_id": "IC_kwDODLZ_YM5I0HcZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-21T21:10:15Z", "updated_at": "2022-08-21T21:11:26Z", "author_association": "MEMBER", "body": "Just saw that's what's implemented here already!\r\n- #7", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1345452427, "label": "-a option is used for \"--auth\" and for \"--all\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/pull/7#issuecomment-1221621700", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/7", "id": 1221621700, "node_id": "IC_kwDODLZ_YM5I0HfE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-21T21:11:12Z", "updated_at": "2022-08-21T21:11:12Z", "author_association": "MEMBER", "body": "I thought this might need a documentation update but `--all` is already covered: https://github.com/dogsheep/pocket-to-sqlite/blob/0.2.1/README.md", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 750141615, "label": "Fixed conflicting CLI flags"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1221622873", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10", "id": 1221622873, "node_id": "IC_kwDODLZ_YM5I0HxZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-21T21:19:25Z", "updated_at": "2022-08-21T21:19:25Z", "author_association": "MEMBER", "body": "Agreed, that would be a much better implementation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1246826792, "label": "When running `auth` command, don't overwrite an existing auth.json file"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627563202", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12", "id": 1627563202, "node_id": "IC_kwDODLZ_YM5hAqTC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-09T01:14:27Z", "updated_at": "2023-07-09T01:14:27Z", "author_association": "MEMBER", "body": "I tested this locally with `python -m build` and then `pip install ...whl` in a fresh virtual environment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1795187493, "label": "Switch to pyproject.toml"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627564127", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12", "id": 1627564127, "node_id": "IC_kwDODLZ_YM5hAqhf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-09T01:19:42Z", "updated_at": "2023-07-09T01:19:42Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/pocket-to-sqlite/tree/0.2.3 and https://pypi.org/project/pocket-to-sqlite/0.2.3/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1795187493, "label": "Switch to pyproject.toml"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886136224", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2", "id": 886136224, "node_id": "IC_kwDODtX3eM400V2g", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T02:08:29Z", "updated_at": "2021-07-25T02:08:29Z", "author_association": "MEMBER", "body": "Prototype:\r\n\r\n curl \"https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000\" | \\\r\n jq .hits | sqlite-utils insert hn.db items - --pk objectID --alter", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952179830, "label": "Command for fetching Hacker News threads from the search API"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135562", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2", "id": 886135562, "node_id": "IC_kwDODtX3eM400VsK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T02:01:11Z", "updated_at": "2021-07-25T02:01:11Z", "author_association": "MEMBER", "body": "That page doesn't have an API but does look easy to scrape.\r\n\r\nThe other option here is the HN Search API powered by Algolia, documented at https://hn.algolia.com/api", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952179830, "label": "Command for fetching Hacker News threads from the search API"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135922", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2", "id": 886135922, "node_id": "IC_kwDODtX3eM400Vxy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T02:06:20Z", "updated_at": "2021-07-25T02:06:20Z", "author_association": "MEMBER", "body": "https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url looks like it does what I want.\r\n\r\nhttps://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000 - returns 1000 at once.\r\n\r\nOtherwise you have to paginate using `&page=2` etc - up to `nbPages` pages.\r\n\r\nhttps://www.algolia.com/doc/api-reference/api-parameters/hitsPerPage/ says 1000 is the maximum.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952179830, "label": "Command for fetching Hacker News threads from the search API"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886140431", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2", "id": 886140431, "node_id": "IC_kwDODtX3eM400W4P", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T03:12:57Z", "updated_at": "2021-07-25T03:12:57Z", "author_association": "MEMBER", "body": "I'm going to build a general-purpose `hacker-new-to-sqlite search ...` command, where one of the options is to search within the URL.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952179830, "label": "Command for fetching Hacker News threads from the search API"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886142671", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3", "id": 886142671, "node_id": "IC_kwDODtX3eM400XbP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T03:51:05Z", "updated_at": "2021-07-25T03:51:05Z", "author_association": "MEMBER", "body": "Prototype:\r\n\r\n curl 'https://hn.algolia.com/api/v1/items/27941108' \\\r\n | jq '[recurse(.children[]) | del(.children)]' \\\r\n | sqlite-utils insert hn.db items - --pk id\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952189173, "label": "Use HN algolia endpoint to retrieve trees"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886237834", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3", "id": 886237834, "node_id": "IC_kwDODtX3eM400uqK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T18:05:32Z", "updated_at": "2021-07-25T18:05:32Z", "author_association": "MEMBER", "body": "If you hit the endpoint for a comment that's part of a thread you get that comment and its recursive children: https://hn.algolia.com/api/v1/items/27941552\r\n\r\nYou can tell that it's not the top-level because the `parent_id` isn't `null`. You can use `story_id` to figure out what the top-level item is.\r\n\r\n```json\r\n{\r\n \"id\": 27941552,\r\n \"created_at\": \"2021-07-24T15:08:39.000Z\",\r\n \"created_at_i\": 1627139319,\r\n \"type\": \"comment\",\r\n \"author\": \"nine_k\",\r\n \"title\": null,\r\n \"url\": null,\r\n \"text\": \"

I wish ...\",\r\n \"points\": null,\r\n \"parent_id\": 27941108,\r\n \"story_id\": 27941108\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952189173, "label": "Use HN algolia endpoint to retrieve trees"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886241674", "issue_url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3", "id": 886241674, "node_id": "IC_kwDODtX3eM400vmK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-25T18:41:17Z", "updated_at": "2021-07-25T18:41:17Z", "author_association": "MEMBER", "body": "Got a TIL out of this: https://til.simonwillison.net/jq/extracting-objects-recursively", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 952189173, "label": "Use HN algolia endpoint to retrieve trees"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905203570", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 905203570, "node_id": "IC_kwDOEhK-wc419E9y", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-25T05:51:22Z", "updated_at": "2021-08-25T05:53:27Z", "author_association": "MEMBER", "body": "The debugger showed me that it broke on a string that looked like this:\r\n```xml\r\n\r\n\r\n\r\n

\r\n \r\n Q3 2018 Reflection & Development\r\n \r\n

\r\n ...\r\n```\r\nYeah that is not valid XML!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978743426, "label": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905206234", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 905206234, "node_id": "IC_kwDOEhK-wc419Fna", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-25T05:58:42Z", "updated_at": "2021-08-25T05:58:42Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/evernote-to-sqlite/blob/36a466f142e5bad52719851c2fbda0c05cd35b99/evernote_to_sqlite/utils.py#L34-L42\r\n\r\nNot sure why I was round-tripping the `content_xml` like that - I will try not doing that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978743426, "label": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906635938", "issue_url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13", "id": 906635938, "node_id": "IC_kwDOEhK-wc42Ciqi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-26T18:18:27Z", "updated_at": "2021-08-26T18:18:27Z", "author_association": "MEMBER", "body": "It looks like I was using the round-trip to dump the `` and `` content - it's just HTML, not even trying to be XML.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978743426, "label": "xml.etree.ElementTree.ParseError: not well-formed (invalid token)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461226402", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1", "id": 1461226402, "node_id": "IC_kwDOJHON9s5XGIui", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T03:47:11Z", "updated_at": "2023-03-09T03:47:11Z", "author_association": "MEMBER", "body": "Me:\r\n> Write applescript to loop through all of my Apple Notes and output their contents\r\n\r\nChatGPT:\r\n\r\n```applescript\r\ntell application \"Notes\"\r\n repeat with eachNote in every note\r\n set noteTitle to the name of eachNote\r\n set noteBody to the body of eachNote\r\n display dialog noteTitle & return & return & noteBody\r\n end repeat\r\nend tell\r\n```\r\nMe:\r\n> How can I turn that into a zsh script which will output the results to my terminal\r\n\r\nChatGPT:\r\n```zsh\r\n#!/usr/bin/zsh\r\n\r\nosascript -e 'tell application \"Notes\"\r\n repeat with eachNote in every note\r\n set noteTitle to the name of eachNote\r\n set noteBody to the body of eachNote\r\n set output to noteTitle & \"\\n\" & noteBody & \"\\n\"\r\n display dialog output\r\n log output\r\n end repeat\r\nend tell'\r\n```\r\nI had to change that first line to `#!/bin/zsh` - then when I ran it it spammed my screen with dialog boxes.\r\n\r\nMe:\r\n\r\n> I want to log the notes contents to the terminal, not display it in a dialog\r\n\r\nChatGPT:\r\n\r\n```zsh\r\n#!/bin/zsh\r\n\r\nosascript -e 'tell application \"Notes\"\r\n repeat with eachNote in every note\r\n set noteTitle to the name of eachNote\r\n set noteBody to the body of eachNote\r\n set output to noteTitle & \"\\n\" & noteBody & \"\\n\"\r\n log output\r\n end repeat\r\nend tell'\r\n```\r\nThis really helps!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616347574, "label": "Initial proof of concept with ChatGPT"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461230197", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1", "id": 1461230197, "node_id": "IC_kwDOJHON9s5XGJp1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T03:51:36Z", "updated_at": "2023-03-09T03:51:36Z", "author_association": "MEMBER", "body": "After a few more rounds I got to this script, which outputs them to a `/tmp/notes.txt` file:\r\n\r\n```zsh\r\n#!/bin/zsh\r\n\r\nosascript -e '\r\nset notesFile to \"/tmp/notes.txt\"\r\nset fileRef to open for access notesFile with write permission\r\ntell application \"Notes\"\r\n repeat with eachNote in every note\r\n set noteId to the id of eachNote\r\n set noteTitle to the name of eachNote\r\n set noteBody to the body of eachNote\r\n write \"------------------------\" & \"\\n\" to fileRef\r\n write noteId & \"\\n\" to fileRef\r\n write noteTitle & \"\\n\\n\" to fileRef\r\n write noteBody & \"\\n\" to fileRef\r\n end repeat\r\nend tell\r\nclose access fileRef'\r\n```\r\nThen I wrote this little Python script to load them into a database:\r\n```python\r\nimport sqlite_utils\r\nsplit = b\"------------------------\\n\"\r\ns = open(\"/tmp/notes.txt\", \"rb\").read()\r\nnotes = [n.decode(\"mac_roman\") for n in s.split(split) if n]\r\n\r\ncleaned_notes = [{\r\n \"id\": n.split(\"\\n\")[0],\r\n \"title\": n.split(\"\\n\")[1],\r\n \"body\": \"\\n\".join(n.split(\"\\n\")[2:]).strip()\r\n} for n in notes]\r\n\r\ndb = sqlite_utils.Database(\"/tmp/notes.db\")\r\ndb[\"notes\"].insert_all(cleaned_notes)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616347574, "label": "Initial proof of concept with ChatGPT"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461230436", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1", "id": 1461230436, "node_id": "IC_kwDOJHON9s5XGJtk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T03:51:52Z", "updated_at": "2023-03-09T03:51:52Z", "author_association": "MEMBER", "body": "This did the job! Next step is to turn that into a Python script.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616347574, "label": "Initial proof of concept with ChatGPT"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461232709", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461232709, "node_id": "IC_kwDOJHON9s5XGKRF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T03:54:28Z", "updated_at": "2023-03-09T03:54:28Z", "author_association": "MEMBER", "body": "I think the AppleScript I want to pass to `osascript` looks like this:\r\n```applescript\r\ntell application \"Notes\"\r\n repeat with eachNote in every note\r\n set noteId to the id of eachNote\r\n set noteTitle to the name of eachNote\r\n set noteBody to the body of eachNote\r\n log \"------------------------\" & \"\\n\"\r\n log noteId & \"\\n\"\r\n log noteTitle & \"\\n\\n\"\r\n log noteBody & \"\\n\"\r\n end repeat\r\nend tell\r\n```\r\nBut there are a few more properties I'd like to get - created and updated date for example.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461234311", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461234311, "node_id": "IC_kwDOJHON9s5XGKqH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T03:56:24Z", "updated_at": "2023-03-09T03:56:24Z", "author_association": "MEMBER", "body": "I opened the \"Script Editor\" app on my computer, used Window -> Library to open the Library panel, then clicked on the Notes app there. I got this:\r\n\r\n\"image\"\r\n\r\nSo the notes object has these properties:\r\n\r\n- name (text) : the name of the note (normally the first line of the body)\r\n- id (text, r/o) : the unique identifier of the note\r\n- container ([folder](applewebdata://621FA8D9-C995-4081-B3B3-149B0EA04C7F#Notes-Suite.folder), r/o) : the folder of the note\r\n- body (text) : the HTML content of the note\r\n- plaintext (text, r/o) : the plaintext content of the note\r\n- creation date (date, r/o) : the creation date of the note\r\n- modification date (date, r/o) : the modification date of the note\r\n- password protected (boolean, r/o) : Is the note password protected?\r\n- shared (boolean, r/o) : Is the note shared?\r\n\r\nI'm going to ignore the concept of attachments for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461234591", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461234591, "node_id": "IC_kwDOJHON9s5XGKuf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T03:56:45Z", "updated_at": "2023-03-09T03:56:45Z", "author_association": "MEMBER", "body": "My prototype showed that images embedded in notes come out in the HTML export as bas64 image URLs, which is neat.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461259490", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461259490, "node_id": "IC_kwDOJHON9s5XGQzi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T04:24:27Z", "updated_at": "2023-03-09T04:24:27Z", "author_association": "MEMBER", "body": "Converting AppleScript date strings to ISO format is hard!\r\n\r\nhttps://forum.latenightsw.com/t/formatting-dates/841 has a recipe I'll try:\r\n\r\n set todayISO to (todayDate as \u00abclass isot\u00bb as string)\r\n\r\nNot clear to me how timezones work here. I'm going to ignore them for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461260978", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461260978, "node_id": "IC_kwDOJHON9s5XGRKy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T04:27:18Z", "updated_at": "2023-03-09T04:27:18Z", "author_association": "MEMBER", "body": "Before that conversion:\r\n\r\n Monday, March 6, 2023 at 11:55:15 AM\r\n\r\nAfter:\r\n\r\n 2023-03-06T11:55:15", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461262577", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461262577, "node_id": "IC_kwDOJHON9s5XGRjx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T04:30:00Z", "updated_at": "2023-03-09T04:30:00Z", "author_association": "MEMBER", "body": "It doesn't have tests yet. I guess I'll need to mock `subprocess` to test this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461285545", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461285545, "node_id": "IC_kwDOJHON9s5XGXKp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T05:06:24Z", "updated_at": "2023-03-09T05:06:24Z", "author_association": "MEMBER", "body": "OK, this works!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/4#issuecomment-1462554175", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4", "id": 1462554175, "node_id": "IC_kwDOJHON9s5XLM4_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T18:19:34Z", "updated_at": "2023-03-09T18:19:34Z", "author_association": "MEMBER", "body": "It looks like the iteration order is most-recently-modified-first - I tried editing a note a bit further back in my notes app and it was the first one output by `apple-notes-to-sqlite --dump`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616429236, "label": "Support incremental updates"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/4#issuecomment-1462556829", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4", "id": 1462556829, "node_id": "IC_kwDOJHON9s5XLNid", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T18:20:56Z", "updated_at": "2023-03-09T18:20:56Z", "author_association": "MEMBER", "body": "In terms of the UI: I'm tempted to say that the default behaviour is for it to run until it sees a note that it already knows about AND that has matching update/created dates, and then stop.\r\n\r\nYou can do a full import again ignoring that logic with `apple-notes-to-sqlite notes.db --full`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616429236, "label": "Support incremental updates"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462562735", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7", "id": 1462562735, "node_id": "IC_kwDOJHON9s5XLO-v", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T18:23:56Z", "updated_at": "2023-03-09T18:25:22Z", "author_association": "MEMBER", "body": "From the Script Editor library docs:\r\n\r\nA note has a:\r\n\r\n> - `container` (folder), r/o) : the folder of the note\r\n\r\nHere's what a folder looks like:\r\n\r\n> folder\u2002n : a folder containing notes\r\n> elements:\r\n>\r\n> - contains folders, notes; contained by application, accounts, folders.\r\n>\r\n> properties:\r\n>\r\n> - `name` (text) : the name of the folder\r\n> - `id` (text, r/o) : the unique identifier of the folder\r\n> - `shared` (boolean, r/o) : Is the folder shared?\r\n> - `container` (account or folder, r/o) : the container of the folder\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617769847, "label": "Folder support"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462564717", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7", "id": 1462564717, "node_id": "IC_kwDOJHON9s5XLPdt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T18:25:39Z", "updated_at": "2023-03-09T18:25:39Z", "author_association": "MEMBER", "body": "So it looks like folders can be hierarchical?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617769847, "label": "Folder support"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462570187", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7", "id": 1462570187, "node_id": "IC_kwDOJHON9s5XLQzL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T18:30:24Z", "updated_at": "2023-03-09T18:30:24Z", "author_association": "MEMBER", "body": "I used ChatGPT to write this:\r\n```\r\nosascript -e 'tell application \"Notes\"\r\n set allFolders to folders\r\n repeat with aFolder in allFolders\r\n set folderId to id of aFolder\r\n set folderName to name of aFolder\r\n set folderContainer to container of aFolder\r\n set folderContainerName to name of folderContainer\r\n log \"Folder ID: \" & folderId\r\n log \"Folder Name: \" & folderName\r\n log \"Folder Container: \" & folderContainerName\r\n log \" \"\r\n --check for nested folders\r\n if count of folders of aFolder > 0 then\r\n set nestedFolders to folders of aFolder\r\n repeat with aNestedFolder in nestedFolders\r\n set nestedFolderId to id of aNestedFolder\r\n set nestedFolderName to name of aNestedFolder\r\n set nestedFolderContainer to container of aNestedFolder\r\n set nestedFolderContainerName to name of nestedFolderContainer\r\n log \" Nested Folder ID: \" & nestedFolderId\r\n log \" Nested Folder Name: \" & nestedFolderName\r\n log \" Nested Folder Container: \" & nestedFolderContainerName\r\n log \" \"\r\n end repeat\r\n end if\r\n end repeat\r\nend tell\r\n'\r\n```\r\nWhich for my account output this:\r\n```\r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113\r\nFolder Name: Blog posts\r\nFolder Container: iCloud\r\n \r\n Nested Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995\r\n Nested Folder Name: Nested inside blog posts\r\n Nested Folder Container: Blog posts\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698\r\nFolder Name: JSK\r\nFolder Container: iCloud\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995\r\nFolder Name: Nested inside blog posts\r\nFolder Container: Blog posts\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526\r\nFolder Name: New Folder\r\nFolder Container: iCloud\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839\r\nFolder Name: New Folder 1\r\nFolder Container: iCloud\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2\r\nFolder Name: Notes\r\nFolder Container: iCloud\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059\r\nFolder Name: Quick Notes\r\nFolder Container: iCloud\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283\r\nFolder Name: UK Christmas 2022\r\nFolder Container: iCloud\r\n```\r\nSo I think the correct approach here is to run code at the start to list all of the folders (no need to do fancy recursion though, just a flat list with the parent containers is enough) and create a model of that hierarchy in SQLite.\r\n\r\nThen when I import notes I can foreign key reference them back to their containing folder.\r\n\r\nI'm tempted to use `rowid` for the foreign keys because the official IDs are pretty long.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617769847, "label": "Folder support"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462682795", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7", "id": 1462682795, "node_id": "IC_kwDOJHON9s5XLsSr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T19:52:20Z", "updated_at": "2023-03-09T19:52:44Z", "author_association": "MEMBER", "body": "Created through several rounds with ChatGPT (including hints like \"rewrite that using setdefault()\"):\r\n```python\r\ndef topological_sort(nodes):\r\n children = {}\r\n for node in nodes:\r\n parent_id = node[\"parent\"]\r\n if parent_id is not None:\r\n children.setdefault(parent_id, []).append(node)\r\n\r\n def traverse(node, result):\r\n result.append(node)\r\n if node[\"id\"] in children:\r\n for child in children[node[\"id\"]]:\r\n traverse(child, result)\r\n\r\n sorted_data = []\r\n\r\n for node in nodes:\r\n if node[\"parent\"] is None:\r\n traverse(node, sorted_data)\r\n\r\n return sorted_data\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617769847, "label": "Folder support"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462693867", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7", "id": 1462693867, "node_id": "IC_kwDOJHON9s5XLu_r", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T20:01:39Z", "updated_at": "2023-03-09T20:02:11Z", "author_association": "MEMBER", "body": "My `folders` table will have:\r\n\r\n- `id` - rowid\r\n- `long_id` - that long unique string ID\r\n- `name` - the name\r\n- `parent` - foreign key to `id`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617769847, "label": "Folder support"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462691466", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7", "id": 1462691466, "node_id": "IC_kwDOJHON9s5XLuaK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T19:59:52Z", "updated_at": "2023-03-09T19:59:52Z", "author_association": "MEMBER", "body": "Improved script:\r\n```zsh\r\nosascript -e 'tell application \"Notes\"\r\n set allFolders to folders\r\n repeat with aFolder in allFolders\r\n set folderId to id of aFolder\r\n set folderName to name of aFolder\r\n set folderContainer to container of aFolder\r\n if class of folderContainer is folder then\r\n set folderContainerId to id of folderContainer\r\n else\r\n set folderContainerId to \"\"\r\n end if\r\n log \"ID: \" & folderId\r\n log \"Name: \" & folderName\r\n log \"Container: \" & folderContainerId\r\n log \" \"\r\n end repeat\r\nend tell\r\n'\r\n```\r\n```\r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113\r\nName: Blog posts\r\nContainer: \r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698\r\nName: JSK\r\nContainer: \r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995\r\nName: Nested inside blog posts\r\nContainer: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113\r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526\r\nName: New Folder\r\nContainer: \r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839\r\nName: New Folder 1\r\nContainer: \r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2\r\nName: Notes\r\nContainer: \r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059\r\nName: Quick Notes\r\nContainer: \r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283\r\nName: UK Christmas 2022\r\nContainer: \r\n```\r\nI filtered out things where the parent was an account and not a folder using `if class of folderContainer is folder then`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617769847, "label": "Folder support"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462962682", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11", "id": 1462962682, "node_id": "IC_kwDOJHON9s5XMwn6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T23:20:35Z", "updated_at": "2023-03-09T23:22:41Z", "author_association": "MEMBER", "body": "Here's a query that returns all notes in folder 1, including notes in descendant folders:\r\n```sql\r\nwith recursive nested_folders(folder_id, descendant_folder_id) as (\r\n -- base case: select all immediate children of the root folder\r\n select id, id from folders where parent is null\r\n union all\r\n -- recursive case: select all children of the previous level of nested folders\r\n select nf.folder_id, f.id from nested_folders nf\r\n join folders f on nf.descendant_folder_id = f.parent\r\n)\r\n-- Find notes within all descendants of folder 1\r\nselect *\r\nfrom notes\r\nwhere folder in (\r\n select descendant_folder_id from nested_folders where folder_id = 1\r\n);\r\n```\r\nWith assistance from ChatGPT. Prompts were:\r\n\r\n```\r\nSQLite schema:\r\n\r\nCREATE TABLE [folders] (\r\n [id] INTEGER PRIMARY KEY,\r\n [long_id] TEXT,\r\n [name] TEXT,\r\n [parent] INTEGER,\r\n FOREIGN KEY([parent]) REFERENCES [folders]([id])\r\n);\r\n\r\nWrite a recursive CTE that returns the following:\r\n\r\nfolder_id | descendant_folder_id\r\n\r\nWith a row for every nested child of every folder - so the top level folder has lots of rows\r\n```\r\nThen I tweaked it a bit, then ran this:\r\n```\r\nWITH RECURSIVE nested_folders(folder_id, descendant_folder_id) AS (\r\n -- base case: select all immediate children of the root folder\r\n SELECT id, id FROM folders WHERE parent IS NULL\r\n UNION ALL\r\n -- recursive case: select all children of the previous level of nested folders\r\n SELECT nf.folder_id, f.id FROM nested_folders nf\r\n JOIN folders f ON nf.descendant_folder_id = f.parent\r\n)\r\n-- select all rows from the recursive CTE\r\nSELECT * from notes where folder in (select descendant_folder_id FROM nested_folders where folder_id = 1)\r\n\r\nConvert all SQL keywords to lower case, and re-indent\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1618130434, "label": "Implement a SQL view to make it easier to query files in a nested folder"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462968053", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11", "id": 1462968053, "node_id": "IC_kwDOJHON9s5XMx71", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T23:24:01Z", "updated_at": "2023-03-09T23:24:01Z", "author_association": "MEMBER", "body": "I improved the readability by removing some unnecessary table aliases:\r\n```sql\r\nwith recursive nested_folders(folder_id, descendant_folder_id) as (\r\n -- base case: select all immediate children of the root folder\r\n select id, id from folders where parent is null\r\n union all\r\n -- recursive case: select all children of the previous level of nested folders\r\n select nested_folders.folder_id, folders.id from nested_folders\r\n join folders on nested_folders.descendant_folder_id = folders.parent\r\n)\r\n-- Find notes within all descendants of folder 1\r\nselect *\r\nfrom notes\r\nwhere folder in (\r\n select descendant_folder_id from nested_folders where folder_id = 1\r\n);\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1618130434, "label": "Implement a SQL view to make it easier to query files in a nested folder"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462965256", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11", "id": 1462965256, "node_id": "IC_kwDOJHON9s5XMxQI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T23:22:12Z", "updated_at": "2023-03-09T23:22:12Z", "author_association": "MEMBER", "body": "Here's what the CTE from that looks like:\r\n\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1618130434, "label": "Implement a SQL view to make it easier to query files in a nested folder"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/4#issuecomment-540879620", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4", "id": 540879620, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MDg3OTYyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T02:59:16Z", "updated_at": "2019-10-11T02:59:16Z", "author_association": "MEMBER", "body": "Also import ad preferences and all that other junk.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488835586, "label": "Command for importing data from a Twitter Export file"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541141169", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18", "id": 541141169, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTE0MTE2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T16:51:29Z", "updated_at": "2019-10-11T16:51:29Z", "author_association": "MEMBER", "body": "Documented here: https://github.com/dogsheep/twitter-to-sqlite/blob/master/README.md#retrieving-tweets-from-your-home-timeline", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 505928530, "label": "Command to import home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112108", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17", "id": 541112108, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTExMjEwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T15:30:15Z", "updated_at": "2019-10-11T15:30:15Z", "author_association": "MEMBER", "body": "It should delete the tables entirely. That way it will work even if the table schema has changed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 505674949, "label": "import command should empty all archive-* tables first"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112588", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17", "id": 541112588, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTExMjU4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T15:31:30Z", "updated_at": "2019-10-11T15:31:30Z", "author_association": "MEMBER", "body": "No need for an option:\r\n\r\n> This command will delete and recreate all of your `archive-*` tables every time you run it. If this is not what you want, run the command against a fresh SQLite database rather than running it again one that already exists.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 505674949, "label": "import command should empty all archive-* tables first"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541118773", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18", "id": 541118773, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTExODc3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T15:48:31Z", "updated_at": "2019-10-11T15:48:31Z", "author_association": "MEMBER", "body": "https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-home_timeline", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 505928530, "label": "Command to import home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541118934", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18", "id": 541118934, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTExODkzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T15:48:54Z", "updated_at": "2019-10-11T15:48:54Z", "author_association": "MEMBER", "body": "Rate limit is tight: 15 requests every 15 mins!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 505928530, "label": "Command to import home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541119834", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18", "id": 541119834, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTExOTgzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T15:51:22Z", "updated_at": "2019-10-11T16:51:33Z", "author_association": "MEMBER", "body": "In order to support multiple user timelines being saved in the same database, I'm going to import the tweets into the `tweets` table AND add a new `timeline_tweets` table recording that a specific tweet showed up in a specific user's timeline.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 505928530, "label": "Command to import home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-541248629", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19", "id": 541248629, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTI0ODYyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-11T22:48:56Z", "updated_at": "2019-10-11T22:48:56Z", "author_association": "MEMBER", "body": "`since_id` documented here: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-home_timeline\r\n\r\n> Returns results with an ID greater than (that is, more recent than) the specified ID. There are limits to the number of Tweets which can be accessed through the API. If the limit of Tweets has occured since the since_id, the since_id will be forced to the oldest ID available.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506087267, "label": "since_id support for home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/6#issuecomment-541387822", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6", "id": 541387822, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTM4NzgyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-13T05:27:39Z", "updated_at": "2019-10-13T05:27:39Z", "author_association": "MEMBER", "body": "This should be fixed by https://github.com/dogsheep/github-to-sqlite/commit/552543a74970f8a3a3f87f887be23a0c6eb1cb5b", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 504238461, "label": "sqlite3.OperationalError: table users has no column named bio"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/6#issuecomment-541387941", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6", "id": 541387941, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTM4Nzk0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-13T05:30:19Z", "updated_at": "2019-10-13T05:30:19Z", "author_association": "MEMBER", "body": "Fix released in 0.5: https://github.com/dogsheep/github-to-sqlite/releases/tag/0.5", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 504238461, "label": "sqlite3.OperationalError: table users has no column named bio"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-541388038", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20", "id": 541388038, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTM4ODAzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-13T05:31:58Z", "updated_at": "2019-10-13T05:31:58Z", "author_association": "MEMBER", "body": "For favourites a `--stop_after=200` option is probably good enough.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506268945, "label": "--since support for various commands for refresh-by-cron"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-541493242", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3", "id": 541493242, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTQ5MzI0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-14T03:35:36Z", "updated_at": "2019-10-14T03:35:36Z", "author_association": "MEMBER", "body": "https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488833975, "label": "Command for running a search and saving tweets for that search"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-541748580", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10", "id": 541748580, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTc0ODU4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-14T15:30:44Z", "updated_at": "2019-10-14T15:30:44Z", "author_association": "MEMBER", "body": "Had several recommendations for https://github.com/tqdm/tqdm which is what goodreads-to-sqlite uses.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 492297930, "label": "Rethink progress bars for various commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/7#issuecomment-541721437", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/7", "id": 541721437, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MTcyMTQzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-14T14:44:12Z", "updated_at": "2019-10-14T14:44:12Z", "author_association": "MEMBER", "body": "Docs: https://github.com/dogsheep/github-to-sqlite/blob/0.5/README.md#retrieving-issue-comments-for-a-repository", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506276893, "label": "issue-comments command for importing issue comments"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/21#issuecomment-542333836", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/21", "id": 542333836, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MjMzMzgzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-15T18:00:48Z", "updated_at": "2019-10-15T18:00:48Z", "author_association": "MEMBER", "body": "I'll use `html.unescape()` for this: https://docs.python.org/3/library/html.html#html.unescape", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506432572, "label": "Fix & escapes in tweet text"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542849963", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19", "id": 542849963, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg0OTk2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T19:13:06Z", "updated_at": "2019-10-16T19:13:06Z", "author_association": "MEMBER", "body": "Updated documentation: https://github.com/dogsheep/twitter-to-sqlite/blob/fced2a9b67d2cbdf9817f1eb75f7c28e413c963b/README.md#retrieving-tweets-from-your-home-timeline", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506087267, "label": "since_id support for home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-542854749", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20", "id": 542854749, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg1NDc0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T19:26:01Z", "updated_at": "2019-10-16T19:26:01Z", "author_association": "MEMBER", "body": "I'm not going to do this for \"accounts that have followed me\" and \"new accounts that I have followed\" - instead I will recommend running the `friend_ids` and `followers_ids` commands on a daily basis since that data doesn't really change much by the hour.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506268945, "label": "--since support for various commands for refresh-by-cron"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855081", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12", "id": 542855081, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg1NTA4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T19:26:56Z", "updated_at": "2019-10-16T19:26:56Z", "author_association": "MEMBER", "body": "This may be the first case where I want to be able to repair existing databases rather than discarding their contents.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503053800, "label": "Extract \"source\" into a separate lookup table"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855427", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12", "id": 542855427, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg1NTQyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T19:27:55Z", "updated_at": "2019-10-16T19:27:55Z", "author_association": "MEMBER", "body": "I can do that by keeping `source` as a `TEXT` column but turning it into a non-enforced foreign key against a new `sources` table. Then I can run code that scans that column for any values beginning with a `<` and converts them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503053800, "label": "Extract \"source\" into a separate lookup table"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542858025", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12", "id": 542858025, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg1ODAyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T19:35:31Z", "updated_at": "2019-10-16T19:36:09Z", "author_association": "MEMBER", "body": "Maybe this means I need an `upgrade` command to apply these kinds of migrations? Total feature creep!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503053800, "label": "Extract \"source\" into a separate lookup table"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542875885", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3", "id": 542875885, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg3NTg4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T20:23:08Z", "updated_at": "2019-10-16T20:23:08Z", "author_association": "MEMBER", "body": "https://developer.foursquare.com/docs/api/users/checkins documents `afterTimestamp`:\r\n> Retrieve the first results to follow these seconds since epoch. This should be useful for paging forward in time, or when polling for changes. To avoid missing results when polling, we recommend subtracting several seconds from the last poll time and then de-duplicating.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487600595, "label": "Option to fetch only checkins more recent than the current max checkin"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542876047", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3", "id": 542876047, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg3NjA0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T20:23:36Z", "updated_at": "2019-10-16T20:23:36Z", "author_association": "MEMBER", "body": "I'm going to go with `--since=1d/2w/3h` for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487600595, "label": "Option to fetch only checkins more recent than the current max checkin"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542882604", "issue_url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3", "id": 542882604, "node_id": "MDEyOklzc3VlQ29tbWVudDU0Mjg4MjYwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T20:41:23Z", "updated_at": "2019-10-16T20:41:23Z", "author_association": "MEMBER", "body": "Documented here: https://github.com/dogsheep/swarm-to-sqlite/blob/0.2/README.md#usage", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487600595, "label": "Option to fetch only checkins more recent than the current max checkin"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542832952", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19", "id": 542832952, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MjgzMjk1Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-16T18:30:11Z", "updated_at": "2019-10-16T18:30:11Z", "author_association": "MEMBER", "body": "The `--since` option will derive the `since_id` from the max ID in the `timeline_tweets` table:\r\n\r\n $ twitter-to-sqlite home-timeline --since\r\n\r\nThe `--since_id=xxx` option lets you specify that ID directly.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506087267, "label": "since_id support for home-timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543265058", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25", "id": 543265058, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI2NTA1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T16:51:12Z", "updated_at": "2019-10-17T16:51:12Z", "author_association": "MEMBER", "body": "This migration function only runs if there is a table called `tweets` and the migration has not run before.\r\n\r\nI think this can happen if the database has just been freshly created (by a command that fetches the user's user timeline for example) and is then run a SECOND time.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 508578780, "label": "Ensure migrations don't accidentally create foreign key twice"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543266947", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25", "id": 543266947, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI2Njk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T16:56:06Z", "updated_at": "2019-10-17T16:56:06Z", "author_association": "MEMBER", "body": "I wrote a test that proves that this is a problem. Should be an easy fix though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 508578780, "label": "Ensure migrations don't accidentally create foreign key twice"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543269396", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10", "id": 543269396, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI2OTM5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T17:02:07Z", "updated_at": "2019-10-17T17:02:07Z", "author_association": "MEMBER", "body": "A neat trick that Click does is detecting if an interactive terminal is attached and NOT showing a progress bar if there isn't one. Need to figure out how to do that with tqdm.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 492297930, "label": "Rethink progress bars for various commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543270714", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10", "id": 543270714, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI3MDcxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T17:05:16Z", "updated_at": "2019-10-17T17:05:16Z", "author_association": "MEMBER", "body": "https://github.com/pallets/click/blob/716a5be90f56ce6cd506bb53d5739d09374b1636/click/_termui_impl.py#L93 is how Click does this:\r\n```\r\n self.is_hidden = not isatty(self.file)\r\n```\r\nWhere `isatty` is a Click utility function: `from ._compat import isatty`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 492297930, "label": "Rethink progress bars for various commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543271000", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10", "id": 543271000, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI3MTAwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T17:05:59Z", "updated_at": "2019-10-17T17:05:59Z", "author_association": "MEMBER", "body": "Looks like tqdm already does a TTY check here: https://github.com/tqdm/tqdm/blob/89b73bdc30c099c5b53725806e7edf3a121c9b3a/tqdm/std.py#L889-L890", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 492297930, "label": "Rethink progress bars for various commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543273540", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3", "id": 543273540, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI3MzU0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T17:12:51Z", "updated_at": "2019-10-17T17:12:51Z", "author_association": "MEMBER", "body": "Just importing tweets here isn't enough - how are we supposed to know which tweets were imported by which search?\r\n\r\nSo I think the right thing to do here is to also create a `search_runs` table, which records each individual run of this tool (with a timestamp and the search terms used). Then have a `search_runs_tweets` m2m table which shows which Tweets were found by that search.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488833975, "label": "Command for running a search and saving tweets for that search"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543290744", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3", "id": 543290744, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzI5MDc0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T17:57:14Z", "updated_at": "2019-10-17T17:57:14Z", "author_association": "MEMBER", "body": "I have a working command now. I'm going to ship it early because it could do with some other people trying it out.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488833975, "label": "Command for running a search and saving tweets for that search"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543217890", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23", "id": 543217890, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzIxNzg5MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T15:03:10Z", "updated_at": "2019-10-17T15:03:10Z", "author_association": "MEMBER", "body": "Thinking about this further: the concept of migrations may end up being in direct conflict with the `sqlite-utils` concept of creating tables on demand the first time they are used - and of creating table schemas automatically to fit the shape of the JSON that is being inserted into them.\r\n\r\nI'm going to forge ahead anyway and build this because I think it will be an interesting exploration, but it's very likely this turns out to be a bad idea in the long run!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 508190730, "label": "Extremely simple migration system"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543222239", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23", "id": 543222239, "node_id": "MDEyOklzc3VlQ29tbWVudDU0MzIyMjIzOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-17T15:12:33Z", "updated_at": "2019-10-17T15:12:33Z", "author_association": "MEMBER", "body": "Migrations will run only if you open a database that previously existed (as opposed to opening a brand new empty database).\r\n\r\nThis means that the first time you run a command against a fresh database, migrations will not run and the `migrations` table will not be created. The _second_ time you run any command against that database the migrations will execute and populate the `migrations` table.\r\n\r\nThis also means that each migration needs to be able to sanity check the database to see if it should run or not. If it should NOT run, it will do nothing but still be marked as having executed by adding to the `migrations` table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 508190730, "label": "Extremely simple migration system"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-544335363", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20", "id": 544335363, "node_id": "MDEyOklzc3VlQ29tbWVudDU0NDMzNTM2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-21T03:32:04Z", "updated_at": "2019-10-21T03:32:04Z", "author_association": "MEMBER", "body": "In case anyone is interested, here's an extract from the crontab I'm running these under at the moment:\r\n```\r\n1,11,21,31,41,51 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite user-timeline /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --since\r\n2,7,12,17,22,27,32,37,42,47,52,57 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite home-timeline /home/ubuntu/timeline.db -a /home/ubuntu/auth.json --since\r\n6,16,26,36,46,56 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite favorites /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --stop_after=50\r\n```", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 506268945, "label": "--since support for various commands for refresh-by-cron"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544646516", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 544646516, "node_id": "MDEyOklzc3VlQ29tbWVudDU0NDY0NjUxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-21T18:30:14Z", "updated_at": "2019-10-21T18:30:14Z", "author_association": "MEMBER", "body": "Thanks to help from Dr. Laura Cantino at Science Hack Day San Francisco I've been able to pull together this query:\r\n\r\n```sql\r\nselect rsid, genotype, case genotype\r\n when 'AA' then 'brown eye color, 80% of the time'\r\n when 'AG' then 'brown eye color'\r\n when 'GG' then 'blue eye color, 99% of the time'\r\nend as interpretation from genome where rsid = 'rs12913832'\r\n```\r\n\r\nSee also https://www.snpedia.com/index.php/Rs12913832 - in particular this table:\r\n\r\n\"rs12913832_-_SNPedia\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544648863", "issue_url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1", "id": 544648863, "node_id": "MDEyOklzc3VlQ29tbWVudDU0NDY0ODg2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-21T18:36:03Z", "updated_at": "2019-10-21T18:36:03Z", "author_association": "MEMBER", "body": "\"natalie__select_rsid__genotype__case_genotype_when__AA__then__brown_eye_color__80__of_the_time__when__AG__then__brown_eye_color__when__GG__then__blue_eye_color__99__of_the_time__end_as_interpretation_from_genome_where_rsid____rs12913832__an\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 496415321, "label": "Figure out some interesting example SQL queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/26#issuecomment-547713287", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/26", "id": 547713287, "node_id": "MDEyOklzc3VlQ29tbWVudDU0NzcxMzI4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-10-30T02:36:13Z", "updated_at": "2019-10-30T02:36:13Z", "author_association": "MEMBER", "body": "Shipped this in 0.13: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.13\r\n\r\nSee also this Twitter thread: https://twitter.com/simonw/status/1189369677509623809", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 513074501, "label": "Command for importing mentions timeline"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549094195", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8", "id": 549094195, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTA5NDE5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-03T00:43:16Z", "updated_at": "2019-11-03T00:43:28Z", "author_association": "MEMBER", "body": "Also need to take #5 into account - if this command creates incomplete user records, how do we repair them?\r\n\r\nAnd make sure that if we run this command first any future commands that populate users don't break (probably just a case of using `alter=True` in a few places).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 516763727, "label": "stargazers command, refs #4"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/5#issuecomment-549094229", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5", "id": 549094229, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTA5NDIyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-03T00:44:03Z", "updated_at": "2019-11-03T00:44:03Z", "author_association": "MEMBER", "body": "Might not need an incomplete boolean - may be possible to handle this with `alter=True` and then by filtering for users with null values in certain columns.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 493671014, "label": "Add \"incomplete\" boolean to users table for incomplete profiles"}, "performed_via_github_app": null}