{"html_url": "https://github.com/simonw/datasette/pull/1999#issuecomment-1461070937", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1999", "id": 1461070937, "node_id": "IC_kwDOBm6k_c5XFixZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T00:18:52Z", "updated_at": "2023-03-09T00:19:36Z", "author_association": "OWNER", "body": "I managed to get HTML view working! I did it by continuing to add more things to the extras and the `_html` bundle until the page loaded for me:\r\n\r\n```diff\r\n async def extra_extras():\r\n \"Available ?_extra= blocks\"\r\n return {\r\n@@ -1981,6 +2053,14 @@ async def extra_extras():\r\n \"query\",\r\n \"display_columns\",\r\n \"display_rows\",\r\n+ \"database\",\r\n+ \"table\",\r\n+ \"database_color\",\r\n+ \"table_actions\",\r\n+ \"filters\",\r\n+ \"renderers\",\r\n+ \"custom_table_templates\",\r\n+ \"sorted_facet_results\",\r\n ]\r\n }\r\n \r\n@@ -2006,6 +2086,14 @@ async def extra_extras():\r\n extra_query,\r\n extra_metadata,\r\n extra_extras,\r\n+ extra_database,\r\n+ extra_table,\r\n+ extra_database_color,\r\n+ extra_table_actions,\r\n+ extra_filters,\r\n+ extra_renderers,\r\n+ extra_custom_table_templates,\r\n+ extra_sorted_facet_results,\r\n )\r\n```\r\nI'll probably refactor this into something cleaner, and maybe but a bunch of them in a `\"html\"` dictionary and update the templates to use `{{ html.filters }}` or similar. Will look at that once the tests are passing.\r\n\r\n![\"image\"](\"https://user-images.githubusercontent.com/9599/223882676-942ff28b-1a4b-4ef0-adaa-ccd664877dcd.png\")
\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1551694938, "label": "?_extra= support (draft)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1999#issuecomment-1461074526", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1999", "id": 1461074526, "node_id": "IC_kwDOBm6k_c5XFjpe", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T00:23:06Z", "updated_at": "2023-03-09T00:23:06Z", "author_association": "OWNER", "body": " pytest tests/test_table_html.py\r\n\r\nCurrently 44 failed, 24 passed in 7.53s\r\n\r\nFailures here: https://gist.github.com/simonw/df0a52cd7d820b776dc3dfc50e7cb778", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1551694938, "label": "?_extra= support (draft)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1999#issuecomment-1461075648", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1999", "id": 1461075648, "node_id": "IC_kwDOBm6k_c5XFj7A", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T00:24:22Z", "updated_at": "2023-03-09T00:24:22Z", "author_association": "OWNER", "body": "`127.0.0.1:8001/fixtures/neighborhood_search` fails because the forwarding to a canned query does not yet work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1551694938, "label": "?_extra= support (draft)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1999#issuecomment-1461148254", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1999", "id": 1461148254, "node_id": "IC_kwDOBm6k_c5XF1pe", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T01:53:41Z", "updated_at": "2023-03-09T01:53:41Z", "author_association": "OWNER", "body": "Solving this is proving difficult: https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L1500-L1503\r\n\r\nThe problem is that calling `.data()` on `QueryView` only works here because we expect to ourselves be inside a `.data()` method, with all of the existing magic that knows how to render things that are returned by that.\r\n\r\nSo I may need to substantially re-engineer how `QueryView` works in order to get this to work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1551694938, "label": "?_extra= support (draft)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1999#issuecomment-1461148579", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1999", "id": 1461148579, "node_id": "IC_kwDOBm6k_c5XF1uj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T01:54:10Z", "updated_at": "2023-03-09T01:55:33Z", "author_association": "OWNER", "body": "Or... I could temporarily build a quick additional `CannedQueryView` subclass that just does the necessary bits to get the existing code to work. I'm going to try that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1551694938, "label": "?_extra= support (draft)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1999#issuecomment-1461161256", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1999", "id": 1461161256, "node_id": "IC_kwDOBm6k_c5XF40o", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T02:10:07Z", "updated_at": "2023-03-09T02:10:07Z", "author_association": "OWNER", "body": "Just ran into a `no such table: columns` error - which I think is because my new view code sometimes bypasses calling this method (currently done in `BaseView.dispatch_request()`):\r\n\r\nhttps://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/base.py#L101-L103", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1551694938, "label": "?_extra= support (draft)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461226402", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1", "id": 1461226402, "node_id": "IC_kwDOJHON9s5XGIui", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T03:47:11Z", "updated_at": "2023-03-09T03:47:11Z", "author_association": "MEMBER", "body": "Me:\r\n> Write applescript to loop through all of my Apple Notes and output their contents\r\n\r\nChatGPT:\r\n\r\n```applescript\r\ntell application \"Notes\"\r\n repeat with eachNote in every note\r\n set noteTitle to the name of eachNote\r\n set noteBody to the body of eachNote\r\n display dialog noteTitle & return & return & noteBody\r\n end repeat\r\nend tell\r\n```\r\nMe:\r\n> How can I turn that into a zsh script which will output the results to my terminal\r\n\r\nChatGPT:\r\n```zsh\r\n#!/usr/bin/zsh\r\n\r\nosascript -e 'tell application \"Notes\"\r\n repeat with eachNote in every note\r\n set noteTitle to the name of eachNote\r\n set noteBody to the body of eachNote\r\n set output to noteTitle & \"\\n\" & noteBody & \"\\n\"\r\n display dialog output\r\n log output\r\n end repeat\r\nend tell'\r\n```\r\nI had to change that first line to `#!/bin/zsh` - then when I ran it it spammed my screen with dialog boxes.\r\n\r\nMe:\r\n\r\n> I want to log the notes contents to the terminal, not display it in a dialog\r\n\r\nChatGPT:\r\n\r\n```zsh\r\n#!/bin/zsh\r\n\r\nosascript -e 'tell application \"Notes\"\r\n repeat with eachNote in every note\r\n set noteTitle to the name of eachNote\r\n set noteBody to the body of eachNote\r\n set output to noteTitle & \"\\n\" & noteBody & \"\\n\"\r\n log output\r\n end repeat\r\nend tell'\r\n```\r\nThis really helps!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616347574, "label": "Initial proof of concept with ChatGPT"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461230197", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1", "id": 1461230197, "node_id": "IC_kwDOJHON9s5XGJp1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T03:51:36Z", "updated_at": "2023-03-09T03:51:36Z", "author_association": "MEMBER", "body": "After a few more rounds I got to this script, which outputs them to a `/tmp/notes.txt` file:\r\n\r\n```zsh\r\n#!/bin/zsh\r\n\r\nosascript -e '\r\nset notesFile to \"/tmp/notes.txt\"\r\nset fileRef to open for access notesFile with write permission\r\ntell application \"Notes\"\r\n repeat with eachNote in every note\r\n set noteId to the id of eachNote\r\n set noteTitle to the name of eachNote\r\n set noteBody to the body of eachNote\r\n write \"------------------------\" & \"\\n\" to fileRef\r\n write noteId & \"\\n\" to fileRef\r\n write noteTitle & \"\\n\\n\" to fileRef\r\n write noteBody & \"\\n\" to fileRef\r\n end repeat\r\nend tell\r\nclose access fileRef'\r\n```\r\nThen I wrote this little Python script to load them into a database:\r\n```python\r\nimport sqlite_utils\r\nsplit = b\"------------------------\\n\"\r\ns = open(\"/tmp/notes.txt\", \"rb\").read()\r\nnotes = [n.decode(\"mac_roman\") for n in s.split(split) if n]\r\n\r\ncleaned_notes = [{\r\n \"id\": n.split(\"\\n\")[0],\r\n \"title\": n.split(\"\\n\")[1],\r\n \"body\": \"\\n\".join(n.split(\"\\n\")[2:]).strip()\r\n} for n in notes]\r\n\r\ndb = sqlite_utils.Database(\"/tmp/notes.db\")\r\ndb[\"notes\"].insert_all(cleaned_notes)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616347574, "label": "Initial proof of concept with ChatGPT"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461230436", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1", "id": 1461230436, "node_id": "IC_kwDOJHON9s5XGJtk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T03:51:52Z", "updated_at": "2023-03-09T03:51:52Z", "author_association": "MEMBER", "body": "This did the job! Next step is to turn that into a Python script.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616347574, "label": "Initial proof of concept with ChatGPT"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461232709", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461232709, "node_id": "IC_kwDOJHON9s5XGKRF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T03:54:28Z", "updated_at": "2023-03-09T03:54:28Z", "author_association": "MEMBER", "body": "I think the AppleScript I want to pass to `osascript` looks like this:\r\n```applescript\r\ntell application \"Notes\"\r\n repeat with eachNote in every note\r\n set noteId to the id of eachNote\r\n set noteTitle to the name of eachNote\r\n set noteBody to the body of eachNote\r\n log \"------------------------\" & \"\\n\"\r\n log noteId & \"\\n\"\r\n log noteTitle & \"\\n\\n\"\r\n log noteBody & \"\\n\"\r\n end repeat\r\nend tell\r\n```\r\nBut there are a few more properties I'd like to get - created and updated date for example.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461234311", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461234311, "node_id": "IC_kwDOJHON9s5XGKqH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T03:56:24Z", "updated_at": "2023-03-09T03:56:24Z", "author_association": "MEMBER", "body": "I opened the \"Script Editor\" app on my computer, used Window -> Library to open the Library panel, then clicked on the Notes app there. I got this:\r\n\r\n![\"image\"](\"https://user-images.githubusercontent.com/9599/223912876-09e8660e-f59a-4f04-8992-668e09c8d7d4.png\")
\r\n\r\nSo the notes object has these properties:\r\n\r\n- name (text) : the name of the note (normally the first line of the body)\r\n- id (text, r/o) : the unique identifier of the note\r\n- container ([folder](applewebdata://621FA8D9-C995-4081-B3B3-149B0EA04C7F#Notes-Suite.folder), r/o) : the folder of the note\r\n- body (text) : the HTML content of the note\r\n- plaintext (text, r/o) : the plaintext content of the note\r\n- creation date (date, r/o) : the creation date of the note\r\n- modification date (date, r/o) : the modification date of the note\r\n- password protected (boolean, r/o) : Is the note password protected?\r\n- shared (boolean, r/o) : Is the note shared?\r\n\r\nI'm going to ignore the concept of attachments for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461234591", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461234591, "node_id": "IC_kwDOJHON9s5XGKuf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T03:56:45Z", "updated_at": "2023-03-09T03:56:45Z", "author_association": "MEMBER", "body": "My prototype showed that images embedded in notes come out in the HTML export as bas64 image URLs, which is neat.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461259490", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461259490, "node_id": "IC_kwDOJHON9s5XGQzi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T04:24:27Z", "updated_at": "2023-03-09T04:24:27Z", "author_association": "MEMBER", "body": "Converting AppleScript date strings to ISO format is hard!\r\n\r\nhttps://forum.latenightsw.com/t/formatting-dates/841 has a recipe I'll try:\r\n\r\n set todayISO to (todayDate as \u00abclass isot\u00bb as string)\r\n\r\nNot clear to me how timezones work here. I'm going to ignore them for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461260978", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461260978, "node_id": "IC_kwDOJHON9s5XGRKy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T04:27:18Z", "updated_at": "2023-03-09T04:27:18Z", "author_association": "MEMBER", "body": "Before that conversion:\r\n\r\n Monday, March 6, 2023 at 11:55:15 AM\r\n\r\nAfter:\r\n\r\n 2023-03-06T11:55:15", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461262577", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461262577, "node_id": "IC_kwDOJHON9s5XGRjx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T04:30:00Z", "updated_at": "2023-03-09T04:30:00Z", "author_association": "MEMBER", "body": "It doesn't have tests yet. I guess I'll need to mock `subprocess` to test this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461285545", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2", "id": 1461285545, "node_id": "IC_kwDOJHON9s5XGXKp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T05:06:24Z", "updated_at": "2023-03-09T05:06:24Z", "author_association": "MEMBER", "body": "OK, this works!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616354999, "label": "First working version"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/4#issuecomment-1462554175", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4", "id": 1462554175, "node_id": "IC_kwDOJHON9s5XLM4_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T18:19:34Z", "updated_at": "2023-03-09T18:19:34Z", "author_association": "MEMBER", "body": "It looks like the iteration order is most-recently-modified-first - I tried editing a note a bit further back in my notes app and it was the first one output by `apple-notes-to-sqlite --dump`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616429236, "label": "Support incremental updates"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/4#issuecomment-1462556829", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4", "id": 1462556829, "node_id": "IC_kwDOJHON9s5XLNid", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T18:20:56Z", "updated_at": "2023-03-09T18:20:56Z", "author_association": "MEMBER", "body": "In terms of the UI: I'm tempted to say that the default behaviour is for it to run until it sees a note that it already knows about AND that has matching update/created dates, and then stop.\r\n\r\nYou can do a full import again ignoring that logic with `apple-notes-to-sqlite notes.db --full`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1616429236, "label": "Support incremental updates"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462562735", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7", "id": 1462562735, "node_id": "IC_kwDOJHON9s5XLO-v", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T18:23:56Z", "updated_at": "2023-03-09T18:25:22Z", "author_association": "MEMBER", "body": "From the Script Editor library docs:\r\n\r\nA note has a:\r\n\r\n> - `container` (folder), r/o) : the folder of the note\r\n\r\nHere's what a folder looks like:\r\n\r\n> folder\u2002n : a folder containing notes\r\n> elements:\r\n>\r\n> - contains folders, notes; contained by application, accounts, folders.\r\n>\r\n> properties:\r\n>\r\n> - `name` (text) : the name of the folder\r\n> - `id` (text, r/o) : the unique identifier of the folder\r\n> - `shared` (boolean, r/o) : Is the folder shared?\r\n> - `container` (account or folder, r/o) : the container of the folder\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617769847, "label": "Folder support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462564717", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7", "id": 1462564717, "node_id": "IC_kwDOJHON9s5XLPdt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T18:25:39Z", "updated_at": "2023-03-09T18:25:39Z", "author_association": "MEMBER", "body": "So it looks like folders can be hierarchical?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617769847, "label": "Folder support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462570187", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7", "id": 1462570187, "node_id": "IC_kwDOJHON9s5XLQzL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T18:30:24Z", "updated_at": "2023-03-09T18:30:24Z", "author_association": "MEMBER", "body": "I used ChatGPT to write this:\r\n```\r\nosascript -e 'tell application \"Notes\"\r\n set allFolders to folders\r\n repeat with aFolder in allFolders\r\n set folderId to id of aFolder\r\n set folderName to name of aFolder\r\n set folderContainer to container of aFolder\r\n set folderContainerName to name of folderContainer\r\n log \"Folder ID: \" & folderId\r\n log \"Folder Name: \" & folderName\r\n log \"Folder Container: \" & folderContainerName\r\n log \" \"\r\n --check for nested folders\r\n if count of folders of aFolder > 0 then\r\n set nestedFolders to folders of aFolder\r\n repeat with aNestedFolder in nestedFolders\r\n set nestedFolderId to id of aNestedFolder\r\n set nestedFolderName to name of aNestedFolder\r\n set nestedFolderContainer to container of aNestedFolder\r\n set nestedFolderContainerName to name of nestedFolderContainer\r\n log \" Nested Folder ID: \" & nestedFolderId\r\n log \" Nested Folder Name: \" & nestedFolderName\r\n log \" Nested Folder Container: \" & nestedFolderContainerName\r\n log \" \"\r\n end repeat\r\n end if\r\n end repeat\r\nend tell\r\n'\r\n```\r\nWhich for my account output this:\r\n```\r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113\r\nFolder Name: Blog posts\r\nFolder Container: iCloud\r\n \r\n Nested Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995\r\n Nested Folder Name: Nested inside blog posts\r\n Nested Folder Container: Blog posts\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698\r\nFolder Name: JSK\r\nFolder Container: iCloud\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995\r\nFolder Name: Nested inside blog posts\r\nFolder Container: Blog posts\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526\r\nFolder Name: New Folder\r\nFolder Container: iCloud\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839\r\nFolder Name: New Folder 1\r\nFolder Container: iCloud\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2\r\nFolder Name: Notes\r\nFolder Container: iCloud\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059\r\nFolder Name: Quick Notes\r\nFolder Container: iCloud\r\n \r\nFolder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283\r\nFolder Name: UK Christmas 2022\r\nFolder Container: iCloud\r\n```\r\nSo I think the correct approach here is to run code at the start to list all of the folders (no need to do fancy recursion though, just a flat list with the parent containers is enough) and create a model of that hierarchy in SQLite.\r\n\r\nThen when I import notes I can foreign key reference them back to their containing folder.\r\n\r\nI'm tempted to use `rowid` for the foreign keys because the official IDs are pretty long.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617769847, "label": "Folder support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462682795", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7", "id": 1462682795, "node_id": "IC_kwDOJHON9s5XLsSr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T19:52:20Z", "updated_at": "2023-03-09T19:52:44Z", "author_association": "MEMBER", "body": "Created through several rounds with ChatGPT (including hints like \"rewrite that using setdefault()\"):\r\n```python\r\ndef topological_sort(nodes):\r\n children = {}\r\n for node in nodes:\r\n parent_id = node[\"parent\"]\r\n if parent_id is not None:\r\n children.setdefault(parent_id, []).append(node)\r\n\r\n def traverse(node, result):\r\n result.append(node)\r\n if node[\"id\"] in children:\r\n for child in children[node[\"id\"]]:\r\n traverse(child, result)\r\n\r\n sorted_data = []\r\n\r\n for node in nodes:\r\n if node[\"parent\"] is None:\r\n traverse(node, sorted_data)\r\n\r\n return sorted_data\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617769847, "label": "Folder support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462691466", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7", "id": 1462691466, "node_id": "IC_kwDOJHON9s5XLuaK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T19:59:52Z", "updated_at": "2023-03-09T19:59:52Z", "author_association": "MEMBER", "body": "Improved script:\r\n```zsh\r\nosascript -e 'tell application \"Notes\"\r\n set allFolders to folders\r\n repeat with aFolder in allFolders\r\n set folderId to id of aFolder\r\n set folderName to name of aFolder\r\n set folderContainer to container of aFolder\r\n if class of folderContainer is folder then\r\n set folderContainerId to id of folderContainer\r\n else\r\n set folderContainerId to \"\"\r\n end if\r\n log \"ID: \" & folderId\r\n log \"Name: \" & folderName\r\n log \"Container: \" & folderContainerId\r\n log \" \"\r\n end repeat\r\nend tell\r\n'\r\n```\r\n```\r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113\r\nName: Blog posts\r\nContainer: \r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698\r\nName: JSK\r\nContainer: \r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995\r\nName: Nested inside blog posts\r\nContainer: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113\r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526\r\nName: New Folder\r\nContainer: \r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839\r\nName: New Folder 1\r\nContainer: \r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2\r\nName: Notes\r\nContainer: \r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059\r\nName: Quick Notes\r\nContainer: \r\n \r\nID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283\r\nName: UK Christmas 2022\r\nContainer: \r\n```\r\nI filtered out things where the parent was an account and not a folder using `if class of folderContainer is folder then`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617769847, "label": "Folder support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462693867", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7", "id": 1462693867, "node_id": "IC_kwDOJHON9s5XLu_r", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T20:01:39Z", "updated_at": "2023-03-09T20:02:11Z", "author_association": "MEMBER", "body": "My `folders` table will have:\r\n\r\n- `id` - rowid\r\n- `long_id` - that long unique string ID\r\n- `name` - the name\r\n- `parent` - foreign key to `id`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1617769847, "label": "Folder support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/2034#issuecomment-1462921010", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2034", "id": 1462921010, "node_id": "IC_kwDOBm6k_c5XMmcy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T22:34:29Z", "updated_at": "2023-03-09T22:34:29Z", "author_association": "OWNER", "body": "Good catch, thanks.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1613974869, "label": "remove an unused `app` var in cli.py"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/2031#issuecomment-1462921890", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2031", "id": 1462921890, "node_id": "IC_kwDOBm6k_c5XMmqi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T22:35:30Z", "updated_at": "2023-03-09T22:35:30Z", "author_association": "OWNER", "body": "> I've implemented the test (thanks for pointing me in the right direction!).\r\n> \r\n> At [tmcl-it/datasette:0.64.1+row-view-expand-labels](https://github.com/tmcl-it/datasette/tree/0.64.1%2Brow-view-expand-labels) I also have a variant of this patch that applies to the 0.64.x branch. Please let me know if you'd be interested in merging that as well and I'll open another PR.\r\n\r\nSure, let's merge that one too - it can go out in the next `0.64.x` series release (maybe even a 0.65).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1605481359, "label": "Expand foreign key references in row view as well"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462962682", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11", "id": 1462962682, "node_id": "IC_kwDOJHON9s5XMwn6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T23:20:35Z", "updated_at": "2023-03-09T23:22:41Z", "author_association": "MEMBER", "body": "Here's a query that returns all notes in folder 1, including notes in descendant folders:\r\n```sql\r\nwith recursive nested_folders(folder_id, descendant_folder_id) as (\r\n -- base case: select all immediate children of the root folder\r\n select id, id from folders where parent is null\r\n union all\r\n -- recursive case: select all children of the previous level of nested folders\r\n select nf.folder_id, f.id from nested_folders nf\r\n join folders f on nf.descendant_folder_id = f.parent\r\n)\r\n-- Find notes within all descendants of folder 1\r\nselect *\r\nfrom notes\r\nwhere folder in (\r\n select descendant_folder_id from nested_folders where folder_id = 1\r\n);\r\n```\r\nWith assistance from ChatGPT. Prompts were:\r\n\r\n```\r\nSQLite schema:\r\n\r\nCREATE TABLE [folders] (\r\n [id] INTEGER PRIMARY KEY,\r\n [long_id] TEXT,\r\n [name] TEXT,\r\n [parent] INTEGER,\r\n FOREIGN KEY([parent]) REFERENCES [folders]([id])\r\n);\r\n\r\nWrite a recursive CTE that returns the following:\r\n\r\nfolder_id | descendant_folder_id\r\n\r\nWith a row for every nested child of every folder - so the top level folder has lots of rows\r\n```\r\nThen I tweaked it a bit, then ran this:\r\n```\r\nWITH RECURSIVE nested_folders(folder_id, descendant_folder_id) AS (\r\n -- base case: select all immediate children of the root folder\r\n SELECT id, id FROM folders WHERE parent IS NULL\r\n UNION ALL\r\n -- recursive case: select all children of the previous level of nested folders\r\n SELECT nf.folder_id, f.id FROM nested_folders nf\r\n JOIN folders f ON nf.descendant_folder_id = f.parent\r\n)\r\n-- select all rows from the recursive CTE\r\nSELECT * from notes where folder in (select descendant_folder_id FROM nested_folders where folder_id = 1)\r\n\r\nConvert all SQL keywords to lower case, and re-indent\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1618130434, "label": "Implement a SQL view to make it easier to query files in a nested folder"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462965256", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11", "id": 1462965256, "node_id": "IC_kwDOJHON9s5XMxQI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T23:22:12Z", "updated_at": "2023-03-09T23:22:12Z", "author_association": "MEMBER", "body": "Here's what the CTE from that looks like:\r\n\r\n![\"image\"](\"https://user-images.githubusercontent.com/9599/224182888-50f315eb-417f-4fb1-b624-151e1740ffed.png\")
\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1618130434, "label": "Implement a SQL view to make it easier to query files in a nested folder"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462968053", "issue_url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11", "id": 1462968053, "node_id": "IC_kwDOJHON9s5XMx71", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T23:24:01Z", "updated_at": "2023-03-09T23:24:01Z", "author_association": "MEMBER", "body": "I improved the readability by removing some unnecessary table aliases:\r\n```sql\r\nwith recursive nested_folders(folder_id, descendant_folder_id) as (\r\n -- base case: select all immediate children of the root folder\r\n select id, id from folders where parent is null\r\n union all\r\n -- recursive case: select all children of the previous level of nested folders\r\n select nested_folders.folder_id, folders.id from nested_folders\r\n join folders on nested_folders.descendant_folder_id = folders.parent\r\n)\r\n-- Find notes within all descendants of folder 1\r\nselect *\r\nfrom notes\r\nwhere folder in (\r\n select descendant_folder_id from nested_folders where folder_id = 1\r\n);\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1618130434, "label": "Implement a SQL view to make it easier to query files in a nested folder"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1999#issuecomment-1462997800", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1999", "id": 1462997800, "node_id": "IC_kwDOBm6k_c5XM5Mo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T23:39:47Z", "updated_at": "2023-03-09T23:39:47Z", "author_association": "OWNER", "body": "Found a neat trick:\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 186f192d..40416713 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -23,7 +23,13 @@ from pathlib import Path\r\n \r\n from markupsafe import Markup, escape\r\n from itsdangerous import URLSafeSerializer\r\n-from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader\r\n+from jinja2 import (\r\n+ ChoiceLoader,\r\n+ Environment,\r\n+ FileSystemLoader,\r\n+ PrefixLoader,\r\n+ StrictUndefined,\r\n+)\r\n from jinja2.environment import Template\r\n from jinja2.exceptions import TemplateNotFound\r\n \r\n@@ -394,7 +400,10 @@ class Datasette:\r\n ]\r\n )\r\n self.jinja_env = Environment(\r\n- loader=template_loader, autoescape=True, enable_async=True\r\n+ loader=template_loader,\r\n+ autoescape=True,\r\n+ enable_async=True,\r\n+ undefined=StrictUndefined,\r\n )\r\n self.jinja_env.filters[\"escape_css_string\"] = escape_css_string\r\n self.jinja_env.filters[\"quote_plus\"] = urllib.parse.quote_plus\r\n```\r\nThis causes Jinja to raise a hard error if there are any variables referenced in the template that are not available in the context.\r\n\r\nIt's helping me spot things that are still missing, rather than just relying on failed unit tests.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1551694938, "label": "?_extra= support (draft)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1999#issuecomment-1463005744", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1999", "id": 1463005744, "node_id": "IC_kwDOBm6k_c5XM7Iw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-09T23:52:15Z", "updated_at": "2023-03-09T23:52:23Z", "author_association": "OWNER", "body": "I need to figure out what to do about `extra_context_from_filters` - which was previously passed straight to the HTML context.\r\n\r\nhttps://github.com/simonw/datasette/blob/11f7feb7a3f7166c71389786880863d60ed3d165/datasette/views/table.py#L406-L422", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1551694938, "label": "?_extra= support (draft)"}, "performed_via_github_app": null}