{"html_url": "https://github.com/simonw/datasette/issues/2104#issuecomment-1641082395", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2104", "id": 1641082395, "node_id": "IC_kwDOBm6k_c5h0O4b", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-07-18T22:41:37Z", "updated_at": "2023-07-18T22:41:37Z", "author_association": "CONTRIBUTOR", "body": "For filtering virtual table's \"shadow tables\" (ex the FTS5 _content and most the spatialite tables), you can use `pragma_table_list` (first appeared in SQLite 3.37 (2021-11-27), which has a `type` column that calls out `type=\"shadow\"` tables https://www.sqlite.org/pragma.html#pragma_table_list", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1808215339, "label": "Tables starting with an underscore should be treated as hidden"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1640826795", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/433", "id": 1640826795, "node_id": "IC_kwDOCGYnMM5hzQer", "user": {"value": 76528036, "label": "J450n-4-W"}, "created_at": "2023-07-18T19:08:50Z", "updated_at": "2023-07-18T19:08:50Z", "author_association": "NONE", "body": "Came here to report this, but instead I'll confirm the issue across two terminal emulators (Gnome Terminal and Alacritty) on Pop_OS! 22.04 (currently based on Ubuntu/Gnome). Also messes up the formatting of the terminal. Can also confirm that reset fixes it until the next sqlite-utils command. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239034903, "label": "CLI eats my cursor"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2102#issuecomment-1640064620", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2102", "id": 1640064620, "node_id": "IC_kwDOBm6k_c5hwWZs", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-18T11:47:21Z", "updated_at": "2023-07-18T11:47:21Z", "author_association": "OWNER", "body": "I think I've figured out the problem here.\r\n\r\nThe question being asked is \"can this actor access this resource, which is within this database within this instance\".\r\n\r\nThe answer to this question needs to consider the full set of questions at once - yes they can access within this instance IF they have access to the specified table and that's the table being asked about.\r\n\r\nBut the questions are currently being asked independently, which means the plugin hook acting on `view-instance` can't see that the answer here should be yes because it's actually about a table that the actor has explicit permission to view.\r\n\r\nSo I think I may need to redesign the plugin hook to always see the full hierarchy of checks, not just a single check at a time.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1805076818, "label": "API tokens with view-table but not view-database/view-instance cannot access the table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638926655", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/567", "id": 1638926655, "node_id": "IC_kwDOCGYnMM5hsAk_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-17T21:42:37Z", "updated_at": "2023-07-17T21:42:37Z", "author_association": "OWNER", "body": "I really like this. I'm also interested in:\r\n- Plugins that make new custom SQL functions available - similar to this Datasette hook: https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette\r\n- Plugins that register functions that can be used as recipes for `sqlite-utils convert` https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes\r\n\r\nThe upload-data-to-Datasette problem is planned to be solved by a future version of https://github.com/simonw/dclient ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1801394744, "label": "Plugin system"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638910473", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/567", "id": 1638910473, "node_id": "IC_kwDOCGYnMM5hr8oJ", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-07-17T21:27:41Z", "updated_at": "2023-07-17T21:27:41Z", "author_association": "NONE", "body": "Another use-case: I want to make a `sqlite-utils` plugin that'll help me insert data into Datasette. \r\n\r\n```bash\r\nsqlite-utils insert-datasette \\\r\n --token $DATASETTE_API_KEY \\\r\n https://latest.datasette.io/fixtures/my-table \\\r\n 'select ...'\r\n```\r\n\r\n\r\nThis could also be a datasette plugin (ex `datasette upload-data ...`, but you can also think of `sqlite-utils` plugins that upload to S3, a postgres table, other DBMS's, etc.)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1801394744, "label": "Plugin system"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2102#issuecomment-1638567228", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2102", "id": 1638567228, "node_id": "IC_kwDOBm6k_c5hqo08", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-17T17:24:19Z", "updated_at": "2023-07-17T17:25:12Z", "author_association": "OWNER", "body": "Confirmed that this is an issue with regular Datasette signed tokens as well. I created one on https://latest.datasette.io/-/create-token with these details:\r\n```json\r\n{\r\n \"_r\": {\r\n \"r\": {\r\n \"fixtures\": {\r\n \"sortable\": [\r\n \"vt\"\r\n ]\r\n }\r\n }\r\n },\r\n \"a\": \"root\",\r\n \"d\": 3600,\r\n \"t\": 1689614483\r\n}\r\n```\r\nRun like this:\r\n```\r\ncurl -H 'Authorization: Bearer dstok_eyJhIjoicm9vdCIsInQiOjE2ODk2MTQ0ODMsImQiOjM2MDAsIl9yIjp7InIiOnsiZml4dHVyZXMiOnsic29ydGFibGUiOlsidnQiXX19fX0.n-VGxxawz1Q0WK7sqLfhXUgcvY0' \\\r\n https://latest.datasette.io/fixtures/sortable.json\r\n```\r\nReturned an HTML Forbidden page:\r\n```html\r\n\r\n\r\n\r\n Forbidden\r\n \r\n...\r\n```\r\nSame token againts `/-/actor.json` returns:\r\n```json\r\n{\r\n \"actor\": {\r\n \"id\": \"root\",\r\n \"token\": \"dstok\",\r\n \"_r\": {\r\n \"r\": {\r\n \"fixtures\": {\r\n \"sortable\": [\r\n \"vt\"\r\n ]\r\n }\r\n }\r\n },\r\n \"token_expires\": 1689618083\r\n }\r\n}\r\n```\r\nReminder - `\"_r\"` means restrict, `\"r\"` means resource.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1805076818, "label": "API tokens with view-table but not view-database/view-instance cannot access the table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2104#issuecomment-1638552567", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2104", "id": 1638552567, "node_id": "IC_kwDOBm6k_c5hqlP3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-17T17:14:20Z", "updated_at": "2023-07-17T17:14:20Z", "author_association": "OWNER", "body": "Relevant code: https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/database.py#L391-L451", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1808215339, "label": "Tables starting with an underscore should be treated as hidden"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/670#issuecomment-1637293044", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/670", "id": 1637293044, "node_id": "IC_kwDOBm6k_c5hlxv0", "user": {"value": 4863782, "label": "yairlenga"}, "created_at": "2023-07-17T02:23:32Z", "updated_at": "2023-07-17T02:23:32Z", "author_association": "NONE", "body": "Is there any working version of datasette/postgresql ?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564833696, "label": "Prototoype for Datasette on PostgreSQL"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2087#issuecomment-1636134091", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2087", "id": 1636134091, "node_id": "IC_kwDOBm6k_c5hhWzL", "user": {"value": 653549, "label": "adarshp"}, "created_at": "2023-07-14T17:02:03Z", "updated_at": "2023-07-14T17:02:03Z", "author_association": "NONE", "body": "@asg017 - the docs say that the autodetection only occurs in configuration directory mode. I for one would also be interested in the `--settings settings.json` feature. \r\n\r\nFor context, I am developing a large database for use with Datasette, but the database lives in a different network volume than my source code, since the volume in which my source code lives is aggressively backed up, while the location where the database lives is meant for temporary files and is not as aggressively backed up (since the backups would get unreasonably large).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1765870617, "label": "`--settings settings.json` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2102#issuecomment-1636093730", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2102", "id": 1636093730, "node_id": "IC_kwDOBm6k_c5hhM8i", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-14T16:26:27Z", "updated_at": "2023-07-14T16:32:49Z", "author_association": "OWNER", "body": "Here's that crucial comment:\r\n\r\n> If _r is defined then we use those to further restrict the actor.\r\n>\r\n>Crucially, we only use this to say NO (return False) - we never use it to return YES (True) because that might over-ride other restrictions placed on this actor\r\n\r\nSo that's why I implemented it like this.\r\n\r\nThe goal here is to be able to issue a token which can't do anything _more_ than the actor it is associated with, but CAN be configured to do less.\r\n\r\nSo I think the solution here is for the `_r` checking code to perhaps implement its own view cascade logic - it notices if you have `view-table` and consequently fails to block `view-table` and `view-instance`.\r\n\r\nI'm not sure that's going to work though - would that mean that granting `view-table` grants `view-database` in a surprising and harmful way?\r\n\r\nMaybe that's OK: if you have `view-database` but permission checks fail for individual tables and queries you shouldn't be able to see a thing that you shouldn't. Need to verify that though.\r\n\r\nAlso, do `Permission` instances have enough information to implement this kind of cascade without hard-coding anything? \r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1805076818, "label": "API tokens with view-table but not view-database/view-instance cannot access the table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2102#issuecomment-1636053060", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2102", "id": 1636053060, "node_id": "IC_kwDOBm6k_c5hhDBE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-14T15:51:36Z", "updated_at": "2023-07-14T16:14:05Z", "author_association": "OWNER", "body": "This might only be an issue with the code that checks `_r` on actors.\r\n\r\nhttps://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/default_permissions.py#L185-L222\r\n\r\nAdded in https://github.com/simonw/datasette/commit/bcc781f4c50a8870e3389c4e60acb625c34b0317 - refs:\r\n\r\n- #1855 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1805076818, "label": "API tokens with view-table but not view-database/view-instance cannot access the table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2102#issuecomment-1636042066", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2102", "id": 1636042066, "node_id": "IC_kwDOBm6k_c5hhAVS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-14T15:41:54Z", "updated_at": "2023-07-14T15:42:32Z", "author_association": "OWNER", "body": "I tried some code spelunking and came across https://github.com/simonw/datasette/commit/d6e03b04302a0852e7133dc030eab50177c37be7 which says:\r\n\r\n> - If you have table permission but not database permission you can now view the table page\r\n\r\nRefs:\r\n- #832 \r\n\r\nWhich suggests that my initial design decision wasn't what appears to be implemented today.\r\n\r\nNeeds more investigation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1805076818, "label": "API tokens with view-table but not view-database/view-instance cannot access the table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2102#issuecomment-1636040164", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2102", "id": 1636040164, "node_id": "IC_kwDOBm6k_c5hg_3k", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-14T15:40:21Z", "updated_at": "2023-07-14T15:40:21Z", "author_association": "OWNER", "body": "Relevant code: \r\nhttps://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/app.py#L822-L855", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1805076818, "label": "API tokens with view-table but not view-database/view-instance cannot access the table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2102#issuecomment-1636036312", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2102", "id": 1636036312, "node_id": "IC_kwDOBm6k_c5hg-7Y", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-14T15:37:14Z", "updated_at": "2023-07-14T15:37:14Z", "author_association": "OWNER", "body": "I think I made this decision because I was thinking about default deny: obviously if a user has been denied access to a database. It doesn't make sense that they could access tables within it.\r\n\r\nBut now that I am spending more time with authentication tokens, which default to denying everything, except for the things that you have explicitly listed, this policy, no longer makes as much sense.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1805076818, "label": "API tokens with view-table but not view-database/view-instance cannot access the table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2101#issuecomment-1634443907", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2101", "id": 1634443907, "node_id": "IC_kwDOBm6k_c5ha6KD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-13T15:24:17Z", "updated_at": "2023-07-13T15:24:17Z", "author_association": "OWNER", "body": "https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/views/table.py#L486-L506", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1803264272, "label": "alter: true support for JSON write API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1632867333", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1632867333, "node_id": "IC_kwDOBm6k_c5hU5QF", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-07-12T16:38:27Z", "updated_at": "2023-07-12T16:38:27Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage has no change and project coverage change: **`-0.02`** :warning:\n> Comparison is base [(`3feed1f`)](https://app.codecov.io/gh/simonw/datasette/commit/3feed1f66e2b746f349ee56970a62246a18bb164?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.46% compared to head [(`cf5a9df`)](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.45%.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #2052 +/- ##\n==========================================\n- Coverage 92.46% 92.45% -0.02% \n==========================================\n Files 38 39 +1 \n Lines 5750 5802 +52 \n==========================================\n+ Hits 5317 5364 +47 \n- Misses 433 438 +5 \n```\n\n\n[see 6 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2052/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1630776144", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1630776144, "node_id": "IC_kwDOBm6k_c5hM6tQ", "user": {"value": 9020979, "label": "hydrosquall"}, "created_at": "2023-07-11T12:54:03Z", "updated_at": "2023-07-11T12:54:03Z", "author_association": "NONE", "body": "Thanks for the review and the code pointers @simonw - I've made the suggested edits, fixed the renamed variable, and confirmed that the panels still render on the `table` and `database` views. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1629337927", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1629337927, "node_id": "IC_kwDOBm6k_c5hHblH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-10T16:43:38Z", "updated_at": "2023-07-10T16:44:23Z", "author_association": "OWNER", "body": "I tried running this locally just now. I made one edit:\r\n```diff\r\ndiff --git a/demos/plugins/example_js_manager_plugins.py b/demos/plugins/example_js_manager_plugins.py\r\nindex 7bdb9f3f..f9dfa8e6 100644\r\n--- a/demos/plugins/example_js_manager_plugins.py\r\n+++ b/demos/plugins/example_js_manager_plugins.py\r\n@@ -15,6 +15,6 @@ def extra_js_urls(view_name):\r\n if view_name in PERMITTED_VIEWS:\r\n return [\r\n {\r\n- \"url\": f\"/-/demos/plugins/static/table-example-plugins.js\",\r\n+ \"url\": f\"/static/table-example-plugins.js\",\r\n }\r\n ]\r\n```\r\nAnd then started it running like this:\r\n```bash\r\nwget https://datasette.io/content.db\r\n```\r\n```bash\r\ndatasette content.db \\\r\n --plugins-dir demos/plugins \\\r\n --static static:datasette/demos/plugins/static\r\n```\r\n\r\nIt didn't quite work for me - I got this error on a table page:\r\n\r\n\"image\"\r\n\r\nAnd this error on a query page:\r\n\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1629123734", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14", "id": 1629123734, "node_id": "IC_kwDOC8tyDs5hGnSW", "user": {"value": 44622670, "label": "philipp-heinrich"}, "created_at": "2023-07-10T14:46:52Z", "updated_at": "2023-07-10T14:46:52Z", "author_association": "NONE", "body": "@simonw any chance to get this fixed soon? ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771608692, "label": "UNIQUE constraint failed: workouts.id"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627598570", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/566", "id": 1627598570, "node_id": "IC_kwDOCGYnMM5hAy7q", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-09T04:13:34Z", "updated_at": "2023-07-09T04:13:34Z", "author_association": "OWNER", "body": "On consulting https://pypi.org/project/tabulate/ it looks like most of those formats don't actually makes sense without headers - so the right thing here might be to raise an error if `--fmt` and `--no-headers` are used at the same time.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1795219865, "label": "`--no-headers` doesn't work on most formats"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627597872", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/566", "id": 1627597872, "node_id": "IC_kwDOCGYnMM5hAyww", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-09T04:09:56Z", "updated_at": "2023-07-09T04:09:56Z", "author_association": "OWNER", "body": "Thanks, looks like a bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1795219865, "label": "`--no-headers` doesn't work on most formats"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627564127", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12", "id": 1627564127, "node_id": "IC_kwDODLZ_YM5hAqhf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-09T01:19:42Z", "updated_at": "2023-07-09T01:19:42Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/pocket-to-sqlite/tree/0.2.3 and https://pypi.org/project/pocket-to-sqlite/0.2.3/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1795187493, "label": "Switch to pyproject.toml"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627563202", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12", "id": 1627563202, "node_id": "IC_kwDODLZ_YM5hAqTC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-09T01:14:27Z", "updated_at": "2023-07-09T01:14:27Z", "author_association": "MEMBER", "body": "I tested this locally with `python -m build` and then `pip install ...whl` in a fresh virtual environment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1795187493, "label": "Switch to pyproject.toml"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-1627480353", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 1627480353, "node_id": "IC_kwDOBm6k_c5hAWEh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-08T20:09:48Z", "updated_at": "2023-07-08T20:09:48Z", "author_association": "OWNER", "body": "https://docs.datasette.io/en/latest/writing_plugins.html#writing-plugins-that-accept-configuration is fixed now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-1627478910", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 1627478910, "node_id": "IC_kwDOBm6k_c5hAVt-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-08T20:01:19Z", "updated_at": "2023-07-08T20:01:19Z", "author_association": "OWNER", "body": "Some examples:\r\n\r\n- https://docs.datasette.io/en/latest/sql_queries.html#canned-queries\r\n- https://docs.datasette.io/en/latest/sql_queries.html#canned-query-parameters\r\n- https://docs.datasette.io/en/latest/authentication.html#access-to-an-instance\r\n- https://docs.datasette.io/en/latest/facets.html#facets-in-metadata\r\n- https://docs.datasette.io/en/latest/full_text_search.html#configuring-full-text-search-for-a-table-or-view\r\n- https://docs.datasette.io/en/latest/metadata.html\r\n- https://docs.datasette.io/en/latest/custom_templates.html#custom-css-and-javascript\r\n- https://docs.datasette.io/en/latest/plugins.html#plugin-configuration\r\n\r\nI need to fix this section: https://docs.datasette.io/en/latest/writing_plugins.html#writing-plugins-that-accept-configuration", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-1627455892", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 1627455892, "node_id": "IC_kwDOBm6k_c5hAQGU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-08T18:39:19Z", "updated_at": "2023-07-08T18:39:19Z", "author_association": "OWNER", "body": "```\r\nERROR: Could not find a version that satisfies the requirement Sphinx==6.1.3; extra == \"docs\" (from datasette[docs,test]) (from versions: 0.1.61611, 0.1.61798, 0.1.61843, 0.1.61945, 0.1.61950, 0.2, 0.3, 0.4, 0.4.1, 0.4.2, 0.4.3, 0.5, 0.5.1, 0.5.2b1, 0.5.2, 0.6b1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.6.7, 1.0b1, 1.0b2, 1.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.0.8, 1.1, 1.1.1, 1.1.2, 1.1.3, 1.2b1, 1.2b2, 1.2b3, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.3b1, 1.3b2, 1.3b3, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.4a1, 1.4b1, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.5a1, 1.5a2, 1.5b1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.6b1, 1.6b2, 1.6b3, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5, 1.6.6, 1.6.7, 1.7.0b1, 1.7.0b2, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6, 1.7.7, 1.7.8, 1.7.9, 1.8.0b1, 1.8.0, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.8.5, 1.8.6, 2.0.0b1, 2.0.0b2, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 3.0.0b1, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.2.1, 3.3.0, 3.3.1, 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.5.0, 3.5.1, 3.5.2, 3.5.3, 3.5.4, 4.0.0b1, 4.0.0b2, 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.1.0, 4.1.1, 4.1.2, 4.2.0, 4.3.0, 4.3.1, 4.3.2, 4.4.0, 4.5.0, 5.0.0b1, 5.0.0, 5.0.1, 5.0.2, 5.1.0, 5.1.1, 5.2.0, 5.2.0.post0, 5.2.1, 5.2.2, 5.2.3, 5.3.0)\r\nERROR: No matching distribution found for Sphinx==6.1.3; extra == \"docs\"\r\n```\r\nI'm going to drop Python 3.7.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-1627451646", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 1627451646, "node_id": "IC_kwDOBm6k_c5hAPD-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-08T18:21:24Z", "updated_at": "2023-07-08T18:21:24Z", "author_association": "OWNER", "body": "This one was tricky:\r\n\r\n\"image\"\r\n\r\nI wanted complete control over the YAML example here, so I could ensure it used multi-line strings correctly.\r\n\r\nI ended up changing my cog helper function to this:\r\n\r\n```python\r\nimport json\r\nimport textwrap\r\nfrom yaml import safe_dump\r\nfrom ruamel.yaml import round_trip_load\r\n\r\n\r\ndef metadata_example(cog, data=None, yaml=None):\r\n assert data or yaml, \"Must provide data= or yaml=\"\r\n assert not (data and yaml), \"Cannot use data= and yaml=\"\r\n output_yaml = None\r\n if yaml:\r\n # dedent it first\r\n yaml = textwrap.dedent(yaml).strip()\r\n # round_trip_load to preserve key order:\r\n data = round_trip_load(yaml)\r\n output_yaml = yaml\r\n else:\r\n output_yaml = safe_dump(data, sort_keys=False)\r\n cog.out(\"\\n.. tab:: YAML\\n\\n\")\r\n cog.out(\" .. code-block:: yaml\\n\\n\")\r\n cog.out(textwrap.indent(output_yaml, \" \"))\r\n cog.out(\"\\n\\n.. tab:: JSON\\n\\n\")\r\n cog.out(\" .. code-block:: json\\n\\n\")\r\n cog.out(textwrap.indent(json.dumps(data, indent=2), \" \"))\r\n cog.out(\"\\n\")\r\n```\r\nThis allows me to call it ith YAML in some places:\r\n```\r\n.. [[[cog\r\n metadata_example(cog, yaml=\"\"\"\r\n databases:\r\n fixtures:\r\n queries:\r\n neighborhood_search:\r\n fragment: fragment-goes-here\r\n hide_sql: true\r\n sql: |-\r\n select neighborhood, facet_cities.name, state\r\n from facetable join facet_cities on facetable.city_id = facet_cities.id\r\n where neighborhood like '%' || :text || '%' order by neighborhood;\r\n \"\"\")\r\n.. ]]]\r\n```\r\nI had to introduce https://pypi.org/project/ruamel.yaml/ as a dependency here in order to load YAML from disk while maintaining key order.\r\n\r\nI'm still using `safe_dump(data, sort_keys=False)` from PyYAML as I couldn't get the result I wanted for outputting YAML from an input of JSON using PyYAML.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-1627450852", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 1627450852, "node_id": "IC_kwDOBm6k_c5hAO3k", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-08T18:17:35Z", "updated_at": "2023-07-08T18:17:35Z", "author_association": "OWNER", "body": "I figured out a workaround:\r\n```python\r\nextensions = [\r\n \"sphinx.ext.extlinks\",\r\n \"sphinx.ext.autodoc\",\r\n \"sphinx_copybutton\",\r\n]\r\nif not os.environ.get(\"DISABLE_SPHINX_INLINE_TABS\"):\r\n extensions += [\"sphinx_inline_tabs\"]\r\n```\r\nThat way I can run `sphinx-build -b xml . _build` successfully if I set that environment variable.\r\n\r\nI get some noisy warnings, but it runs OK. And the resulting `docs.db` file has rows like this, which I think are fine:\r\n\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-1627448542", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 1627448542, "node_id": "IC_kwDOBm6k_c5hAOTe", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-08T18:05:44Z", "updated_at": "2023-07-08T18:05:44Z", "author_association": "OWNER", "body": "Running with `-P` opens a debugger when it hits the error:\r\n```bash\r\nsphinx-build -P -b xml . _build\r\n```\r\n```\r\n(Pdb) list\r\n2023 \t\r\n2024 \t Raise an exception unless overridden.\r\n2025 \t \"\"\"\r\n2026 \t if (self.document.settings.strict_visitor\r\n2027 \t or node.__class__.__name__ not in self.optional):\r\n2028 ->\t raise NotImplementedError(\r\n2029 \t '%s visiting unknown node type: %s'\r\n2030 \t % (self.__class__, node.__class__.__name__))\r\n2031 \t\r\n2032 \t def unknown_departure(self, node):\r\n2033 \t \"\"\"\r\n(Pdb) self.optional\r\n('meta',)\r\n(Pdb) node.__class__.__name__\r\n'TabContainer'\r\n(Pdb) self.document.settings.strict_visitor\r\n(Pdb) type(self.document.settings.strict_visitor)\r\n\r\n```\r\nSo if I can get `TabContainer` into that `self.optional` list I'll have fixed this problem.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-1627448180", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 1627448180, "node_id": "IC_kwDOBm6k_c5hAON0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-08T18:03:31Z", "updated_at": "2023-07-08T18:03:31Z", "author_association": "OWNER", "body": "Relevant code: https://github.com/docutils/docutils/blob/3b53ded52bc439d8068b6ecb20ea0a761247e479/docutils/docutils/nodes.py#L2021-L2031\r\n\r\n```python\r\n def unknown_visit(self, node):\r\n \"\"\"\r\n Called when entering unknown `Node` types.\r\n\r\n Raise an exception unless overridden.\r\n \"\"\"\r\n if (self.document.settings.strict_visitor\r\n or node.__class__.__name__ not in self.optional):\r\n raise NotImplementedError(\r\n '%s visiting unknown node type: %s'\r\n % (self.__class__, node.__class__.__name__))\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-1627447750", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 1627447750, "node_id": "IC_kwDOBm6k_c5hAOHG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-08T18:00:56Z", "updated_at": "2023-07-08T18:00:56Z", "author_association": "OWNER", "body": "Actually no it's in `sphinx-build`:\r\n```\r\n% sphinx-build -b xml . _build\r\nRunning Sphinx v6.1.3\r\nbuilding [mo]: targets for 0 po files that are out of date\r\nwriting output... \r\nbuilding [xml]: targets for 28 source files that are out of date\r\nupdating environment: [new config] 28 added, 0 changed, 0 removed\r\nreading sources... [100%] writing_plugins \r\nlooking for now-outdated files... none found\r\npickling environment... done\r\nchecking consistency... done\r\npreparing documents... done\r\nwriting output... [ 3%] authentication \r\nException occurred:\r\n File \"/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/docutils/nodes.py\", line 2028, in unknown_visit\r\n raise NotImplementedError(\r\nNotImplementedError: visiting unknown node type: TabContainer\r\nThe full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1wkxmkji.log, if you want to report the issue to the developers.\r\nPlease also report this if it was a user error, so that a better error message can be provided next time.\r\nA bug report can be filed in the tracker at . Thanks!\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-1627447478", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 1627447478, "node_id": "IC_kwDOBm6k_c5hAOC2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-08T17:59:25Z", "updated_at": "2023-07-08T17:59:25Z", "author_association": "OWNER", "body": "Hit a problem:\r\n```\r\nException occurred:\r\n File \"/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/docutils/nodes.py\", line 2028, in unknown_visit\r\n raise NotImplementedError(\r\nNotImplementedError: visiting unknown node type: TabContainer\r\nThe full traceback has been saved in /tmp/sphinx-err-tfujyw1h.log, if you want to report the issue to the developers.\r\nPlease also report this if it was a user error, so that a better error message can be provided next time.\r\nA bug report can be filed in the tracker at . Thanks!\r\n```\r\nThat's happening here: https://github.com/simonw/datasette/blob/0183e1a72d4d93b1d9a9363f4d47fcc0b5d5849c/.github/workflows/deploy-latest.yml#L42-L48\r\n\r\nMy https://github.com/simonw/sphinx-to-sqlite tool can't handle the new `TabContainer` elements introduced by `sphinx-inline-tabs`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-1627396658", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 1627396658, "node_id": "IC_kwDOBm6k_c5hABoy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-08T16:40:07Z", "updated_at": "2023-07-08T16:40:07Z", "author_association": "OWNER", "body": "https://docs.datasette.io/en/latest/metadata.html\r\n\r\n![inline-tabs](https://github.com/simonw/datasette/assets/9599/975bdff5-74ac-451e-92c3-a7dd05d4b862)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-1627396450", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 1627396450, "node_id": "IC_kwDOBm6k_c5hABli", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-08T16:38:58Z", "updated_at": "2023-07-08T16:38:58Z", "author_association": "OWNER", "body": "I'm using `cog` and this utility function to generate the YAML/JSON tabs:\r\n\r\nhttps://github.com/simonw/datasette/blob/3b336d8071fb5707bd006de1d614f701d20246a3/docs/metadata_doc.py#L1-L13\r\n\r\nExample usage:\r\n\r\nhttps://github.com/simonw/datasette/blob/3b336d8071fb5707bd006de1d614f701d20246a3/docs/metadata.rst?plain=1#L17-L53", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1153#issuecomment-1627395947", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1153", "id": 1627395947, "node_id": "IC_kwDOBm6k_c5hABdr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-08T16:35:45Z", "updated_at": "2023-07-08T16:35:45Z", "author_association": "OWNER", "body": "I was inspired to finally address this after seeing `sphinx-inline-tabs` at work in https://webcolors.readthedocs.io/en/latest/install.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771202454, "label": "Use YAML examples in documentation by default, not JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1618380888", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/565", "id": 1618380888, "node_id": "IC_kwDOCGYnMM5gdohY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-03T14:09:11Z", "updated_at": "2023-07-03T14:09:31Z", "author_association": "OWNER", "body": "For the CLI:\r\n\r\n```bash\r\nsqlite-utils rename-table data.db old_table_name new_table_name\r\n```\r\n\r\nFor the Python code, should it go on Table or on Database?\r\n```python\r\ndb[\"foo\"].rename_table(\"bar\")\r\n\r\ndb.rename_table(\"foo\", \"bar\")\r\n```\r\nI think I like the second better, it's slightly more clear.\r\n\r\nAlso need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1786258502, "label": "Table renaming utilities"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/563#issuecomment-1617395444", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/563", "id": 1617395444, "node_id": "IC_kwDOCGYnMM5gZ370", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-03T05:44:43Z", "updated_at": "2023-07-03T05:44:43Z", "author_association": "OWNER", "body": "Documentation at the bottom of this section: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-csv-or-tsv-data", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1785360409, "label": "`--empty-null` option when importing CSV"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2087#issuecomment-1616853644", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2087", "id": 1616853644, "node_id": "IC_kwDOBm6k_c5gXzqM", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-07-02T22:00:48Z", "updated_at": "2023-07-02T22:00:48Z", "author_association": "CONTRIBUTOR", "body": "I just saw in the docs that Dasette auto-detects `settings.json`:\r\n\r\n> settings.json - settings that would normally be passed using --setting - here they should be stored as a JSON object of key/value pairs\r\n> [*Source*](https://docs.datasette.io/en/stable/settings.html#:~:text=settings.json%20%2D%20settings%20that%20would%20normally%20be%20passed%20using%20%2D%2Dsetting%20%2D%20here%20they%20should%20be%20stored%20as%20a%20JSON%20object%20of%20key/value%20pairs)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1765870617, "label": "`--settings settings.json` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/562#issuecomment-1616782404", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/562", "id": 1616782404, "node_id": "IC_kwDOCGYnMM5gXiRE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-02T19:24:14Z", "updated_at": "2023-07-02T19:26:39Z", "author_association": "OWNER", "body": "[Dataclasses](https://docs.python.org/3/library/dataclasses.html) were added in Python 3.7 and `sqlite-utils` was originally written for Python 3.6 - but both 3.6 and 3.7 are EOL now.\r\n\r\nThe thing that makes Dataclasses particularly interesting is the potential to use type annotations with them to help specify the types of the related SQLite columns.\r\n\r\nExample for https://datasette.io/content/users\r\n\r\n```sql\r\nCREATE TABLE [users] (\r\n [login] TEXT,\r\n [id] INTEGER PRIMARY KEY,\r\n [node_id] TEXT,\r\n [avatar_url] TEXT,\r\n [gravatar_id] TEXT,\r\n [html_url] TEXT,\r\n [type] TEXT,\r\n [site_admin] INTEGER,\r\n [name] TEXT\r\n);\r\n```\r\nAnd the dataclass:\r\n```python\r\nfrom dataclasses import dataclass\r\n\r\n@dataclass\r\nclass User:\r\n id: int\r\n login: str\r\n node_id: str\r\n avatar_url: str\r\n gravatar_id: str\r\n html_url: str\r\n type: str\r\n site_admin: int\r\n name: str\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1784794489, "label": "Explore the intersection between sqlite-utils and dataclasses"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2093#issuecomment-1616286848", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2093", "id": 1616286848, "node_id": "IC_kwDOBm6k_c5gVpSA", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-07-02T02:17:46Z", "updated_at": "2023-07-02T02:17:46Z", "author_association": "CONTRIBUTOR", "body": "Storing metadata in the database won't be required. I imagine there'll be many different ways to store metadata, including any possible `datasette_metadata` or sqlite-docs, or the older metadata.json way. \r\n\r\nThe next question will be how precedence should work - i'd imagine metadata.json > plugins > datasette_metadata > sqlite-docs", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781530343, "label": "Proposal: Combine settings, metadata, static, etc. into a single `datasette.toml` File"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2093#issuecomment-1616195496", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2093", "id": 1616195496, "node_id": "IC_kwDOBm6k_c5gVS-o", "user": {"value": 273509, "label": "terinjokes"}, "created_at": "2023-07-02T00:06:54Z", "updated_at": "2023-07-02T00:07:17Z", "author_association": "NONE", "body": "I'm not keen on requiring metadata to be within the database. I commonly have multiple DBs, from various sources, and having one config file to provide the metadata works out very well. I use Datasette with databases where I'm not the original source, needing to mutate them to add a metadata table or sqlite-docs makes me uncomfortable.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781530343, "label": "Proposal: Combine settings, metadata, static, etc. into a single `datasette.toml` File"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1616095810", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1616095810, "node_id": "IC_kwDOBm6k_c5gU6pC", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-07-01T20:31:31Z", "updated_at": "2023-07-01T20:31:31Z", "author_association": "CONTRIBUTOR", "body": "> Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?\r\n\r\nI just did a github search for `user:simonw \"def extra_js_urls(\"` ! Though I'm sure other plugins made by people other than Simon also exist out there https://github.com/search?q=user%3Asimonw+%22def+extra_js_urls%28%22&type=code", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1615997736", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1615997736, "node_id": "IC_kwDOBm6k_c5gUiso", "user": {"value": 9020979, "label": "hydrosquall"}, "created_at": "2023-07-01T16:55:24Z", "updated_at": "2023-07-01T16:55:24Z", "author_association": "NONE", "body": "> Ok @hydrosquall a couple things before this PR should be good to go:\r\n\r\nThank you @asg017 ! I've pushed both suggested changes onto this branch.\r\n\r\n> Not sure how difficult it'll be to inject it server-side\r\n\r\nIf we are OK with having a build system, it would free me up to do do many things! We could make datasette-manager.js a server-side rendered file as a \"template\" instead of having it as a static JS file, but I'm not sure it's worth the extra jump in complexity / loss of syntax highlighting in the JS file.\r\n\r\nIn the short-term, I could see an intermediary solution where a unit test in the preferred language was able to read both `version.py` and `datasette-manager.js`, and make sure that the strings versions are in sync. (This assumes that we want the manager and datasette's versions to be synced, and not decoupled). Since the version is not changing very often, a \"manual sync\" might be good enough. \r\n\r\n> In terms of how to integrate this into Datasette, a few options I can see working:\r\n\r\nThis sounds good to me. I'm not sure how to add a settings flag, but will be interested to see the PR that adds support for it.\r\n\r\n> I'm also curious to see how \"plugins for a plugin' would work\r\n\r\nI'm comfortable to wait until we have a realistic usecase for this. In the short term, I think we could give plugins a way to grant access to a \"public API of other plugins\", and also ask to be notified when plugins with other names have loaded, but don't picture the datasette manager getting more involved than that. \r\n\r\n> here's a list of Simon's Datasette plugins that use \"extra_js_urls()\"\r\n\r\nNeat, thanks for compiling this list! Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?\r\n\r\n> I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.)\r\n\r\nI look forward to trying this out \ud83d\udc4d \r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2093#issuecomment-1614652001", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2093", "id": 1614652001, "node_id": "IC_kwDOBm6k_c5gPaJh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-30T13:27:13Z", "updated_at": "2023-06-30T13:27:13Z", "author_association": "OWNER", "body": "I agree, settings in the DB doesn't make sense but metadata does.\r\n\r\nOn the JSON v YAML v TOML issue I just spotted Caddy has a concept of config adapters which they use to resolve exactly that problem: https://caddyserver.com/docs/config-adapters", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781530343, "label": "Proposal: Combine settings, metadata, static, etc. into a single `datasette.toml` File"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2093#issuecomment-1613896210", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2093", "id": 1613896210, "node_id": "IC_kwDOBm6k_c5gMhoS", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-06-29T22:53:33Z", "updated_at": "2023-06-29T22:53:33Z", "author_association": "CONTRIBUTOR", "body": "Maybe we can have a separate issue for revamping `metadata.json`? A `datasette_metadata` table or the `sqlite-docs` extension seem like two reasonable additions that we can work through. Storing metadata inside a SQLite database makes sense, but I don't think storing `datasette.*` style config (ex ports, settings, etc.) inside a SQLite DB makes sense, since it's very environment-dependent", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781530343, "label": "Proposal: Combine settings, metadata, static, etc. into a single `datasette.toml` File"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2093#issuecomment-1613895188", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2093", "id": 1613895188, "node_id": "IC_kwDOBm6k_c5gMhYU", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-06-29T22:51:53Z", "updated_at": "2023-06-29T22:51:53Z", "author_association": "CONTRIBUTOR", "body": "I agree with not liking `metadata.json` stuff in a `datasette.*` config file. Editing description of a table/column in a file like `datasette.*` seems odd to me. \r\n\r\nThough since plugin configuration currently lives in `metadata.json`, I think it should be removed from there and placed in `datasette.*`, at least for top-level config like `datasette-auth-github`'s config. Keeping `metadata.json` strictly for documentation/licensing/column units makes sense to me, but anything plugin related should be in some config file, like `datasette.*`.\r\n\r\nAnd ya, supporting both `datasette.*` and CLI flags makes a lot of sense to me. Any `--setting` flag should override anything in `datasette.*` for easier debugging, with possibly a warning message so people don't get confused. Same with `--port` and a port defined in `datasette.*`", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781530343, "label": "Proposal: Combine settings, metadata, static, etc. into a single `datasette.toml` File"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2093#issuecomment-1613889979", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2093", "id": 1613889979, "node_id": "IC_kwDOBm6k_c5gMgG7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T22:44:08Z", "updated_at": "2023-06-30T13:25:39Z", "author_association": "OWNER", "body": "I do like also being able to set options using command line options though - for things like SQL time limits I'd much rather be able to throw on `--setting sql_time_limit_ms 10000` than have to save a config file to disk.\r\n\r\nSo I'd want to support both. Which maybe means also having a way to set plugin options with CLI options. `datasette publish` kind of has that ability already:\r\n\r\n```\r\ndatasette publish heroku my_database.db \\\r\n --name my-heroku-app-demo \\\r\n --install=datasette-auth-github \\\r\n --plugin-secret datasette-auth-github client_id your_client_id \\\r\n --plugin-secret datasette-auth-github client_secret your_client_secret\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781530343, "label": "Proposal: Combine settings, metadata, static, etc. into a single `datasette.toml` File"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2093#issuecomment-1613887492", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2093", "id": 1613887492, "node_id": "IC_kwDOBm6k_c5gMfgE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T22:40:25Z", "updated_at": "2023-06-29T22:40:25Z", "author_association": "OWNER", "body": "I'm strongly in favour of combining settings, configuration and plugin configuration.\r\n\r\nI'm not keen on mixing in metadata as well - that feels like a different concept to me, and I'm unhappy with how that's already had things like plugin settings leak into it.\r\n\r\nI'm not yet sold on TOML - I actually find it less intuitive than YAML, surprisingly. They all have their warts I guess.\r\n\r\nDatasette already has the ability to consume JSON or YAML for metadata - maybe it could grow TOML support too? That way users could have a `datasette.json` or `datasette.yaml` or `datasette.toml` file depending on their preference.\r\n\r\nIn terms of metadata: since that's means to be driven by a plugin hook anyway, maybe one of the potential sources of metadata is a `metadata` nested object in that `datasette.*` configuration file. Or you can have it in a separate `metadata.json` or bundled into the SQLite database or some other plugin-driven mechanism.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781530343, "label": "Proposal: Combine settings, metadata, static, etc. into a single `datasette.toml` File"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1613778296", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1613778296, "node_id": "IC_kwDOBm6k_c5gME14", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-06-29T20:36:09Z", "updated_at": "2023-06-29T20:36:09Z", "author_association": "CONTRIBUTOR", "body": "Ok @hydrosquall a couple things before this PR should be good to go:\r\n\r\n- Can we move `datasette/static/table-example-plugins.js` into `demos/plugins/static`?\r\n- For `datasetteManager.VERSION`, can we fill that in or just comment it out for now? Not sure how difficult it'll be to inject it server-side. I imagine we could also have a small build process with esbuild/rollup that just injects a version string into `manager.js` directly, so we don't have to worry about server-rendering (but that can be a future PR)\r\n\r\nIn terms of how to integrate this into Datasette, a few options I can see working:\r\n\r\n- Push this as-is and figure it out before the next release\r\n- Hide this feature behind a settings flag (`--setting unstable-js-plugins on`) and use that setting to hide/show `` in `base.html`\r\n\r\nI'll let @simonw decide which one to work with. I kindof like the idea of having an \"unstable\" opt-in process to enable JS plugins, to give us time to try it out with a wide variety of plugins until we feel its ready.\r\n\r\nI'm also curious to see how \"plugins for a plugin' would work, like #1542. For example, if the leaflet plugin showed default markers, but also included its own hook for other plugins to add more markers/styling. I'm imagine that the individual plugin would re-create their own plugin system compared to this, since handling \"plugins of plugins\" at the top with Datasette seems really convoluted. \r\n\r\nAlso for posterity, here's a list of Simon's Datasette plugins that use \"extra_js_urls()\", which probably means they can be ported/re-written to use this new plugin system:\r\n\r\n- [`datasette-vega`](https://github.com/simonw/datasette-vega/blob/00de059ab1ef77394ba9f9547abfacf966c479c4/datasette_vega/__init__.py#L25)\r\n- [`datasette-cluster-map`](https://github.com/simonw/datasette-cluster-map/blob/795d25ad9ff6cba0307191f44fecc8f8070bef5c/datasette_cluster_map/__init__.py#L14)\r\n- [`datasette-leaflet-geojson`](https://github.com/simonw/datasette-leaflet-geojson/blob/64713aa497750400b9ac2c12e8bb6ffab8eb77f3/datasette_leaflet_geojson/__init__.py#L47)\r\n- [`datasette-pretty-traces`](https://github.com/simonw/datasette-pretty-traces/blob/5219d65eca3d7d7a73bb9d3120df42fe046a1315/datasette_pretty_traces/__init__.py#L5)\r\n- [`datasette-youtube-embed`](https://github.com/simonw/datasette-youtube-embed/blob/4b4a0d7e58ebe15f47e9baf68beb9908c1d899da/datasette_youtube_embed/__init__.py#L55)\r\n- [`datasette-leaflet-freedraw`](https://github.com/simonw/datasette-leaflet-freedraw/blob/8f28c2c2080ec9d29f18386cc6a2573a1c8fbde7/datasette_leaflet_freedraw/__init__.py#L66)\r\n- [`datasette-hovercards`](https://github.com/simonw/datasette-hovercards/blob/9439ba46b7140fb03223faff0d21aeba5615a287/datasette_hovercards/__init__.py#L5)\r\n- [`datasette-mp3-audio`](https://github.com/simonw/datasette-mp3-audio/blob/4402168792f452a46ab7b488e40ec49cd4b12185/datasette_mp3_audio/__init__.py#L6)\r\n- [`datasette-geojson-map`](https://github.com/simonw/datasette-geojson-map/blob/32af5f1fd1a07278bbf8071fbb20a61e0f613246/datasette_geojson_map/__init__.py#L30)", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2092#issuecomment-1613381990", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2092", "id": 1613381990, "node_id": "IC_kwDOBm6k_c5gKkFm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T15:23:04Z", "updated_at": "2023-06-29T15:26:28Z", "author_association": "OWNER", "body": "Felt lazy:\r\n```bash\r\nsymbex test_homepage -f tests/test_api.py | \\\r\n llm -m 4 --system 'Change all of the == some integer tests in this code to isinstance(thing, int) instead'\r\n```\r\nOutput:\r\n```python\r\n# File: tests/test_api.py Line: 26\r\n@pytest.mark.asyncio\r\nasync def test_homepage(ds_client):\r\n response = await ds_client.get(\"/.json\")\r\n assert response.status_code == 200\r\n assert \"application/json; charset=utf-8\" == response.headers[\"content-type\"]\r\n data = response.json()\r\n assert isinstance(data.keys(), int)\r\n d = data[\"fixtures\"]\r\n assert d[\"name\"] == \"fixtures\"\r\n assert isinstance(d[\"tables_count\"], int)\r\n assert isinstance(len(d[\"tables_and_views_truncated\"]), int)\r\n assert d[\"tables_and_views_more\"] is True\r\n # 4 hidden FTS tables + no_primary_key (hidden in metadata)\r\n assert isinstance(d[\"hidden_tables_count\"], int)\r\n # 201 in no_primary_key, plus 6 in other hidden tables:\r\n assert isinstance(d[\"hidden_table_rows_sum\"], int), data\r\n assert isinstance(d[\"views_count\"], int)\r\n```\r\nI'll use most of that and delete the obsoleted comments.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781047747, "label": "test_homepage intermittent failure"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2092#issuecomment-1613375407", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2092", "id": 1613375407, "node_id": "IC_kwDOBm6k_c5gKiev", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T15:20:52Z", "updated_at": "2023-06-29T15:21:05Z", "author_association": "OWNER", "body": "I'm going to remove this assertion entirely. The homepage JSON needs a refactor anyway.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781047747, "label": "test_homepage intermittent failure"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2091#issuecomment-1613369355", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2091", "id": 1613369355, "node_id": "IC_kwDOBm6k_c5gKhAL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T15:18:34Z", "updated_at": "2023-06-29T15:18:34Z", "author_association": "OWNER", "body": "Posted on the Glitch feedback forum about this here: https://support.glitch.com/t/upgrade-python-version-from-3-7-which-is-now-eol-to-something-more-recent/63011", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781022369, "label": "Drop support for Python 3.7"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2091#issuecomment-1613360413", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2091", "id": 1613360413, "node_id": "IC_kwDOBm6k_c5gKe0d", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T15:13:04Z", "updated_at": "2023-06-29T15:13:04Z", "author_association": "OWNER", "body": "One problem: https://glitch.com/ still provides 3.7:\r\n```\r\n$ python3 --version\r\nPython 3.7.10\r\n```\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781022369, "label": "Drop support for Python 3.7"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2090#issuecomment-1613346412", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2090", "id": 1613346412, "node_id": "IC_kwDOBm6k_c5gKbZs", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T15:05:04Z", "updated_at": "2023-06-29T15:05:04Z", "author_association": "OWNER", "body": "Decided to fix just those \"Ambiguous variable name\" ones:\r\n\r\n```bash\r\nruff check . | grep E741\r\n```\r\nThen iterated through and fixed them all.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781005740, "label": "Adopt ruff for linting"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2090#issuecomment-1613339404", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2090", "id": 1613339404, "node_id": "IC_kwDOBm6k_c5gKZsM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T15:01:01Z", "updated_at": "2023-06-29T15:01:20Z", "author_association": "OWNER", "body": "I tried it just now and got some interesting results.\r\n\r\nI dropped in a `ruff.toml` file:\r\n```toml\r\nline-length = 160\r\n```\r\nBecause the default line length limit of 88 was causing a lot of noisy errors.\r\n\r\nThen run:\r\n\r\n```bash\r\npip install ruff\r\nruff check .\r\n```\r\nPlenty of warnings about unused imports - running `ruff check . --fix` fixed those automatically, but I think I still need to manually review them as some might be imports which are deliberate and should be in `__all__` to ensure they are visible from that module as well.\r\n\r\nSome lines in tests are longer than even 160 chars, e.g.:\r\n\r\nhttps://github.com/simonw/datasette/blob/99ba05118891db9dc30f1dca22ad6709775560de/tests/test_html.py#L673-L681\r\n\r\nThese can have ` # noqa: E501` added to the end of those lines to skip the check for them.\r\n\r\nThat got it down to:\r\n\r\n```\r\n% ruff check . \r\ndatasette/views/table.py:23:5: F811 Redefinition of unused `format_bytes` from line 19\r\nrun_tests.py:2:5: E401 Multiple imports on one line\r\ntests/test_api.py:591:40: F811 Redefinition of unused `app_client_no_files` from line 7\r\ntests/test_api.py:629:35: F811 Redefinition of unused `app_client_no_files` from line 7\r\ntests/test_api.py:635:54: F811 Redefinition of unused `app_client_with_dot` from line 8\r\ntests/test_api.py:661:25: F811 Redefinition of unused `app_client_shorter_time_limit` from line 9\r\ntests/test_api.py:759:25: F811 Redefinition of unused `app_client_two_attached_databases_one_immutable` from line 10\r\ntests/test_api.py:892:28: F811 Redefinition of unused `app_client_larger_cache_size` from line 11\r\ntests/test_api.py:928:5: F811 Redefinition of unused `app_client_with_cors` from line 12\r\ntests/test_api.py:929:5: F811 Redefinition of unused `app_client_two_attached_databases_one_immutable` from line 10\r\ntests/test_api.py:969:38: F811 Redefinition of unused `app_client_two_attached_databases` from line 13\r\ntests/test_api.py:976:39: F811 Redefinition of unused `app_client_conflicting_database_names` from line 14\r\ntests/test_api.py:987:38: F811 Redefinition of unused `app_client_immutable_and_inspect_file` from line 15\r\ntests/test_api.py:1002:24: F811 Redefinition of unused `app_client` from line 6\r\ntests/test_csv.py:67:33: F811 Redefinition of unused `app_client_with_cors` from line 6\r\ntests/test_csv.py:157:21: F811 Redefinition of unused `app_client_csv_max_mb_one` from line 5\r\ntests/test_csv.py:198:20: F811 Redefinition of unused `app_client_with_trace` from line 7\r\ntests/test_csv.py:209:53: F811 Redefinition of unused `app_client_with_trace` from line 7\r\ntests/test_csv.py:215:53: F811 Redefinition of unused `app_client_with_trace` from line 7\r\ntests/test_filters.py:102:11: F811 Redefinition of unused `test_through_filters_from_request` from line 81\r\ntests/test_html.py:19:19: F811 Redefinition of unused `app_client_two_attached_databases` from line 7\r\ntests/test_html.py:175:25: F811 Redefinition of unused `app_client_shorter_time_limit` from line 6\r\ntests/test_html.py:469:51: F811 Redefinition of unused `app_client` from line 4\r\ntests/test_html.py:797:26: F811 Redefinition of unused `app_client_base_url_prefix` from line 5\r\ntests/test_html.py:840:44: F811 Redefinition of unused `app_client_base_url_prefix` from line 5\r\ntests/test_html.py:850:51: F811 Redefinition of unused `app_client_base_url_prefix` from line 5\r\ntests/test_pagination.py:50:43: F821 Undefined name `parse_next`\r\ntests/test_pagination.py:82:7: F811 Redefinition of unused `KeysetPaginator` from line 36\r\ntests/test_plugins.py:115:15: E741 Ambiguous variable name: `l`\r\ntests/test_plugins.py:482:161: E501 Line too long (170 > 160 characters)\r\ntests/test_plugins.py:543:29: E741 Ambiguous variable name: `l`\r\ntests/test_plugins.py:563:161: E501 Line too long (170 > 160 characters)\r\ntests/test_plugins.py:940:62: E741 Ambiguous variable name: `l`\r\ntests/test_table_api.py:739:5: F811 Redefinition of unused `app_client_returned_rows_matches_page_size` from line 6\r\ntests/test_table_api.py:1066:45: F811 Redefinition of unused `app_client_with_trace` from line 5\r\ntests/test_table_html.py:484:29: E741 Ambiguous variable name: `l`\r\ntests/test_table_html.py:524:29: E741 Ambiguous variable name: `l`\r\ntests/test_table_html.py:675:161: E501 Line too long (165 > 160 characters)\r\ntests/test_table_html.py:897:161: E501 Line too long (164 > 160 characters)\r\ntests/test_table_html.py:902:161: E501 Line too long (164 > 160 characters)\r\ntests/test_utils.py:141:161: E501 Line too long (176 > 160 characters)\r\nFound 41 errors.\r\n```\r\nThose \"Redefinition of unused `app_client_two_attached_databases`\" lines are caused because of the fixtures pattern I'm using here:\r\n\r\nhttps://github.com/simonw/datasette/blob/99ba05118891db9dc30f1dca22ad6709775560de/tests/test_html.py#L3-L20\r\n\r\nI could fix that by getting rid of `fixtures.py` and moving those into `conftest.py`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1781005740, "label": "Adopt ruff for linting"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2089#issuecomment-1613316722", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2089", "id": 1613316722, "node_id": "IC_kwDOBm6k_c5gKUJy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T14:48:10Z", "updated_at": "2023-06-29T14:48:10Z", "author_association": "OWNER", "body": "Spell check is passing now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1780973290, "label": "codespell test failure"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2089#issuecomment-1613315851", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2089", "id": 1613315851, "node_id": "IC_kwDOBm6k_c5gKT8L", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T14:47:38Z", "updated_at": "2023-06-29T14:47:38Z", "author_association": "OWNER", "body": "Confirmed, this was a 2.2.5 change: https://github.com/codespell-project/codespell/releases/tag/v2.2.5\r\n\r\n> - Add displaing->displaying by [@peternewman](https://github.com/peternewman) in [#2808](https://github.com/codespell-project/codespell/pull/2808)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1780973290, "label": "codespell test failure"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2089#issuecomment-1613307716", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2089", "id": 1613307716, "node_id": "IC_kwDOBm6k_c5gKR9E", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T14:42:23Z", "updated_at": "2023-06-29T14:42:23Z", "author_association": "OWNER", "body": "Yes, upgrading locally got me the correct version and the test failure:\r\n```\r\n% pip install -U codespell\r\nRequirement already satisfied: codespell in /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages (2.2.2)\r\nCollecting codespell\r\n Downloading codespell-2.2.5-py3-none-any.whl (242 kB)\r\n \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 242.7/242.7 kB 4.9 MB/s eta 0:00:00\r\nInstalling collected packages: codespell\r\n Attempting uninstall: codespell\r\n Found existing installation: codespell 2.2.2\r\n Uninstalling codespell-2.2.2:\r\n Successfully uninstalled codespell-2.2.2\r\nSuccessfully installed codespell-2.2.5\r\n% codespell docs/metadata.rst\r\ndocs/metadata.rst:192: displaing ==> displaying\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1780973290, "label": "codespell test failure"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2089#issuecomment-1613306787", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2089", "id": 1613306787, "node_id": "IC_kwDOBm6k_c5gKRuj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T14:41:47Z", "updated_at": "2023-06-29T14:41:47Z", "author_association": "OWNER", "body": "Looks like in CI it's running 2.2.5:\r\n```\r\nCollecting codespell (from datasette==1.0a2)\r\n Downloading codespell-2.2.5-py3-none-any.whl (242 kB)\r\n \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 242.7/242.7 kB 31.1 MB/s eta 0:00:00\r\n```\r\nBut on my laptop it's 2.2.2:\r\n```\r\n% codespell --version\r\n2.2.2\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1780973290, "label": "codespell test failure"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2089#issuecomment-1613305070", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2089", "id": 1613305070, "node_id": "IC_kwDOBm6k_c5gKRTu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T14:40:44Z", "updated_at": "2023-06-29T14:40:44Z", "author_association": "OWNER", "body": "I'm not sure why I can't duplicate this failure in my local development environment:\r\n```\r\n% codespell docs/metadata.rst\r\n```\r\nIt finds no errors.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1780973290, "label": "codespell test failure"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2077#issuecomment-1613290899", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2077", "id": 1613290899, "node_id": "IC_kwDOBm6k_c5gKN2T", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-29T14:32:16Z", "updated_at": "2023-06-29T14:32:16Z", "author_association": "OWNER", "body": "@dependabot recreate", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1719759468, "label": "Bump furo from 2023.3.27 to 2023.5.20"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1510#issuecomment-1610512875", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1510", "id": 1610512875, "node_id": "IC_kwDOBm6k_c5f_nnr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-28T02:02:10Z", "updated_at": "2023-06-28T02:05:21Z", "author_association": "OWNER", "body": "I prototyped an approach to this using dataclasses and a `cog` mechanism for turning those into rendered tables in Sphinx. Here's what that prototype looks like:\r\n\r\n\"image\"\r\n\r\nSee https://github.com/simonw/datasette/commit/68223784167fdec4e7ebfca56002a6548ba7b423 for how it works.\r\n\r\nHere's the class that documented:\r\n\r\nhttps://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/datasette/context.py#L54-L68\r\n\r\nAnd the code that generates the rST: https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/datasette/context.py#L19-L45\r\n\r\nAnd the bit that cog executes: https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/docs/template_context.rst?plain=1#L9-L12", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1054244712, "label": "Datasette 1.0 documented template context (maybe via API docs)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/561#issuecomment-1610040517", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/561", "id": 1610040517, "node_id": "IC_kwDOCGYnMM5f90TF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-27T18:44:31Z", "updated_at": "2023-06-27T18:44:38Z", "author_association": "OWNER", "body": "Got this working:\r\n```bash\r\nsqlite-utils insert /tmp/playground.db Playground_Submission_Data \\\r\n ~/Downloads/Playground_Submission_Data.csv --csv --stop-after 2000\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1777548699, "label": "`--stop-after` option for `insert` and `upsert` commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606415188", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1606415188, "node_id": "IC_kwDOCGYnMM5fv_NU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-26T01:46:47Z", "updated_at": "2023-06-26T01:47:01Z", "author_association": "OWNER", "body": "I just tested this in a brand new virtual environment using the macOS Python 3:\r\n```bash\r\npipenv shell --python /Applications/Xcode.app/Contents/Developer/usr/bin/python3\r\n```\r\nThen in that virtual environment I ran:\r\n```bash\r\npip install sqlite-utils\r\n# Confirm the right one is on the path:\r\nwhich sqlite-utils\r\n\r\ncurl \"https://data.nasa.gov/resource/y77d-th95.json\" | \\\r\n sqlite-utils insert meteorites.db meteorites - --pk=id\r\nsqlite-utils extract meteorites.db meteorites recclass\r\n```\r\nThis threw the same error reported above. Then I did this:\r\n```bash\r\nrm meteorites.db\r\npip install sqlean.py\r\n\r\ncurl \"https://data.nasa.gov/resource/y77d-th95.json\" | \\\r\n sqlite-utils insert meteorites.db meteorites - --pk=id\r\nsqlite-utils extract meteorites.db meteorites recclass\r\n```\r\nAnd that second time it worked correctly.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606411508", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1606411508, "node_id": "IC_kwDOCGYnMM5fv-T0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-26T01:42:10Z", "updated_at": "2023-06-26T01:42:22Z", "author_association": "OWNER", "body": "https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-33 - upgrading to `sqlite-utils>=3.33` and then installing both `sqlean.py` and `sqlite-dump` in the same virtual environment as `sqlite-utils` should fix this issue.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1606352600", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1606352600, "node_id": "IC_kwDOBm6k_c5fvv7Y", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-06-26T00:17:04Z", "updated_at": "2023-06-26T00:17:04Z", "author_association": "CONTRIBUTOR", "body": ":wave: would love to see this get merged soon! I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.), and this would help out a bunch", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606315321", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606315321, "node_id": "IC_kwDOCGYnMM5fvm05", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T23:18:33Z", "updated_at": "2023-06-25T23:18:33Z", "author_association": "OWNER", "body": "Documentation preview: https://sqlite-utils--560.org.readthedocs.build/en/560/installation.html#alternatives-to-sqlite3", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606310630", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606310630, "node_id": "IC_kwDOCGYnMM5fvlrm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T23:06:07Z", "updated_at": "2023-06-25T23:06:07Z", "author_association": "OWNER", "body": "Filed an issue about the above with `pysqlite3` (which `sqlean.py` is based on) here:\r\n- https://github.com/coleifer/pysqlite3/issues/58", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606297356", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606297356, "node_id": "IC_kwDOCGYnMM5fvicM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T22:42:41Z", "updated_at": "2023-06-25T22:42:41Z", "author_association": "OWNER", "body": "Yes that does seem to do the trick:\r\n```pycon\r\n>>> import sqlean\r\n>>> db = sqlean.connect(\"/tmp/4.db\")\r\n>>> db.execute('PRAGMA journal_mode;').fetchall()\r\n[('delete',)]\r\n>>> db.isolation_level\r\n''\r\n>>> db.execute('PRAGMA journal_mode=wal;')\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\nsqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction\r\n>>> db.isolation_level = None\r\n>>> db.isolation_level\r\n>>> db.execute('PRAGMA journal_mode=wal;')\r\n\r\n```\r\nWeird how `isolation_level` of empty string causes the error, but setting that to `None` fixes the error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606294627", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606294627, "node_id": "IC_kwDOCGYnMM5fvhxj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T22:40:10Z", "updated_at": "2023-06-25T22:40:10Z", "author_association": "OWNER", "body": "I suspect this has something to do with `autocommit` mode in `sqlite3` - which I may be able to turn off by setting `con.isolation_level = None`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606293382", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606293382, "node_id": "IC_kwDOCGYnMM5fvheG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T22:34:47Z", "updated_at": "2023-06-25T22:34:47Z", "author_association": "OWNER", "body": "```pycon\r\n>>> import sqlite3\r\n>>> db = sqlite3.connect(\"/tmp/1.db\")\r\n>>> db.execute('PRAGMA journal_mode=wal;')\r\n\r\n>>> import sqlean\r\n>>> db2 = sqlean.connect(\"/tmp/2.db\")\r\n>>> db2.execute('PRAGMA journal_mode=wal;')\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\nsqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606290917", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606290917, "node_id": "IC_kwDOCGYnMM5fvg3l", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T22:32:28Z", "updated_at": "2023-06-25T22:32:28Z", "author_association": "OWNER", "body": "I've fixed most of the test failures, but I still need to fix this one:\r\n\r\n> cannot change into wal mode from within a transaction", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606273005", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606273005, "node_id": "IC_kwDOCGYnMM5fvcft", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T21:47:47Z", "updated_at": "2023-06-25T21:47:47Z", "author_association": "OWNER", "body": "I can use https://github.com/simonw/sqlite-dump as an optional dependency to handle the missing `.iterdump()` method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270887", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606270887, "node_id": "IC_kwDOCGYnMM5fvb-n", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T21:37:12Z", "updated_at": "2023-06-26T08:21:00Z", "author_association": "OWNER", "body": "On my own laptop I got a crash running the tests - details here:\r\n\r\n- https://github.com/nalgeon/sqlean.py/issues/3", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270055", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606270055, "node_id": "IC_kwDOCGYnMM5fvbxn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T21:31:56Z", "updated_at": "2023-06-25T21:31:56Z", "author_association": "OWNER", "body": "Lots of failures now that I'm trying to run the tests against `sqlean.py` on macOS and Python 3.10: https://github.com/simonw/sqlite-utils/actions/runs/5371800108/jobs/9744802953\r\n\r\nA bunch of these, because `pysqlite3` chooses not to implement `.iterdump()`:\r\n```\r\n @pytest.fixture\r\n def db_to_analyze_path(db_to_analyze, tmpdir):\r\n path = str(tmpdir / \"test.db\")\r\n db = sqlite3.connect(path)\r\n> db.executescript(\"\\n\".join(db_to_analyze.conn.iterdump()))\r\nE AttributeError: 'sqlean.dbapi2.Connection' object has no attribute 'iterdump'\r\n```\r\nAlso some of these:\r\n```\r\n def test_analyze_whole_database(db):\r\n assert set(db.table_names()) == {\"one_index\", \"two_indexes\"}\r\n db.analyze()\r\n> assert set(db.table_names()) == {\"one_index\", \"two_indexes\", \"sqlite_stat1\"}\r\nE AssertionError: assert {'one_index',...'two_indexes'} == {'one_index',...'two_indexes'}\r\nE Extra items in the left set:\r\nE 'sqlite_stat4'\r\nE Full diff:\r\nE - {'two_indexes', 'sqlite_stat1', 'one_index'}\r\nE + {'two_indexes', 'sqlite_stat1', 'sqlite_stat4', 'one_index'}\r\nE ? ++++++++++++++++\r\n```\r\nApparently `sqlean.py` adds a `sqlite_stat4` table that the tests are not expecting.\r\n\r\nPlus some errors that look like this:\r\n```\r\n def test_enable_wal():\r\n runner = CliRunner()\r\n dbs = [\"test.db\", \"test2.db\"]\r\n with runner.isolated_filesystem():\r\n for dbname in dbs:\r\n db = Database(dbname)\r\n db[\"t\"].create({\"pk\": int}, pk=\"pk\")\r\n assert db.journal_mode == \"delete\"\r\n result = runner.invoke(cli.cli, [\"enable-wal\"] + dbs)\r\n> assert 0 == result.exit_code\r\nE AssertionError: assert 0 == 1\r\nE + where 1 = .exit_code\r\n```\r\nTest summary:\r\n```\r\n============ 13 failed, 909 passed, 16 skipped, 2 errors in 19.29s =============\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606237836", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606237836, "node_id": "IC_kwDOCGYnMM5fvT6M", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-06-25T19:49:45Z", "updated_at": "2023-06-26T08:20:59Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage: **`80.55`**% and project coverage change: **`-0.15`** :warning:\n> Comparison is base [(`2747257`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/2747257a3334d55e890b40ec58fada57ae8cfbfd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`5e7d27e`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.22%.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #560 +/- ##\n==========================================\n- Coverage 96.36% 96.22% -0.15% \n==========================================\n Files 6 6 \n Lines 2726 2752 +26 \n==========================================\n+ Hits 2627 2648 +21 \n- Misses 99 104 +5 \n```\n\n\n| [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/utils.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.56% <63.63%> (-0.62%)` | :arrow_down: |\n| [sqlite\\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.33% <86.36%> (-0.20%)` | :arrow_down: |\n| [sqlite\\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <100.00%> (\u00f8)` | |\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1604379952", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1604379952, "node_id": "IC_kwDOCGYnMM5foOUw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-23T14:39:55Z", "updated_at": "2023-06-23T15:39:32Z", "author_association": "OWNER", "body": "Ideally a workaround for this right now would be to install `pysqlite3` in the same virtual environment:\r\n\r\n sqlite-utils install pysqlite3-binary\r\n\r\nBut `pysqlite3-binary` doesn't yet ship a wheel for macOS so this probably won't work for most people.\r\n\r\nThe \"easiest\" fix at the moment is to use Python from Homebrew - so `brew install sqlite-utils` for example won't suffer from this problem. Not a great solution for people who aren't using Homebrew though!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/260#issuecomment-1600778057", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/260", "id": 1600778057, "node_id": "IC_kwDOBm6k_c5fae9J", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-21T12:51:22Z", "updated_at": "2023-06-21T12:51:22Z", "author_association": "OWNER", "body": "Another example of confusion from this today: https://discord.com/channels/823971286308356157/823971286941302908/1121042411238457374\r\n\r\nSee also https://gist.github.com/BinomeDeNewton/651ac8b50dd5420f8e54d1682eee5fed?permalink_comment_id=4605982#gistcomment-4605982", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323223872, "label": "Validate metadata.json on startup"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592617271", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/535", "id": 1592617271, "node_id": "IC_kwDOCGYnMM5e7Wk3", "user": {"value": 13780613, "label": "erlend-aasland"}, "created_at": "2023-06-15T08:39:49Z", "updated_at": "2023-06-15T08:39:49Z", "author_association": "NONE", "body": "> piping to `jq` is good enough usually\r\n\r\n... or `python -m json.tool`[^1], if you don't have `jq` installed.\r\n\r\n[^1]: no fancy colouring, like `jq`; only pretty-printing", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1655860104, "label": "rows: --transpose or psql extended view-like functionality"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/529#issuecomment-1592110694", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/529", "id": 1592110694, "node_id": "IC_kwDOCGYnMM5e5a5m", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T23:11:47Z", "updated_at": "2023-06-14T23:12:12Z", "author_association": "CONTRIBUTOR", "body": "sorry i was wrong. `sqlite-utils --raw-lines` works correctly\r\n\r\n```\r\nsqlite-utils --raw-lines :memory: \"SELECT * FROM (VALUES ('test'), ('line2'))\" | cat -A\r\ntest$\r\nline2$\r\n\r\nsqlite-utils --csv --no-headers :memory: \"SELECT * FROM (VALUES ('test'), ('line2'))\" | cat -A\r\ntest$\r\nline2$\r\n```\r\n\r\nI think this was fixed somewhat recently", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1581090327, "label": "Microsoft line endings"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592052320", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/535", "id": 1592052320, "node_id": "IC_kwDOCGYnMM5e5Mpg", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T22:05:28Z", "updated_at": "2023-06-14T22:05:28Z", "author_association": "CONTRIBUTOR", "body": "piping to `jq` is good enough usually", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1655860104, "label": "rows: --transpose or psql extended view-like functionality"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/555#issuecomment-1592047502", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/555", "id": 1592047502, "node_id": "IC_kwDOCGYnMM5e5LeO", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T22:00:10Z", "updated_at": "2023-06-14T22:01:57Z", "author_association": "CONTRIBUTOR", "body": "You may want to try doing a performance comparison between this and just selecting all the ids with few constraints and then doing the filtering within python.\r\n\r\nThat might seem like a lazy-programmer, inefficient way but queries with large resultsets are a different profile than what databases like SQLITE are designed for. That is not to say that SQLITE is slow or that python is always faster but when you start reading >20% of an index there is an equilibrium that is reached. Especially when adding in writing extra temp tables and stuff to memory/disk. And especially given the `NOT IN` style of query...\r\n\r\nYou may also try chunking like this:\r\n\r\n```py\r\ndef chunks(lst, n) -> Generator:\r\n for i in range(0, len(lst), n):\r\n yield lst[i : i + n]\r\n\r\nSQLITE_PARAM_LIMIT = 32765\r\n\r\ndata = []\r\nchunked = chunks(video_ids, consts.SQLITE_PARAM_LIMIT)\r\nfor ids in chunked:\r\n data.expand(\r\n list(\r\n db.query(\r\n f\"\"\"SELECT * from videos\r\n WHERE id in (\"\"\"\r\n + \",\".join([\"?\"] * len(ids))\r\n + \")\",\r\n (*ids,),\r\n )\r\n )\r\n )\r\n```\r\n\r\nbut that actually won't work with your `NOT IN` requirements. You need to query the full resultset to check any row.\r\n\r\nSince you are doing stuff with files/videos in SQLITE you might be interested in my side project: https://github.com/chapmanjacobd/library", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1733198948, "label": "Filter table by a large bunch of ids"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1590531892", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/557", "id": 1590531892, "node_id": "IC_kwDOCGYnMM5ezZc0", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T06:09:21Z", "updated_at": "2023-06-14T06:09:21Z", "author_association": "CONTRIBUTOR", "body": "I put together a [simple script](https://github.com/chapmanjacobd/library/blob/42129c5ebe15f9d74653c0f5ca4ed0c991d383e0/xklb/scripts/dedupe_db.py) to upsert and remove duplicate rows based on business keys. If anyone has similar problems with above this might help\r\n\r\n```\r\nCREATE TABLE my_table (\r\n id INTEGER PRIMARY KEY,\r\n column1 TEXT,\r\n column2 TEXT,\r\n column3 TEXT\r\n);\r\n\r\nINSERT INTO my_table (column1, column2, column3)\r\nVALUES\r\n ('Value 1', 'Duplicate 1', 'Duplicate A'),\r\n ('Value 2', 'Duplicate 2', 'Duplicate B'),\r\n ('Value 3', 'Duplicate 2', 'Duplicate C'),\r\n ('Value 4', 'Duplicate 3', 'Duplicate D'),\r\n ('Value 5', 'Duplicate 3', 'Duplicate E'),\r\n ('Value 6', 'Duplicate 3', 'Duplicate F');\r\n```\r\n\r\n```\r\nlibrary dedupe-db test.db my_table --bk column2\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1740150327, "label": "Aliased ROWID option for tables created from alter=True commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1585149909", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1585149909, "node_id": "IC_kwDOBm6k_c5ee3fV", "user": {"value": 9020979, "label": "hydrosquall"}, "created_at": "2023-06-09T21:35:00Z", "updated_at": "2023-06-09T21:35:00Z", "author_association": "NONE", "body": "Thanks @cldellow for the thoughtful comments! These are all things that I'll keep in mind as we figure out how/if this API is actually used by plugin authors once it's actually out in the world.\r\n\r\n> Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.\r\n\r\nAh, I understand now! Thanks for explaining. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1578840450", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/433", "id": 1578840450, "node_id": "IC_kwDOCGYnMM5eGzGC", "user": {"value": 392720, "label": "jonafato"}, "created_at": "2023-06-06T14:09:04Z", "updated_at": "2023-06-06T14:09:04Z", "author_association": "NONE", "body": "I also ran into this recently. See below for a patch for one possible solution (tested via \"it works on my machine\", but I don't expect that this behavior would vary a whole lot across terminal emulators and shells). Another possible solution might be to subclass click's `ProgressBar` to keep the logic within the original context manager. Happy to send a PR or for this patch to serve as the basis for a fix that someone else authors.\r\n\r\n```patch\r\ndiff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py\r\nindex 06c1a4c..530a3a3 100644\r\n--- a/sqlite_utils/utils.py\r\n+++ b/sqlite_utils/utils.py\r\n@@ -147,14 +147,23 @@ def decode_base64_values(doc):\r\n \r\n \r\n class UpdateWrapper:\r\n- def __init__(self, wrapped, update):\r\n+ def __init__(self, wrapped, update, render_finish):\r\n self._wrapped = wrapped\r\n self._update = update\r\n+ self._render_finish = render_finish\r\n \r\n def __iter__(self):\r\n- for line in self._wrapped:\r\n- self._update(len(line))\r\n- yield line\r\n+ return self\r\n+\r\n+ def __next__(self):\r\n+ try:\r\n+ line = next(self._wrapped)\r\n+ except StopIteration as e:\r\n+ self._render_finish()\r\n+ raise\r\n+\r\n+ self._update(len(line))\r\n+ return line\r\n \r\n def read(self, size=-1):\r\n data = self._wrapped.read(size)\r\n@@ -178,7 +187,7 @@ def file_progress(file, silent=False, **kwargs):\r\n else:\r\n file_length = os.path.getsize(file.name)\r\n with click.progressbar(length=file_length, **kwargs) as bar:\r\n- yield UpdateWrapper(file, bar.update)\r\n+ yield UpdateWrapper(file, bar.update, bar.render_finish)\r\n \r\n \r\n class Format(enum.Enum):\r\n\r\n```", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 1, \"eyes\": 0}", "issue": {"value": 1239034903, "label": "CLI eats my cursor"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1577355134", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/557", "id": 1577355134, "node_id": "IC_kwDOCGYnMM5eBId-", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-05T19:26:26Z", "updated_at": "2023-06-05T19:26:26Z", "author_association": "CONTRIBUTOR", "body": "this isn't really actionable... I'm just being a whiny baby. I have tasted the milk of being able to use `upsert_all`, `insert_all`, etc without having to write DDL to create tables. The meat of the issue is that SQLITE doesn't make rowid stable between vacuums so it is not possible to take shortcuts", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1740150327, "label": "Aliased ROWID option for tables created from alter=True commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/556#issuecomment-1575310378", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/556", "id": 1575310378, "node_id": "IC_kwDOCGYnMM5d5VQq", "user": {"value": 601708, "label": "mcint"}, "created_at": "2023-06-04T01:21:15Z", "updated_at": "2023-06-04T01:21:15Z", "author_association": "CONTRIBUTOR", "body": "I've resolved my use, with the line-buffered output and while read loop for line buffered input, but I leave this here so the incremental saving or line-buffered use-case can be explicitly handled or rejected (or deferred).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1740026046, "label": "Support storing incrementally piped values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2053#issuecomment-1565058994", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2053", "id": 1565058994, "node_id": "IC_kwDOBm6k_c5dSOey", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-26T23:13:02Z", "updated_at": "2023-05-26T23:13:02Z", "author_association": "OWNER", "body": "I should have an extra called `extra_html_context` which bundles together all of the weird extra stuff needed by the HTML template, and is then passed as the root context when the template is rendered (with the other stuff from extras patched into it).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1656432059, "label": "WIP new JSON for queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2053#issuecomment-1563793781", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2053", "id": 1563793781, "node_id": "IC_kwDOBm6k_c5dNZl1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-26T04:27:55Z", "updated_at": "2023-05-26T04:27:55Z", "author_association": "OWNER", "body": "I should split out a `canned_query.html` template too, as something that extends the `query.html` template.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1656432059, "label": "WIP new JSON for queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2053#issuecomment-1563667574", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2053", "id": 1563667574, "node_id": "IC_kwDOBm6k_c5dM6x2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-26T00:40:22Z", "updated_at": "2023-05-26T00:40:22Z", "author_association": "OWNER", "body": "Or maybe...\r\n\r\n- `BaseQueryView(View)` - knows how to render the results of a SQL query\r\n- `QueryView(BaseQueryView)` - renders from `?sql=`\r\n- `CannedQueryView(BaseQueryView)` - renders for a named canned query\r\n\r\nAnd then later perhaps:\r\n\r\n- `RowQueryView(BaseQueryView)` - renders the `select * from t where pk = ?`\r\n- `TableQueryView(BaseQueryView)` - replaces the super complex existing `TableView`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1656432059, "label": "WIP new JSON for queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2053#issuecomment-1563663925", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2053", "id": 1563663925, "node_id": "IC_kwDOBm6k_c5dM541", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-26T00:32:47Z", "updated_at": "2023-05-26T00:35:47Z", "author_association": "OWNER", "body": "I'm going to entirely split canned queries off from `?sql=` queries - they share a bunch of code right now which is just making everything much harder to follow.\r\n\r\nI'll refactor their shared bits into functions that they both call.\r\n\r\nOr _maybe_ I'll try having `CannedQueryView` as a subclass of `QueryView`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1656432059, "label": "WIP new JSON for queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2053#issuecomment-1563663616", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2053", "id": 1563663616, "node_id": "IC_kwDOBm6k_c5dM50A", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-26T00:32:08Z", "updated_at": "2023-05-26T00:32:08Z", "author_association": "OWNER", "body": "Now that I have the new `View` subclass from #2078 I want to use it to simplify this code.\r\n\r\nChallenge: there are several things to consider here:\r\n\r\n- The `/db` page without `?sql=` displays a list of tables in that database\r\n- With `?sql=` it shows the query results for that query (or an error)\r\n- If it's a `/db/name-of-canned-query` it works a bit like the query page, but executes a canned query instead of the `?sql=` query\r\n- POST `/db/name-of-canned-query` is support for writable canned queries", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1656432059, "label": "WIP new JSON for queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2080#issuecomment-1563650990", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2080", "id": 1563650990, "node_id": "IC_kwDOBm6k_c5dM2uu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-26T00:08:59Z", "updated_at": "2023-05-26T00:08:59Z", "author_association": "OWNER", "body": "I'm not going to document this yet, I want to let it bake for a bit longer first.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1726603778, "label": "New View base class"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2080#issuecomment-1563629348", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2080", "id": 1563629348, "node_id": "IC_kwDOBm6k_c5dMxck", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-05-25T23:31:10Z", "updated_at": "2023-05-26T00:07:34Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage: **`95.45`**% and project coverage change: **`+0.01`** :tada:\n> Comparison is base [(`b49fa44`)](https://app.codecov.io/gh/simonw/datasette/commit/b49fa446d683ddcaf6faf2944dacc0d866bf2d70?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`df5fd73`)](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.41%.\n\n> :exclamation: Current head df5fd73 differs from pull request most recent head e990fbc. Consider uploading reports for the commit e990fbc to get more accurate results\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #2080 +/- ##\n==========================================\n+ Coverage 92.40% 92.41% +0.01% \n==========================================\n Files 39 39 \n Lines 5768 5790 +22 \n==========================================\n+ Hits 5330 5351 +21 \n- Misses 438 439 +1 \n```\n\n\n| [Impacted Files](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/views/base.py](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `92.97% <95.45%> (+0.18%)` | :arrow_up: |\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1726603778, "label": "New View base class"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/2080#issuecomment-1563626231", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2080", "id": 1563626231, "node_id": "IC_kwDOBm6k_c5dMwr3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-25T23:25:17Z", "updated_at": "2023-05-25T23:25:17Z", "author_association": "OWNER", "body": "I'm going to try using this for the `/-/patterns` page.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1726603778, "label": "New View base class"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2078#issuecomment-1563625093", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2078", "id": 1563625093, "node_id": "IC_kwDOBm6k_c5dMwaF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-25T23:23:15Z", "updated_at": "2023-05-25T23:23:15Z", "author_association": "OWNER", "body": "Rest of the work on this will happen in the PR:\r\n- #2080", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1726236847, "label": "Resolve the difference between `wrap_view()` and `BaseView`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2079#issuecomment-1563607291", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2079", "id": 1563607291, "node_id": "IC_kwDOBm6k_c5dMsD7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-25T22:56:28Z", "updated_at": "2023-05-25T22:56:28Z", "author_association": "OWNER", "body": "Wrote this up as a TIL: https://til.simonwillison.net/http/testing-cors-max-age", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1726531350, "label": "Datasette should serve Access-Control-Max-Age"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2079#issuecomment-1563597589", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2079", "id": 1563597589, "node_id": "IC_kwDOBm6k_c5dMpsV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-25T22:42:07Z", "updated_at": "2023-05-25T22:42:07Z", "author_association": "OWNER", "body": "Mystery solved as to why I wasn't seeing this work:\r\n\r\n\"CleanShot\r\n\r\nI had \"Disable Cache\" checked!\r\n\r\nI ran this experiment after un-checking that box:\r\n\r\n```javascript\r\nfetch('https://latest.datasette.io/ephemeral/foo/1/-/update', {\r\n method: 'POST',\r\n headers: {\r\n 'Content-Type': 'application/json'\r\n },\r\n body: JSON.stringify({ test: 'test' })\r\n});\r\n// And run it again\r\nfetch('https://latest.datasette.io/ephemeral/foo/1/-/update', {\r\n method: 'POST',\r\n headers: {\r\n 'Content-Type': 'application/json'\r\n },\r\n body: JSON.stringify({ test: 'test' })\r\n});\r\n// Now try a thing that doesn't serve that max-age header yet:\r\nfetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', {\r\n method: 'POST',\r\n headers: {\r\n 'Content-Type': 'application/json'\r\n },\r\n body: JSON.stringify({ test: 'test' })\r\n});\r\n// And a second time but within 5s\r\nfetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', {\r\n method: 'POST',\r\n headers: {\r\n 'Content-Type': 'application/json'\r\n },\r\n body: JSON.stringify({ test: 'test' })\r\n});\r\n// Third time after waiting longer than 5s\r\nfetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', {\r\n method: 'POST',\r\n headers: {\r\n 'Content-Type': 'application/json'\r\n },\r\n body: JSON.stringify({ test: 'test' })\r\n});\r\n// Try that original one again - still within the 1hr cache time\r\nfetch('https://latest.datasette.io/ephemeral/foo/1/-/update', {\r\n method: 'POST',\r\n headers: {\r\n 'Content-Type': 'application/json'\r\n },\r\n body: JSON.stringify({ test: 'test' })\r\n});\r\n```\r\nThe results show that the cache of 1hr was being obeyed for `latest.datasette.io` while the `latest-with-plugins.datasette.io` default cache of 5s was being obeyed too.\r\n\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1726531350, "label": "Datasette should serve Access-Control-Max-Age"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2079#issuecomment-1563588199", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2079", "id": 1563588199, "node_id": "IC_kwDOBm6k_c5dMnZn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-25T22:29:47Z", "updated_at": "2023-05-25T22:30:12Z", "author_association": "OWNER", "body": "https://fetch.spec.whatwg.org/#http-access-control-max-age says:\r\n\r\n> Indicates the number of seconds (5 by default) the information provided by the [Access-Control-Allow-Methods](https://fetch.spec.whatwg.org/#http-access-control-allow-methods) and [Access-Control-Allow-Headers](https://fetch.spec.whatwg.org/#http-access-control-allow-headers) [headers](https://fetch.spec.whatwg.org/#concept-header) can be cached.\r\n\r\nSo there was already a 5s cache anyway.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1726531350, "label": "Datasette should serve Access-Control-Max-Age"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/2079#issuecomment-1563587230", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2079", "id": 1563587230, "node_id": "IC_kwDOBm6k_c5dMnKe", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-25T22:28:20Z", "updated_at": "2023-05-25T22:28:20Z", "author_association": "OWNER", "body": "Weird... after the deploy went out:\r\n\r\n\"image\"\r\n\r\nBut the request did indeed get the new header:\r\n\r\n\"image\"\r\n\r\nSo I'm not sure why it's making multiple `POST` requests like that.\r\n\r\nMaybe it's because the attempted `POST` failed with a 404?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1726531350, "label": "Datasette should serve Access-Control-Max-Age"}, "performed_via_github_app": null}