{"html_url": "https://github.com/simonw/datasette/pull/1368#issuecomment-869068554", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1368", "id": 869068554, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA2ODU1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:23:57Z", "updated_at": "2021-06-26T22:23:57Z", "author_association": "OWNER", "body": "The only test failure is Black. I'm going to merge this and then reformat.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 913865304, "label": "DRAFT: A new plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869069655", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869069655, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA2OTY1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:36:14Z", "updated_at": "2021-06-26T22:37:37Z", "author_association": "OWNER", "body": "Documentation for the new hook is now live at https://docs.datasette.io/en/latest/plugin_hooks.html#get-metadata-datasette-key-database-table-fallback\r\n\r\nLink to the current snapshot of that documentation: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/docs/plugin_hooks.rst#get-metadata-datasette-key-database-table-fallback", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869069768", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869069768, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA2OTc2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:37:53Z", "updated_at": "2021-06-26T22:37:53Z", "author_association": "OWNER", "body": "The documentation doesn't describe the ``fallback`` argument at the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869069926", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869069926, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA2OTkyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:40:15Z", "updated_at": "2021-06-26T22:40:53Z", "author_association": "OWNER", "body": "The documentation says:\r\n\r\n> **datasette**:     You can use this to access plugin configuration options via `datasette.plugin_config(your_plugin_name)`, or to execute SQL queries.\r\n\r\nThat's not accurate: since the plugin hook is a regular function, not an awaitable, you can't use it to run `await db.execute(...)` so you can't execute SQL queries.\r\n\r\nI can fix this with the await-me-maybe pattern, used for other plugin hooks: https://simonwillison.net/2020/Sep/2/await-me-maybe/\r\n\r\nBUT... that requires changing the `ds.metadata()` function to be awaitable, which will affect every existing plugn that uses that documented internal method!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869070076", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869070076, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MDA3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:42:21Z", "updated_at": "2021-06-26T22:42:21Z", "author_association": "OWNER", "body": "Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.\r\n\r\n@brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869070348", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869070348, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MDM0OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:46:18Z", "updated_at": "2021-06-26T22:46:18Z", "author_association": "OWNER", "body": "Here's where the plugin hook is called, demonstrating the `fallback=` argument: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472\r\n\r\nI'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869070941", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869070941, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MDk0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:53:34Z", "updated_at": "2021-06-26T22:53:34Z", "author_association": "OWNER", "body": "The `await` thing is worrying me a lot - it feels like this plugin hook is massively less useful if it can't make it's own DB queries and generally do asynchronous stuff - but I'd also like not to break every existing plugin that calls `datasette.metadata(...)`.\r\n\r\nOne solution that could work: introduce a new method, maybe `await datasette.get_metadata(...)`, which uses this plugin hook - and keep the existing `datasette.metadata()` method (which doesn't call the hook) around. This would ensure existing plugins keep on working.\r\n\r\nThen, upgrade those plugins separately - with the goal of deprecating and removing `.metadata()` entirely in Datasette 1.0 - having upgraded the plugins in the meantime.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869071167", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869071167, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MTE2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:55:36Z", "updated_at": "2021-06-26T22:55:36Z", "author_association": "OWNER", "body": "Just realized I already have an issue open for this, at #860. I'm going to close that and continue work on this in this issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869071435", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869071435, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MTQzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:59:26Z", "updated_at": "2021-06-26T22:59:26Z", "author_association": "OWNER", "body": "The other alternative is to finish the work to build a `_metadata` internal table, see #1168. The idea there was that if we want to support efficient pagination and search across the metadata for thousands of attached tables powering it with a plugin hook doesn't work well - we don't want to call the hook once for every one of 1,000+ tables just to implement the homepage.\r\n\r\nSo instead, all metadata for all attached databases would be loaded into an in-memory database called `_metadata`. Plugins that want to modify stored metadata could then do so by directly writing to that table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869071790", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869071790, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MTc5MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T23:04:12Z", "updated_at": "2021-06-26T23:04:12Z", "author_association": "OWNER", "body": "> Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.\r\n> \r\n> @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?\r\n\r\nAnswering my own question: here's how Brandon implements it in his `datasette-live-config` plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160\r\n\r\nThat's using a completely separate SQLite connection (actually wrapped in `sqlite-utils`) and making blocking synchronous calls to it.\r\n\r\nThis is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.\r\n\r\nBut... it's weird. Everywhere else in Datasette land uses `await db.execute(...)` - but here's an example where users are encouraged to use blocking calls instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869074182", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869074182, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3NDE4Mg==", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2021-06-26T23:37:42Z", "updated_at": "2021-06-26T23:37:42Z", "author_association": "CONTRIBUTOR", "body": "> > Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.\r\n> > @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?\r\n> \r\n> Answering my own question: here's how Brandon implements it in his `datasette-live-config` plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160\r\n> \r\n> That's using a completely separate SQLite connection (actually wrapped in `sqlite-utils`) and making blocking synchronous calls to it.\r\n> \r\n> This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.\r\n> \r\n> But... it's weird. Everywhere else in Datasette land uses `await db.execute(...)` - but here's an example where users are encouraged to use blocking calls instead.\r\n\r\n_Ideally_ this hook would be asynchronous, but when I started down that path I quickly realized how large of a change this would be, since metadata gets used synchronously across the entire Datasette codebase. (And calling async code from sync is non-trivial.)\r\n\r\nIn my live-configuration implementation I use synchronous reads using a persistent sqlite connection. This works pretty well in practice, but I agree it's limiting. My thinking around this was to go with the path of least change as `Datasette.metadata()` is a critical core function.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869074701", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869074701, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3NDcwMQ==", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2021-06-26T23:45:18Z", "updated_at": "2021-06-26T23:45:37Z", "author_association": "CONTRIBUTOR", "body": "> Here's where the plugin hook is called, demonstrating the `fallback=` argument:\r\n> \r\n> https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472\r\n> \r\n> I'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?\r\n\r\nI think you're right. I can't think of a reason why the plugin would care about the `fallback` parameter since plugins are currently mandated to return a full, global metadata dict.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869075368", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869075368, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3NTM2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T23:53:55Z", "updated_at": "2021-06-26T23:53:55Z", "author_association": "OWNER", "body": "Great, let's drop fallback then.\r\n\r\nMy instinct at the moment is to ship this plugin hook as-is but with a warning that it may change before Datasette 1.0 - then before 1.0 either figure out an async variant or finish the database-backed metadata concept from #1168 and recommend that as an alternative.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869075395", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869075395, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3NTM5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T23:54:21Z", "updated_at": "2021-06-26T23:59:21Z", "author_association": "OWNER", "body": "(It may well be that implementing #1168 involves a switch to async metadata)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/860#issuecomment-869071236", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/860", "id": 869071236, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MTIzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:56:28Z", "updated_at": "2021-06-26T22:56:28Z", "author_association": "OWNER", "body": "This work is continuing in #1384.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642651572, "label": "Plugin hook for instance/database/table metadata"}, "performed_via_github_app": null}