html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/1384#issuecomment-869075395,https://api.github.com/repos/simonw/datasette/issues/1384,869075395,MDEyOklzc3VlQ29tbWVudDg2OTA3NTM5NQ==,9599,simonw,2021-06-26T23:54:21Z,2021-06-26T23:59:21Z,OWNER,(It may well be that implementing #1168 involves a switch to async metadata),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869075368,https://api.github.com/repos/simonw/datasette/issues/1384,869075368,MDEyOklzc3VlQ29tbWVudDg2OTA3NTM2OA==,9599,simonw,2021-06-26T23:53:55Z,2021-06-26T23:53:55Z,OWNER,"Great, let's drop fallback then.

My instinct at the moment is to ship this plugin hook as-is but with a warning that it may change before Datasette 1.0 - then before 1.0 either figure out an async variant or finish the database-backed metadata concept from #1168 and recommend that as an alternative.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869074701,https://api.github.com/repos/simonw/datasette/issues/1384,869074701,MDEyOklzc3VlQ29tbWVudDg2OTA3NDcwMQ==,2670795,brandonrobertz,2021-06-26T23:45:18Z,2021-06-26T23:45:37Z,CONTRIBUTOR,"> Here's where the plugin hook is called, demonstrating the `fallback=` argument:
> 
> https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472
> 
> I'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?

I think you're right. I can't think of a reason why the plugin would care about the `fallback` parameter since plugins are currently mandated to return a full, global metadata dict.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869074182,https://api.github.com/repos/simonw/datasette/issues/1384,869074182,MDEyOklzc3VlQ29tbWVudDg2OTA3NDE4Mg==,2670795,brandonrobertz,2021-06-26T23:37:42Z,2021-06-26T23:37:42Z,CONTRIBUTOR,"> > Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.
> > @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?
> 
> Answering my own question: here's how Brandon implements it in his `datasette-live-config` plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160
> 
> That's using a completely separate SQLite connection (actually wrapped in `sqlite-utils`) and making blocking synchronous calls to it.
> 
> This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.
> 
> But... it's weird. Everywhere else in Datasette land uses `await db.execute(...)` - but here's an example where users are encouraged to use blocking calls instead.

_Ideally_ this hook would be asynchronous, but when I started down that path I quickly realized how large of a change this would be, since metadata gets used synchronously across the entire Datasette codebase. (And calling async code from sync is non-trivial.)

In my live-configuration implementation I use synchronous reads using a persistent sqlite connection. This works pretty well in practice, but I agree it's limiting. My thinking around this was to go with the path of least change as `Datasette.metadata()` is a critical core function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869071790,https://api.github.com/repos/simonw/datasette/issues/1384,869071790,MDEyOklzc3VlQ29tbWVudDg2OTA3MTc5MA==,9599,simonw,2021-06-26T23:04:12Z,2021-06-26T23:04:12Z,OWNER,"> Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.
> 
> @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?

Answering my own question: here's how Brandon implements it in his `datasette-live-config` plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160

That's using a completely separate SQLite connection (actually wrapped in `sqlite-utils`) and making blocking synchronous calls to it.

This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.

But... it's weird. Everywhere else in Datasette land uses `await db.execute(...)` - but here's an example where users are encouraged to use blocking calls instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869071435,https://api.github.com/repos/simonw/datasette/issues/1384,869071435,MDEyOklzc3VlQ29tbWVudDg2OTA3MTQzNQ==,9599,simonw,2021-06-26T22:59:26Z,2021-06-26T22:59:26Z,OWNER,"The other alternative is to finish the work to build a `_metadata` internal table, see #1168. The idea there was that if we want to support efficient pagination and search across the metadata for thousands of attached tables powering it with a plugin hook doesn't work well - we don't want to call the hook once for every one of 1,000+ tables just to implement the homepage.

So instead, all metadata for all attached databases would be loaded into an in-memory database called `_metadata`. Plugins that want to modify stored metadata could then do so by directly writing to that table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-869071236,https://api.github.com/repos/simonw/datasette/issues/860,869071236,MDEyOklzc3VlQ29tbWVudDg2OTA3MTIzNg==,9599,simonw,2021-06-26T22:56:28Z,2021-06-26T22:56:28Z,OWNER,This work is continuing in #1384.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869071167,https://api.github.com/repos/simonw/datasette/issues/1384,869071167,MDEyOklzc3VlQ29tbWVudDg2OTA3MTE2Nw==,9599,simonw,2021-06-26T22:55:36Z,2021-06-26T22:55:36Z,OWNER,"Just realized I already have an issue open for this, at #860. I'm going to close that and continue work on this in this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869070941,https://api.github.com/repos/simonw/datasette/issues/1384,869070941,MDEyOklzc3VlQ29tbWVudDg2OTA3MDk0MQ==,9599,simonw,2021-06-26T22:53:34Z,2021-06-26T22:53:34Z,OWNER,"The `await` thing is worrying me a lot - it feels like this plugin hook is massively less useful if it can't make it's own DB queries and generally do asynchronous stuff - but I'd also like not to break every existing plugin that calls `datasette.metadata(...)`.

One solution that could work: introduce a new method, maybe `await datasette.get_metadata(...)`, which uses this plugin hook - and keep the existing `datasette.metadata()` method (which doesn't call the hook) around. This would ensure existing plugins keep on working.

Then, upgrade those plugins separately - with the goal of deprecating and removing `.metadata()` entirely in Datasette 1.0 - having upgraded the plugins in the meantime.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869070348,https://api.github.com/repos/simonw/datasette/issues/1384,869070348,MDEyOklzc3VlQ29tbWVudDg2OTA3MDM0OA==,9599,simonw,2021-06-26T22:46:18Z,2021-06-26T22:46:18Z,OWNER,"Here's where the plugin hook is called, demonstrating the `fallback=` argument: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472

I'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869070076,https://api.github.com/repos/simonw/datasette/issues/1384,869070076,MDEyOklzc3VlQ29tbWVudDg2OTA3MDA3Ng==,9599,simonw,2021-06-26T22:42:21Z,2021-06-26T22:42:21Z,OWNER,"Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.

@brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869069926,https://api.github.com/repos/simonw/datasette/issues/1384,869069926,MDEyOklzc3VlQ29tbWVudDg2OTA2OTkyNg==,9599,simonw,2021-06-26T22:40:15Z,2021-06-26T22:40:53Z,OWNER,"The documentation says:

> **datasette**:     You can use this to access plugin configuration options via `datasette.plugin_config(your_plugin_name)`, or to execute SQL queries.

That's not accurate: since the plugin hook is a regular function, not an awaitable, you can't use it to run `await db.execute(...)` so you can't execute SQL queries.

I can fix this with the await-me-maybe pattern, used for other plugin hooks: https://simonwillison.net/2020/Sep/2/await-me-maybe/

BUT... that requires changing the `ds.metadata()` function to be awaitable, which will affect every existing plugn that uses that documented internal method!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869069768,https://api.github.com/repos/simonw/datasette/issues/1384,869069768,MDEyOklzc3VlQ29tbWVudDg2OTA2OTc2OA==,9599,simonw,2021-06-26T22:37:53Z,2021-06-26T22:37:53Z,OWNER,The documentation doesn't describe the ``fallback`` argument at the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869069655,https://api.github.com/repos/simonw/datasette/issues/1384,869069655,MDEyOklzc3VlQ29tbWVudDg2OTA2OTY1NQ==,9599,simonw,2021-06-26T22:36:14Z,2021-06-26T22:37:37Z,OWNER,"Documentation for the new hook is now live at https://docs.datasette.io/en/latest/plugin_hooks.html#get-metadata-datasette-key-database-table-fallback

Link to the current snapshot of that documentation: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/docs/plugin_hooks.rst#get-metadata-datasette-key-database-table-fallback","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/pull/1368#issuecomment-869068554,https://api.github.com/repos/simonw/datasette/issues/1368,869068554,MDEyOklzc3VlQ29tbWVudDg2OTA2ODU1NA==,9599,simonw,2021-06-26T22:23:57Z,2021-06-26T22:23:57Z,OWNER,The only test failure is Black. I'm going to merge this and then reformat.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,