{"html_url": "https://github.com/simonw/datasette/issues/1641#issuecomment-1049879118", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1641", "id": 1049879118, "node_id": "IC_kwDOBm6k_c4-k-JO", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-02-24T13:49:26Z", "updated_at": "2022-02-24T13:49:26Z", "author_association": "CONTRIBUTOR", "body": "maybe worth considering adding buttons for paren, asterisk, etc. under the input text box on mobile?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1149310456, "label": "Tweak mobile keyboard settings"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1401#issuecomment-884910320", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1401", "id": 884910320, "node_id": "IC_kwDOBm6k_c40vqjw", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-07-22T13:26:01Z", "updated_at": "2021-07-22T13:26:01Z", "author_association": "CONTRIBUTOR", "body": "ordered lists didn't work either, btw", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 950664971, "label": "unordered list is not rendering bullet points in description_html on database page"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-904982056", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 904982056, "node_id": "IC_kwDOBm6k_c418O4o", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2021-08-24T21:15:04Z", "updated_at": "2021-08-24T21:15:30Z", "author_association": "CONTRIBUTOR", "body": "I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1419#issuecomment-892276385", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1419", "id": 892276385, "node_id": "IC_kwDOBm6k_c41Lw6h", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-08-04T00:58:49Z", "updated_at": "2021-08-04T00:58:49Z", "author_association": "CONTRIBUTOR", "body": "yes, [filter clause on aggregate queries were added to sqlite3 in 3.30](https://www.sqlite.org/releaselog/3_30_1.html)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959710008, "label": "`publish cloudrun` should deploy a more recent SQLite version"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1419#issuecomment-893114612", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1419", "id": 893114612, "node_id": "IC_kwDOBm6k_c41O9j0", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-08-05T02:29:06Z", "updated_at": "2021-08-05T02:29:06Z", "author_association": "CONTRIBUTOR", "body": "there's a lot of complexity here, that's probably not worth addressing. i got what i needed by patching the dockerfile that cloudrun uses to install a newer version of sqlite.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959710008, "label": "`publish cloudrun` should deploy a more recent SQLite version"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1425#issuecomment-895003796", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1425", "id": 895003796, "node_id": "IC_kwDOBm6k_c41WKyU", "user": {"value": 3243482, "label": "abdusco"}, "created_at": "2021-08-09T07:14:35Z", "updated_at": "2021-08-09T07:14:35Z", "author_association": "CONTRIBUTOR", "body": "I believe this also provides a workaround for the problem I face in https://github.com/simonw/datasette/issues/1300. \r\n\r\nNow I should be able to get table PKs and generate a row URL. I'll test this out and report my findings.\r\n\r\n\r\n```py\r\nfrom datasette.utils import path_from_row_pks\r\n\r\npks = await db.primary_keys(table)\r\nurl = self.ds.urls.row_blob(\r\n database,\r\n table,\r\n path_from_row_pks(row, pks, not pks),\r\n column,\r\n)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963528457, "label": "render_cell() hook should support returning an awaitable"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-905899177", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 905899177, "node_id": "IC_kwDOBm6k_c41_uyp", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2021-08-25T21:48:00Z", "updated_at": "2021-08-25T21:48:00Z", "author_association": "CONTRIBUTOR", "body": "Upon first stab, there's two issues here:\r\n- DB/table/row counts (as discussed above). This isn't too bad if the DBs are actually above the MAX limit check.\r\n- Populating the internal DB. On first load of a giant set of DBs, it can take 10-20 mins to populate. By altering datasette and persisting the internal DB to disk, this problem is vastly improved, but I'm sure this will cause problems elsewhere.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-905904540", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 905904540, "node_id": "IC_kwDOBm6k_c41_wGc", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2021-08-25T21:59:14Z", "updated_at": "2021-08-25T21:59:55Z", "author_association": "CONTRIBUTOR", "body": "I did two tests: one with 1000 5-30mb DBs and a second with 20 multi gig DBs. For the second, I created them like so:\r\n`for i in {1..20}; do sqlite-generate db$i.db --tables ${i}00 --rows 100,2000 --columns 5,100 --pks 0 --fks 0; done`\r\n\r\nThis was for deciding whether to use lots of small DBs or to group things into a smaller number of bigger DBs. The second strategy wins.\r\n\r\nBy simply persisting the `_internal` DB to disk, I was able to avoid most of the performance issues I was experiencing previously. (To do this, I changed the `datasette/internal_db.py:init_internal_db` creates to if not exists, and changed the `_internal` DB instantiation in `datasette/app.py:Datasette.__init__` to a path with `is_mutable=True`.) Super rough, but the pages now load so I can continue testing ideas.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1473#issuecomment-922363640", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1473", "id": 922363640, "node_id": "IC_kwDOBm6k_c42-ib4", "user": {"value": 192568, "label": "mroswell"}, "created_at": "2021-09-18T19:45:47Z", "updated_at": "2021-09-18T19:45:47Z", "author_association": "CONTRIBUTOR", "body": "An update, if I remove the `img` tag and replace it with the text, \"Safer or Toxic?\" it links to the right place.\r\n\r\nAlso, if I keep things exactly as they are, and it improperly, but consistently goes to the `undefined` page, on THAT 404 page, a click on the image properly clicks through to the www.SaferOrToxic.org page.\r\n\r\nWeird stuff.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 999902754, "label": "base logo link visits `undefined` rather than href url"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1473#issuecomment-922394999", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1473", "id": 922394999, "node_id": "IC_kwDOBm6k_c42-qF3", "user": {"value": 192568, "label": "mroswell"}, "created_at": "2021-09-19T00:44:39Z", "updated_at": "2021-09-19T00:45:32Z", "author_association": "CONTRIBUTOR", "body": "I replaced:\r\n```\r\n\r\n![]()
\r\n\r\n```\r\nwith:\r\n```\r\n![](\"https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp\")
\r\n```\r\n\r\nI'd still love to know what caused this (and how to troubleshoot to figure it out), so I'll leave it open for a bit, but I do have a functional logo linking to the Hugo home page, at least locally. I'll likely push tomorrow.\r\n\r\n(Before trying this, I tried to apply a background image to the `a` tag. That didn't work.)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 999902754, "label": "base logo link visits `undefined` rather than href url"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1455#issuecomment-913001282", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1455", "id": 913001282, "node_id": "IC_kwDOBm6k_c42a0tC", "user": {"value": 51016, "label": "ctb"}, "created_at": "2021-09-04T16:31:24Z", "updated_at": "2021-09-04T16:31:24Z", "author_association": "CONTRIBUTOR", "body": "I love it! maybe 'researchers' instead? Or 'scientists and researchers'?", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 988325628, "label": "Add scientists to target groups"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1464#issuecomment-915279711", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1464", "id": 915279711, "node_id": "IC_kwDOBm6k_c42jg9f", "user": {"value": 51016, "label": "ctb"}, "created_at": "2021-09-08T14:16:49Z", "updated_at": "2021-09-08T14:16:49Z", "author_association": "CONTRIBUTOR", "body": "on commit d57ab156b35ec642", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 991191951, "label": "clean checkout & clean environment has test failures"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1464#issuecomment-915299013", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1464", "id": 915299013, "node_id": "IC_kwDOBm6k_c42jlrF", "user": {"value": 7476523, "label": "bobwhitelock"}, "created_at": "2021-09-08T14:40:28Z", "updated_at": "2021-09-08T14:40:28Z", "author_association": "CONTRIBUTOR", "body": "What are the full errors you're getting?\r\n\r\nThis *may* be the same issue as described in https://github.com/simonw/datasette/pull/1223 - essentially the test suite (and corresponding Datasette features I assume) are by default implicitly dependent on your Sqlite installation having been compiled with the `SQLITE_ENABLE_FTS3_PARENTHESIS` option. If this is the same issue then I think this can be fixed either by recompiling with that option or (probably more easily) by running `pip install pysqlite3-binary`, which will be used in preference to your system Sqlite installation and has this option enabled.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 991191951, "label": "clean checkout & clean environment has test failures"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1464#issuecomment-915302885", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1464", "id": 915302885, "node_id": "IC_kwDOBm6k_c42jmnl", "user": {"value": 51016, "label": "ctb"}, "created_at": "2021-09-08T14:44:50Z", "updated_at": "2021-09-08T14:44:50Z", "author_association": "CONTRIBUTOR", "body": "thanks for the response! full errors attached; excerpt:\r\n\r\n```\r\n...\r\n\r\n\r\n def test_searchmode(table_metadata, querystring, expected_rows):\r\n with make_app_client(\r\n metadata={\"databases\": {\"fixtures\": {\"tables\": {\"searchable\": table_metadata}}}}\r\n ) as client:\r\n response = client.get(\"/fixtures/searchable.json?\" + querystring)\r\n> assert expected_rows == response.json[\"rows\"]\r\nE AssertionError: assert [[1, 'barry c...sel', 'puma']] == []\r\nE Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']\r\nE Use -v to get the full diff\r\n\r\n/Users/t/dev/datasette/tests/test_api.py:1115: AssertionError\r\n```\r\n\r\n[errors.txt](https://github.com/simonw/datasette/files/7129719/errors.txt)\r\n\r\nA quick scan of #1223 suggests you're right. Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 991191951, "label": "clean checkout & clean environment has test failures"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1464#issuecomment-915343886", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1464", "id": 915343886, "node_id": "IC_kwDOBm6k_c42jwoO", "user": {"value": 7476523, "label": "bobwhitelock"}, "created_at": "2021-09-08T15:32:06Z", "updated_at": "2021-09-08T15:32:06Z", "author_association": "CONTRIBUTOR", "body": "Thanks, that does look similar!\r\n\r\n> Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.\r\n\r\nAh that makes sense, I guess that's why this isn't just always installed already. I wonder if a possible solution to this issue could be doing feature detection on whether this feature is supported by the current Sqlite version, and if not these tests could be disabled locally? But possibly there's a better way to handle this, will see what @simonw thinks", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 991191951, "label": "clean checkout & clean environment has test failures"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1464#issuecomment-917642487", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1464", "id": 917642487, "node_id": "IC_kwDOBm6k_c42shz3", "user": {"value": 51016, "label": "ctb"}, "created_at": "2021-09-12T14:03:09Z", "updated_at": "2021-09-12T14:03:09Z", "author_association": "CONTRIBUTOR", "body": "haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 991191951, "label": "clean checkout & clean environment has test failures"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1464#issuecomment-918621705", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1464", "id": 918621705, "node_id": "IC_kwDOBm6k_c42wQ4J", "user": {"value": 7476523, "label": "bobwhitelock"}, "created_at": "2021-09-13T22:17:17Z", "updated_at": "2021-09-13T22:17:17Z", "author_association": "CONTRIBUTOR", "body": "> haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?\r\n\r\nNot certain, but I think tests in CI run on Ubuntu and don't appear to install any additional Sqlite-related dependencies, and so my guess is the version of Sqlite installed by default on Ubuntu has the `SQLITE_ENABLE_FTS3_PARENTHESIS` option enabled and so doesn't run into this issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 991191951, "label": "clean checkout & clean environment has test failures"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1453#issuecomment-919135732", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1453", "id": 919135732, "node_id": "IC_kwDOBm6k_c42yOX0", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2021-09-14T13:10:38Z", "updated_at": "2021-09-14T13:10:38Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #1471.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 982780906, "label": "Bump black from 21.7b0 to 21.8b0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1480#issuecomment-938171377", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1480", "id": 938171377, "node_id": "IC_kwDOBm6k_c4361vx", "user": {"value": 110420, "label": "ghing"}, "created_at": "2021-10-07T21:33:12Z", "updated_at": "2021-10-07T21:33:12Z", "author_association": "CONTRIBUTOR", "body": "Thanks for the reply @simonw. What services have you had better success with than Cloud Run for larger database?\r\n\r\nAlso, what about my issue description makes you think there may be a workaround?\r\n\r\nIs there any instrumentation I could add to see at which point in the deploy the memory usage spikes? Should I be able to see this whether it's running under Docker locally, or do you suspect this is Cloud Run-specific?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1015646369, "label": "Exceeding Cloud Run memory limits when deploying a 4.8G database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1380#issuecomment-953334718", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1380", "id": 953334718, "node_id": "IC_kwDOBm6k_c440ru-", "user": {"value": 813732, "label": "glasnt"}, "created_at": "2021-10-27T21:45:04Z", "updated_at": "2021-10-27T21:45:04Z", "author_association": "CONTRIBUTOR", "body": "I am also getting this issue, using the currently most recent version of datasette\r\n\r\n```\r\n$ datasette --version\r\ndatasette, version 0.59.1\r\n```\r\n\r\nIf I run `datasette` within just a folder of files, \r\n\r\n```\r\n$ datasette serve .\r\n```\r\n\r\nAdding new files while datasette is running shows no new files, and removing files causes datasette to return 500 errors. \r\n\r\n\r\n```\r\nhome\r\nError 500\r\n[Errno 2] No such file or directory: 'mydatabase.db'\r\nPowered by Datasette\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924748955, "label": "Serve all db files in a folder"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1380#issuecomment-953366110", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1380", "id": 953366110, "node_id": "IC_kwDOBm6k_c440zZe", "user": {"value": 813732, "label": "glasnt"}, "created_at": "2021-10-27T22:48:55Z", "updated_at": "2021-10-27T22:48:55Z", "author_association": "CONTRIBUTOR", "body": "It looks like if the files argument is a directory, `config_dir` is set, but files in that folder are only loaded into `self.files` at the `Datasette` class initialisation. \r\n\r\nI tried seeing if I could get `--reload` to work, but I'm getting issues trying to use that command when specifying a directory, as the command `serve` ends up in the files list(?): \r\n\r\n```\r\ndatasette serve . --reload\r\nError: Invalid value for '[FILES]...': Path 'serve' does not exist.\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924748955, "label": "Serve all db files in a folder"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1495#issuecomment-954384496", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1495", "id": 954384496, "node_id": "IC_kwDOBm6k_c444sBw", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-10-29T03:07:13Z", "updated_at": "2021-10-29T03:07:13Z", "author_association": "CONTRIBUTOR", "body": "okay @simonw, made the requested changes. tests are running locally. i think this is ready for you to look at again.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1033678984, "label": "Allow routes to have extra options"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1489#issuecomment-943594735", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1489", "id": 943594735, "node_id": "IC_kwDOBm6k_c44Phzv", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2021-10-14T18:04:12Z", "updated_at": "2021-10-14T18:04:12Z", "author_association": "CONTRIBUTOR", "body": "Looks like this PR is closed. If you re-open it I'll rebase it as long as no-one else has edited it (you can use `@dependabot reopen` if the branch has been deleted).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1026379132, "label": "Update pyyaml requirement from ~=5.3 to >=5.3,<7.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1489#issuecomment-943594738", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1489", "id": 943594738, "node_id": "IC_kwDOBm6k_c44Phzy", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2021-10-14T18:04:13Z", "updated_at": "2021-10-14T18:04:13Z", "author_association": "CONTRIBUTOR", "body": "OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting `@dependabot ignore this major version` or `@dependabot ignore this minor version`. You can also ignore all major, minor, or patch releases for a dependency by adding an [`ignore` condition](https://docs.github.com/en/code-security/supply-chain-security/configuration-options-for-dependency-updates#ignore) with the desired `update_types` to your config file.\n\nIf you change your mind, just re-open this PR and I'll resolve any conflicts on it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1026379132, "label": "Update pyyaml requirement from ~=5.3 to >=5.3,<7.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1432#issuecomment-946255239", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1432", "id": 946255239, "node_id": "IC_kwDOBm6k_c44ZrWH", "user": {"value": 192568, "label": "mroswell"}, "created_at": "2021-10-18T23:55:25Z", "updated_at": "2021-10-18T23:55:25Z", "author_association": "CONTRIBUTOR", "body": "I am getting this when I visit my live Datasette page:\r\n```\r\nThis Serverless Function has crashed.\r\nYour connection is working correctly.\r\nVercel is working correctly.\r\n500: INTERNAL_SERVER_ERROR\r\nCode: FUNCTION_INVOCATION_FAILED\r\nID: ...\r\n```\r\nAnd in the server logs, I'm getting\r\n\r\n```\r\n[GET] /disinfectants/listN\r\n19:53:14:23\r\nmodule initialization error: __init__() got an unexpected keyword argument 'config'\r\nmodule initialization error\r\n__init__() got an unexpected keyword argument 'config'\r\n```\r\n Which is the same error that @ashishdotme reported above.\r\n \r\n \r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 969855774, "label": "Rename Datasette.__init__(config=) parameter to settings="}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1432#issuecomment-946287922", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1432", "id": 946287922, "node_id": "IC_kwDOBm6k_c44ZzUy", "user": {"value": 192568, "label": "mroswell"}, "created_at": "2021-10-19T01:16:41Z", "updated_at": "2021-10-19T01:16:41Z", "author_association": "CONTRIBUTOR", "body": "Resolved, with assistance from @ashishdotme (Thank you!)\r\n\r\nUpdated requirements.txt to include:\r\n```\r\ndatasette==0.59\r\ndatasette-publish-vercel==0.11\r\nsqlite-utils==3.6\r\n```\r\n\r\nRan:\r\n```\r\n$ pip3 install -r requirements.txt\r\n```\r\nThe site is back at work! Yay!\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 969855774, "label": "Rename Datasette.__init__(config=) parameter to settings="}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1396#issuecomment-946467547", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1396", "id": 946467547, "node_id": "IC_kwDOBm6k_c44afLb", "user": {"value": 72577720, "label": "MichaelTiemannOSC"}, "created_at": "2021-10-19T08:10:26Z", "updated_at": "2021-10-19T08:10:26Z", "author_association": "CONTRIBUTOR", "body": "Now that 0.59 has excellent annotated release notes, you can re-confirm this is fixed by updating the published Docker image and checking that these fixes still work ;-)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944903881, "label": "\"invalid reference format\" publishing Docker image"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1480#issuecomment-947196177", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1480", "id": 947196177, "node_id": "IC_kwDOBm6k_c44dRER", "user": {"value": 110420, "label": "ghing"}, "created_at": "2021-10-20T00:05:10Z", "updated_at": "2021-10-20T00:05:10Z", "author_association": "CONTRIBUTOR", "body": "I was looking through the Dockerfile-generation code to see if there was anything that would cause memory usage to be a lot during deployment. \r\n\r\nI noticed that the Dockerfile [runs `datasette --inspect`](https://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L354). Is it possible that this is using a lot of memory usage?\r\n\r\nOr would that come into play when running `gcloud builds submit`, not when it's actually deployed?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1015646369, "label": "Exceeding Cloud Run memory limits when deploying a 4.8G database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1480#issuecomment-947203725", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1480", "id": 947203725, "node_id": "IC_kwDOBm6k_c44dS6N", "user": {"value": 110420, "label": "ghing"}, "created_at": "2021-10-20T00:21:54Z", "updated_at": "2021-10-20T00:21:54Z", "author_association": "CONTRIBUTOR", "body": "This StackOverflow post, [sqlite - Cloud Run: Why does my instance need so much RAM?](https://stackoverflow.com/questions/59812405/cloud-run-why-does-my-instance-need-so-much-ram), points to [this section of the Cloud Run docs](https://cloud.google.com/run/docs/troubleshooting) that says:\r\n\r\n> Note that the Cloud Run container instances run in an environment where the files written to the local filesystem count towards the available memory. This also includes any log files that are not written to /var/log/* or /dev/log.\r\n\r\nDoes datasette write any large files when starting? \r\n\r\nOr does the [`COPY` command in the Dockerfile](https://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L349) count as writing to the local filesystem?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1015646369, "label": "Exceeding Cloud Run memory limits when deploying a 4.8G database"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1284#issuecomment-949604763", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1284", "id": 949604763, "node_id": "IC_kwDOBm6k_c44mdGb", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-10-22T12:54:34Z", "updated_at": "2021-10-22T12:54:34Z", "author_association": "CONTRIBUTOR", "body": "i'm going to take a swing at this today. we'll see.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 845794436, "label": "Feature or Documentation Request: Individual table as home page template"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1401#issuecomment-950150483", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1401", "id": 950150483, "node_id": "IC_kwDOBm6k_c44oiVT", "user": {"value": 418191, "label": "jaywgraves"}, "created_at": "2021-10-23T13:09:10Z", "updated_at": "2021-10-23T13:09:10Z", "author_association": "CONTRIBUTOR", "body": "I think it's because of this in `app.css` \r\n\r\n```\r\nol,\r\nul {\r\n\tlist-style: none;\r\n}\r\n```\r\n\r\nhttps://github.com/simonw/datasette/blame/main/datasette/static/app.css#L35-L38\r\n\r\nYou could probably reinstate that by providing your own CSS.\r\nhttps://docs.datasette.io/en/0.24/custom_templates.html#custom-css-and-javascript", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 950664971, "label": "unordered list is not rendering bullet points in description_html on database page"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1012#issuecomment-970266123", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1012", "id": 970266123, "node_id": "IC_kwDOBm6k_c451RYL", "user": {"value": 45380, "label": "bollwyvl"}, "created_at": "2021-11-16T13:18:36Z", "updated_at": "2021-11-16T13:18:36Z", "author_association": "CONTRIBUTOR", "body": "Congratulations, looks like it went through! There was a bit of a hold-up\non the JupyterLab ones, but it's semi automated: a dependabot pr to\nwarehouse and a CI deploy, with a click in between.\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 718540751, "label": "For 1.0 update trove classifier in setup.py"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1500#issuecomment-971568829", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1500", "id": 971568829, "node_id": "IC_kwDOBm6k_c456Pa9", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2021-11-17T13:13:58Z", "updated_at": "2021-11-17T13:13:58Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #1514.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1041158024, "label": "Bump black from 21.9b0 to 21.10b0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1495#issuecomment-960420237", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1495", "id": 960420237, "node_id": "IC_kwDOBm6k_c45PtmN", "user": {"value": 192568, "label": "mroswell"}, "created_at": "2021-11-04T03:12:01Z", "updated_at": "2021-11-04T03:12:01Z", "author_association": "CONTRIBUTOR", "body": "This all looks promising! I will need detailed documentation on how to upgrade datasette once it's available, and how to implement. (@fgregg example looks very straightforward on the plugin front.) \r\nI'll be so excited if I can get:\r\nhttps://list.saferdisinfectants.org/ \r\ninstead of\r\nhttps://list.saferdisinfectants.org/disinfectants/listN\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1033678984, "label": "Allow routes to have extra options"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1514#issuecomment-972852184", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1514", "id": 972852184, "node_id": "IC_kwDOBm6k_c45_IvY", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2021-11-18T13:11:15Z", "updated_at": "2021-11-18T13:11:15Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #1516.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1056117435, "label": "Bump black from 21.9b0 to 21.11b0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1380#issuecomment-967747190", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1380", "id": 967747190, "node_id": "IC_kwDOBm6k_c45rqZ2", "user": {"value": 813732, "label": "glasnt"}, "created_at": "2021-11-13T00:47:26Z", "updated_at": "2021-11-13T00:47:26Z", "author_association": "CONTRIBUTOR", "body": "Would it make sense to run datasette with a fswatch/inotifywait on a folder, then? ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924748955, "label": "Serve all db files in a folder"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1495#issuecomment-974108455", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1495", "id": 974108455, "node_id": "IC_kwDOBm6k_c46D7cn", "user": {"value": 192568, "label": "mroswell"}, "created_at": "2021-11-19T14:14:35Z", "updated_at": "2021-11-19T14:14:35Z", "author_association": "CONTRIBUTOR", "body": "A nudge on this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1033678984, "label": "Allow routes to have extra options"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1528#issuecomment-975955589", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1528", "id": 975955589, "node_id": "IC_kwDOBm6k_c46K-aF", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2021-11-22T22:00:30Z", "updated_at": "2021-11-22T22:00:30Z", "author_association": "CONTRIBUTOR", "body": "Oh, another thing to consider: I believe this would be the first `\"_file\"` key in datasette's metadata, compared to other `\"_url\"` keys like `\"license_url\"` or `\"about_url\"`. Not too sure what considerations to include with this (ex should missing files cause Datasette to stop before starting, should build scripts bundle these sql files somewhere during `datasette package`, etc.)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1060631257, "label": "Add new `\"sql_file\"` key to Canned Queries in metadata?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1522#issuecomment-976117989", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1522", "id": 976117989, "node_id": "IC_kwDOBm6k_c46LmDl", "user": {"value": 813732, "label": "glasnt"}, "created_at": "2021-11-23T03:00:34Z", "updated_at": "2021-11-23T03:00:34Z", "author_association": "CONTRIBUTOR", "body": "I tried deploying the most recent version of the Dockerfile in this thread ([link to comment](https://github.com/simonw/datasette/issues/1522#issuecomment-974605128)), and after trying a few different different combinations, I was only successful when I used `--no-cpu-throttling` (\"CPU Is always allocated\" in the UI)\r\n\r\nUsing this method, I got a very similar issue to you: The first time I'd load the site I'd get a 503. But after that first load, I didn't get the issue again. It would re-occur if the service started from cold boot. \r\n\r\nI suspect this is a race condition in the supervisord configuration. The errors I got were the same `Connection refused: AH00957: http: attempt to connect to 127.0.0.1:8001 (127.0.0.1) failed`, and that seems to indicate that `datasette` hadn't yet started. \r\n\r\nLooking at the order of logs getting back, the processes reported successfully completing loading after the first 503 was returned, so that makes me think race condition. \r\n\r\nI can replicate this locally, if I `docker run` and request `localhost:5000/prefix` _before_ I get the `datasette entered RUNNING state` message. Cloud Run wakes up when requests are received, so this test would semi-replicate that, but local docker would be the equivalent of a persistent process, hence it doesn't normally exhibit the same issues.\r\n\r\nUnfortunately supervisor/supervisor issue 122 (not linking as to prevent cross-project link spam) seems to say that dependency chaining is a feature that's been asked for for a long time, but hasn't been implemented. You could try some suggestions in that thread. ", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058896236, "label": "Deploy a live instance of demos/apache-proxy"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1549#issuecomment-991754237", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1549", "id": 991754237, "node_id": "IC_kwDOBm6k_c47HPf9", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-12-11T19:14:39Z", "updated_at": "2021-12-11T19:14:39Z", "author_association": "CONTRIBUTOR", "body": "that option is not available on [custom queries](https://labordata.bunkum.us/odpr-962a140?sql=with+local_union_filings+as+%28%0D%0A++select+*+from+lm_data+%0D%0A++where%0D%0A++++yr_covered+%3E+cast%28strftime%28%27%25Y%27%2C+%27now%27%2C+%27-5+years%27%29+as+int%29%0D%0A++++and+desig_name+%3D+%27LU%27%0D%0A++order+by+yr_covered+desc%0D%0A%29%2C%0D%0Amost_recent_filing+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from+local_union_filings%0D%0A++group+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++most_recent_filing%0D%0Awhere%0D%0A++next_election+%3E%3D+strftime%28%27%25Y-%25m%27%2C+%27now%27%29%0D%0A++and+next_election+%3C+strftime%28%27%25Y-%25m%27%2C+%27now%27%2C+%27%2B1+year%27%29%0D%0Aorder+by%0D%0A++members+desc%3B).\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077620955, "label": "Redesign CSV export to improve usability"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-992971072", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 992971072, "node_id": "IC_kwDOBm6k_c47L4lA", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-12-13T22:29:34Z", "updated_at": "2021-12-13T22:29:34Z", "author_association": "CONTRIBUTOR", "body": "just came by to open this issue. would make my data analysis in observable a lot better!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1553#issuecomment-992986587", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1553", "id": 992986587, "node_id": "IC_kwDOBm6k_c47L8Xb", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-12-13T22:57:04Z", "updated_at": "2021-12-13T22:57:04Z", "author_association": "CONTRIBUTOR", "body": "would also be good if the header said the what the max row limit was", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079111498, "label": "if csv export is truncated in non streaming mode set informative response header"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1553#issuecomment-993014772", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1553", "id": 993014772, "node_id": "IC_kwDOBm6k_c47MDP0", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-12-13T23:46:18Z", "updated_at": "2021-12-13T23:46:18Z", "author_association": "CONTRIBUTOR", "body": "these headers would also be relevant for json exports of custom queries", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079111498, "label": "if csv export is truncated in non streaming mode set informative response header"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-993078038", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 993078038, "node_id": "IC_kwDOBm6k_c47MSsW", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-12-14T01:46:52Z", "updated_at": "2021-12-14T01:46:52Z", "author_association": "CONTRIBUTOR", "body": "the nested query idea is very nice, and i stole if for [my client side paginator](https://observablehq.com/d/1d5da3a3c3f2f347#DatasetteClient). However, it won't do the right thing if the original query orders by random().\r\n\r\nIf you go the nested query route, maybe raise a 4XX status code if the query has such a clause?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1552#issuecomment-995296725", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1552", "id": 995296725, "node_id": "IC_kwDOBm6k_c47UwXV", "user": {"value": 3556, "label": "davidbgk"}, "created_at": "2021-12-15T23:29:32Z", "updated_at": "2021-12-15T23:29:32Z", "author_association": "CONTRIBUTOR", "body": "@simonw thank you for your fast answer and your guidance!\r\n\r\nWhile digging into the code, I found an undocumented way of doing it:\r\n\r\n```yaml\r\nfacets: [\"Facet for a column\", {\"array\": \"Facet for an array\"}]\r\n```\r\n\r\nThe only remaining problem with that solution is here: https://github.com/simonw/datasette/blob/250db8192cb8aba5eb8cd301ccc2a49525bc3d24/datasette/facets.py#L33\r\n\r\nWe have:\r\n\r\n```python\r\ntype, metadata_config = metadata_config.items()[0]\r\n```\r\n\r\nBut it requires to cast the `dict_items` as a list prior to access the first element:\r\n\r\n```python\r\ntype, metadata_config = list(metadata_config.items())[0]\r\n```\r\n\r\nI guess it's an unspotted bug? (I mean, independently of the facets-with-arrays issue.)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1078702875, "label": "Allow to set `facets_array` in metadata (like current `facets`)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1552#issuecomment-996229007", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1552", "id": 996229007, "node_id": "IC_kwDOBm6k_c47YT-P", "user": {"value": 3556, "label": "davidbgk"}, "created_at": "2021-12-16T22:04:39Z", "updated_at": "2021-12-16T22:04:39Z", "author_association": "CONTRIBUTOR", "body": "Wow, that was fast, thank you so much @simonw !\r\n\r\n> I'm also not convinced that this configuration syntax is right. It's a bit weird having a `\"facets\"` list that can either by column-name-strings or `{\"type-of-facet\": \"column-name\"}` objects. Maybe there's a better design for this?\r\n\r\nI agree that it's not ideal, my initial naive approach was to detect if it's an array, like what is done here:\r\n\r\nhttps://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/facets.py#L312-L313\r\n\r\nBut it requires an extra query to determine the type, which is a bit problematic, especially for big tables I guess.\r\n\r\nTaking a look at #510, I wonder if a `facet_delimiter` should be defined for that kind of columns (that would help our team not to have an intermediary conversion step from `foo|bar` to `[\"foo\",\"bar\"]` for instance).\r\n\r\nTo be consistent with the `--extract-column` parameter, maybe an explicit casting/delimiter would be useful: `--set-column 'Foo:Array:|'`.\r\n\r\nThrowing a lot of ideas without knowing the big picture\u2026 but sometimes newcomers have superpowers :).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1078702875, "label": "Allow to set `facets_array` in metadata (like current `facets`)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1561#issuecomment-997128712", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1561", "id": 997128712, "node_id": "IC_kwDOBm6k_c47bvoI", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-12-18T02:35:48Z", "updated_at": "2021-12-18T02:35:48Z", "author_association": "CONTRIBUTOR", "body": "interesting! i love this feature. this + full caching with cloudflare is really super!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1082765654, "label": "add hash id to \"_memory\" url if hashed url mode is turned on and crossdb is also turned on"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1547#issuecomment-997511968", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1547", "id": 997511968, "node_id": "IC_kwDOBm6k_c47dNMg", "user": {"value": 127565, "label": "wragge"}, "created_at": "2021-12-20T01:21:59Z", "updated_at": "2021-12-20T01:21:59Z", "author_association": "CONTRIBUTOR", "body": "I've installed the alpha version but get an error when starting up Datasette:\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/Users/tim/.pyenv/versions/stock-exchange/bin/datasette\", line 5, in \r\n from datasette.cli import cli\r\n File \"/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/cli.py\", line 15, in \r\n from .app import Datasette, DEFAULT_SETTINGS, SETTINGS, SQLITE_LIMIT_ATTACHED, pm\r\n File \"/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/app.py\", line 31, in \r\n from .views.database import DatabaseDownload, DatabaseView\r\n File \"/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/views/database.py\", line 25, in \r\n from datasette.plugins import pm\r\n File \"/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/plugins.py\", line 29, in \r\n mod = importlib.import_module(plugin)\r\n File \"/Users/tim/.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py\", line 127, in import_module\r\n return _bootstrap._gcd_import(name[level:], package, level)\r\n File \"/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/filters.py\", line 9, in \r\n @hookimpl(specname=\"filters_from_request\")\r\nTypeError: __call__() got an unexpected keyword argument 'specname'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1076388044, "label": "Writable canned queries fail to load custom templates"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1547#issuecomment-997519202", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1547", "id": 997519202, "node_id": "IC_kwDOBm6k_c47dO9i", "user": {"value": 127565, "label": "wragge"}, "created_at": "2021-12-20T01:36:58Z", "updated_at": "2021-12-20T01:36:58Z", "author_association": "CONTRIBUTOR", "body": "Yep, that works -- thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1076388044, "label": "Writable canned queries fail to load custom templates"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1583#issuecomment-1002825217", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1583", "id": 1002825217, "node_id": "IC_kwDOBm6k_c47xeYB", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-12-30T00:34:16Z", "updated_at": "2021-12-30T00:34:16Z", "author_association": "CONTRIBUTOR", "body": "if that is not desirable, it might be good to document that users might want to set up a lifecycle rule to automatically delete these build artifacts. something like https://stackoverflow.com/questions/59937542/can-i-delete-container-images-from-google-cloud-storage-artifacts-bucket", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1090810196, "label": "consider adding deletion step of cloudbuild artifacts to gcloud publish"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1612#issuecomment-1021497165", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1612", "id": 1021497165, "node_id": "IC_kwDOBm6k_c484s9N", "user": {"value": 639012, "label": "jsfenfen"}, "created_at": "2022-01-25T18:44:23Z", "updated_at": "2022-01-25T18:44:23Z", "author_association": "CONTRIBUTOR", "body": "OMG, this might be the fastest OS ticket I've ever filed, thanks so much @simonw ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1114147905, "label": "Move canned queries closer to the SQL input area"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1574#issuecomment-1007844190", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1574", "id": 1007844190, "node_id": "IC_kwDOBm6k_c48Ente", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-01-08T00:42:12Z", "updated_at": "2022-01-08T00:42:12Z", "author_association": "CONTRIBUTOR", "body": "is there a reason to not always use the slim option?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1084193403, "label": "introduce new option for datasette package to use a slim base image"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1591#issuecomment-1010947634", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1591", "id": 1010947634, "node_id": "IC_kwDOBm6k_c48QdYy", "user": {"value": 82988, "label": "psychemedia"}, "created_at": "2022-01-12T11:32:17Z", "updated_at": "2022-01-12T11:32:17Z", "author_association": "CONTRIBUTOR", "body": "Is it possible to parse things like `--ext-{plugin}-{arg} VALUE` ?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1100015398, "label": "Maybe let plugins define custom serve options?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1601#issuecomment-1016651485", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1601", "id": 1016651485, "node_id": "IC_kwDOBm6k_c48mN7d", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-01-19T16:39:03Z", "updated_at": "2022-01-19T16:39:03Z", "author_association": "CONTRIBUTOR", "body": "I think both of these are Spatialite specific. They get generated when you first initialize the extension. KNN is actually deprecated in favor of [KNN2](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN2), as I understand it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1105916061, "label": "Add KNN and data_licenses to hidden tables list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1605#issuecomment-1016994329", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1605", "id": 1016994329, "node_id": "IC_kwDOBm6k_c48nhoZ", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-01-20T00:27:17Z", "updated_at": "2022-01-20T00:27:17Z", "author_association": "CONTRIBUTOR", "body": "Right now, I usually have a line in a Makefile like this:\r\n\r\n```make\r\ncombined.geojson: project.db\r\n pipenv run datasette project.db --get /project/combined.geojson \\\r\n --load-extension spatialite \\\r\n --setting sql_time_limit_ms 5000 \\\r\n --setting max_returned_rows 20000 \\\r\n -m metadata.yml > $@\r\n```\r\n\r\nThat all assumes I've loaded whatever I need into `project.db` and created a canned query called `combined` (and then uses `datasette-geojson` for geojson output). \r\n\r\nIt works, but as you can see, it's a lot to manage, a lot of boilerplate, and it wasn't obvious how to get there. If there's an error in the canned query, I get an HTML error page, so that's hard to debug. And it's only one query, so each output needs a line like this. Make isn't ideal, either, for that reason.\r\n\r\nThe thing I really liked with `datafreeze` was doing templated filenames. I have a project now where I need to export a bunch of litttle geojson files, based on queries, and it would be awesome to be able to do something like this:\r\n\r\n```yml\r\ndatabases:\r\n project:\r\n queries:\r\n boundaries:\r\n sql: \"SELECT * FROM boundaries\"\r\n filename: \"boundaries/{id}.geojson\"\r\n mode: \"item\"\r\n format: geojson\r\n```\r\n\r\nAnd then do:\r\n\r\n```sh\r\ndatasette freeze -m metadata.yml project.db\r\n```\r\n\r\nFor HTML export, maybe there's a `template` argument, or `format: template` or something. And that gets you a static site generator, kinda for free.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1108671952, "label": "Scripted exports"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1605#issuecomment-1018741262", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1605", "id": 1018741262, "node_id": "IC_kwDOBm6k_c48uMIO", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-01-21T18:05:09Z", "updated_at": "2022-01-21T18:05:09Z", "author_association": "CONTRIBUTOR", "body": "Thinking about this more, as well as #1356 and various other tickets related to output formats, I think there's a missing plugin hook for formatting results, separate from `register_output_renderer` (or maybe part of it, depending on #1101). \r\n\r\nRight now, as I understand it, getting output in any format goes through the normal view stack -- a table, a row or a query -- and so by the time `register_output_renderer` gets it, the results have already been truncated or paginated. What I'd want, I think, is to be able to register ways to format results independent of where those results are sent.\r\n\r\nIt's possible this could be done using [`conn.row_factory`](https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.row_factory) (maybe in the `prepare_connection` hook), but I'm not sure that's where it belongs.\r\n\r\nAnother option is some kind of registry of serializers, which `register_output_renderer` and other plugin hooks could use. What I'm trying to avoid here is writing a plugin that also needs plugins for formats I haven't thought of yet.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1108671952, "label": "Scripted exports"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1605#issuecomment-1018778667", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1605", "id": 1018778667, "node_id": "IC_kwDOBm6k_c48uVQr", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-01-21T19:00:01Z", "updated_at": "2022-01-21T19:00:01Z", "author_association": "CONTRIBUTOR", "body": "Let me know if you want help prototyping any of this, because I'm thinking about it and trying stuff out. Happy to be a sounding board, if it helps.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1108671952, "label": "Scripted exports"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1593#issuecomment-1031455498", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1593", "id": 1031455498, "node_id": "IC_kwDOBm6k_c49esMK", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2022-02-07T13:13:22Z", "updated_at": "2022-02-07T13:13:22Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #1631.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1101705012, "label": "Update pytest-asyncio requirement from <0.17,>=0.10 to >=0.10,<0.18"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1655#issuecomment-1062450649", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1655", "id": 1062450649, "node_id": "IC_kwDOBm6k_c4_U7XZ", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-03-09T01:10:46Z", "updated_at": "2022-03-09T01:10:46Z", "author_association": "CONTRIBUTOR", "body": "i increased the max_returned_row, because I have some scripts that get CSVs from this site, and this makes doing pagination of CSVs less annoying for many cases. i know that's streaming csvs is something you are hoping to address in 1.0. let me know if there's anything i can do to help with that.\r\n\r\nas for what if anything can be done about the size of the dom, I don't have any ideas right now, but i'll poke around.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1163369515, "label": "query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-1065940779", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 1065940779, "node_id": "IC_kwDOBm6k_c4_iPcr", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2022-03-12T18:49:29Z", "updated_at": "2022-03-12T18:50:07Z", "author_association": "CONTRIBUTOR", "body": "Hello! Just wanted to chime in and note that there's a plugin to have Datasette [watch for updates to an external metadata.yaml/json and update the internal settings accordingly](https://datasette.io/plugins/datasette-remote-metadata), so I think the cache/poll use case is already covered. @khusmann If you don't need truly dynamic metadata then what you've come up with or the plugin ought to work fine.\r\n\r\nMaking the get_metadata async won't improve the situation by itself as only some of the code paths accessing metadata use that hook. The other paths use the internal metadata dict. Trying to force all paths through a async hook would have performance ramifications and making everything use the internal meta will cause problems for users that need changes to take effect immediately. This is why I came to the non-async solution as it was the path of least change within Datasette. As always, open to new ideas, etc!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-1066006292", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 1066006292, "node_id": "IC_kwDOBm6k_c4_ifcU", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2022-03-13T02:09:44Z", "updated_at": "2022-03-13T02:09:44Z", "author_association": "CONTRIBUTOR", "body": "> If I'm understanding your plugin code correctly, you query the db using the sync handle every time `get_metdata` is called, right? Won't this become a pretty big bottleneck if a hook into `render_cell` is trying to read metadata / plugin config?\r\n\r\nReading from sqlite DBs is pretty quick and I didn't notice significant performance issues when I was benchmarking. I tested on very large Datasette deployments (hundreds of DBs, millions of rows). See [\"Many small queries are efficient in sqlite\"](https://sqlite.org/np1queryprob.html) for more information on the rationale here. Also note that in the [datasette-live-config](https://github.com/next-LI/datasette-live-config) reference plugin, the DB connection is cached, so that eliminated most of the performance worries we had.\r\n\r\nIf you need to ensure fresh metadata is being read inside of a `render_cell` hook specifically, you don't need to do anything further! `get_metadata` gets called before `render_cell` every request, so it already has access to the synced meta. There shouldn't be a need to call `get_metadata(...)` or `metadata(...)` inside `render_cell`, you can just use `datasette._metadata_local` if you're really worried about performance.\r\n\r\n> The plugin is close, but looks like it only grabs remote metadata, is that right? Instead what I'm wanting is to grab metadata embedded in the attached databases.\r\n\r\nYes correct, the datadette-remote-metadata plugin doesn't do that. But the datasette-live-config plugin does. [It supports a `__metadata` table](https://github.com/next-LI/datasette-live-config/blob/main/datasette_live_config/__init__.py#L107-L138) that, when it exists on an attached DB, gets pulled into the Datasette internal `_metadata` and is also accessible via `get_metadata`. Updating is instantaneous so there's no gotchas for users or security issues for users relying on the metadata-based permissions. Simon talked about eventually making something like this a standard feature of Datasette, but I'm not sure what the status is on that!\r\n\r\nGood luck!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-1066169718", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 1066169718, "node_id": "IC_kwDOBm6k_c4_jHV2", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2022-03-13T19:48:49Z", "updated_at": "2022-03-13T19:48:49Z", "author_association": "CONTRIBUTOR", "body": "> For my reference, did you include a `render_cell` plugin calling `get_metadata` in those tests?\r\n\r\nYou shouldn't need to do this, as I mentioned previously. The code inside `render_cell` hook already has access to the most recently sync'd metadata via `datasette._metadata_local`. Refreshing the metadata for every cell seems ... excessive.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-1066222323", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 1066222323, "node_id": "IC_kwDOBm6k_c4_jULz", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2022-03-14T00:36:42Z", "updated_at": "2022-03-14T00:36:42Z", "author_association": "CONTRIBUTOR", "body": "> Ah, sorry, I didn't get what you were saying you the first time. Using _metadata_local in that way makes total sense -- I agree, refreshing metadata each cell was seeming quite excessive. Now I'm on the same page! :)\r\n\r\nAll good. Report back any issues you find with this stuff. Metadata/dynamic config hasn't been tested widely outside of what I've done AFAIK. If you find a strong use case for async meta, it's going to be better to know sooner rather than later!", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1549#issuecomment-1087428593", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1549", "id": 1087428593, "node_id": "IC_kwDOBm6k_c5A0Nfx", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-04-04T11:17:13Z", "updated_at": "2022-04-04T11:17:13Z", "author_association": "CONTRIBUTOR", "body": "another way to get the behavior of downloading the file is to use the download attribute of the anchor tag\r\n\r\nhttps://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#attr-download", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077620955, "label": "Redesign CSV export to improve usability"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1582#issuecomment-1077047152", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1582", "id": 1077047152, "node_id": "IC_kwDOBm6k_c5AMm9w", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-03-24T04:07:58Z", "updated_at": "2022-03-24T04:07:58Z", "author_association": "CONTRIBUTOR", "body": "this has been obviated by the datasette-hashed-urls plugin", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1090055810, "label": "don't set far expiry if hash is '000'"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1581#issuecomment-1077047295", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1581", "id": 1077047295, "node_id": "IC_kwDOBm6k_c5AMm__", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-03-24T04:08:18Z", "updated_at": "2022-03-24T04:08:18Z", "author_association": "CONTRIBUTOR", "body": "this has been addressed by the datasette-hashed-urls plugin", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1089529555, "label": "when hashed urls are turned on, the _memory db has improperly long-lived cache expiry"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1684#issuecomment-1078126065", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1684", "id": 1078126065, "node_id": "IC_kwDOBm6k_c5AQuXx", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-03-24T20:08:56Z", "updated_at": "2022-03-24T20:13:19Z", "author_association": "CONTRIBUTOR", "body": "would be nice if the behavior was\r\n\r\n1. try to facet all the columns\r\n2. for bigger tables try to facet the indexed columns\r\n3. for the biggest tables, turn off autofacetting completely\r\n\r\nThis is based on my assumption that what determines autofaceting is the rarity of unique values. Which may not be true!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1179998071, "label": "Mechanism for disabling faceting on large tables only"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1101#issuecomment-1105588651", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1101", "id": 1105588651, "node_id": "IC_kwDOBm6k_c5B5fGr", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-04-21T18:15:39Z", "updated_at": "2022-04-21T18:15:39Z", "author_association": "CONTRIBUTOR", "body": "What if you split rendering and streaming into two things:\r\n\r\n- `render` is a function that returns a response\r\n- `stream` is a function that sends chunks, or yields chunks passed to an ASGI `send` callback\r\n\r\nThat way current plugins still work, and streaming is purely additive. A `stream` function could get a cursor or iterator of rows, instead of a list, so it could more efficiently handle large queries.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 749283032, "label": "register_output_renderer() should support streaming data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1101#issuecomment-1105642187", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1101", "id": 1105642187, "node_id": "IC_kwDOBm6k_c5B5sLL", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-04-21T18:59:08Z", "updated_at": "2022-04-21T18:59:08Z", "author_association": "CONTRIBUTOR", "body": "Ha! That was your idea (and a good one).\r\n\r\nBut it's probably worth measuring to see what overhead it adds. It did require both passing in the database and making the whole thing `async`. \r\n\r\nJust timing the queries themselves:\r\n\r\n1. [Using `AsGeoJSON(geometry) as geometry`](https://alltheplaces-datasette.fly.dev/alltheplaces?sql=select%0D%0A++id%2C%0D%0A++properties%2C%0D%0A++AsGeoJSON%28geometry%29+as+geometry%2C%0D%0A++spider%0D%0Afrom%0D%0A++places%0D%0Aorder+by%0D%0A++id%0D%0Alimit%0D%0A++1000) takes 10.235 ms\r\n2. [Leaving as binary](https://alltheplaces-datasette.fly.dev/alltheplaces?sql=select%0D%0A++id%2C%0D%0A++properties%2C%0D%0A++geometry%2C%0D%0A++spider%0D%0Afrom%0D%0A++places%0D%0Aorder+by%0D%0A++id%0D%0Alimit%0D%0A++1000) takes 8.63 ms\r\n\r\nLooking at the network panel:\r\n\r\n1. Takes about 200 ms for the `fetch` request\r\n2. Takes about 300 ms\r\n\r\nI'm not sure how best to time the GeoJSON generation, but it would be interesting to check. Maybe I'll write a plugin to add query times to response headers.\r\n\r\nThe other thing to consider with async streaming is that it might be well-suited for a slower response. When I have to get the whole result and send a response in a fixed amount of time, I need the most efficient query possible. If I can hang onto a connection and get things one chunk at a time, maybe it's ok if there's some overhead.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 749283032, "label": "register_output_renderer() should support streaming data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1699#issuecomment-1092357672", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1699", "id": 1092357672, "node_id": "IC_kwDOBm6k_c5BHA4o", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-04-08T01:39:40Z", "updated_at": "2022-04-08T01:39:40Z", "author_association": "CONTRIBUTOR", "body": "> My best thought on how to differentiate them so far is plugins: if Datasette plugins that provide alternative outputs - like .geojson and .yml and suchlike - also work for the datasette query command that would make a lot of sense to me.\r\n\r\nThat's my thinking, too. It's really the thing I've been wanting since writing `datasette-geojson`, since I'm always exporting with `datasette --get`. The workflow I'm always looking for is something like this:\r\n\r\n```sh\r\ncd alltheplaces-datasette\r\ndatasette query dunkin_in_suffolk -f geojson -o dunkin_in_suffolk.geojson\r\n```\r\n\r\nI think this probably needs either a new plugin hook separate from `register_output_renderer` or a way to use that without going through the HTTP stack. Or maybe a render mode that writes to a stream instead of a response. Maybe there's a new key in the dictionary that `register_output_renderer` returns that handles CLI exports.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1193090967, "label": "Proposal: datasette query"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1699#issuecomment-1092370880", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1699", "id": 1092370880, "node_id": "IC_kwDOBm6k_c5BHEHA", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-04-08T02:07:40Z", "updated_at": "2022-04-08T02:07:40Z", "author_association": "CONTRIBUTOR", "body": "So maybe `render_output_render` returns something like this:\r\n\r\n```python\r\n@hookimpl\r\ndef register_output_renderer(datasette):\r\n return {\r\n \"extension\": \"geojson\",\r\n \"render\": render_geojson,\r\n \"stream\": stream_geojson,\r\n \"can_render\": can_render_geojson,\r\n }\r\n```\r\n\r\nAnd stream gets an iterator, instead of a list of rows, so it can efficiently handle large queries. Maybe it also gets passed a destination stream, or it returns an iterator. I'm not sure what makes more sense. Either way, that might cover both CLI exports and streaming responses.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1193090967, "label": "Proposal: datasette query"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1699#issuecomment-1092386254", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1699", "id": 1092386254, "node_id": "IC_kwDOBm6k_c5BHH3O", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-04-08T02:39:25Z", "updated_at": "2022-04-08T02:39:25Z", "author_association": "CONTRIBUTOR", "body": "And just to think this through a little more, here's what `stream_geojson` might look like:\r\n\r\n```python\r\nasync def stream_geojson(datasette, columns, rows, database, stream):\r\n db = datasette.get_database(database)\r\n for row in rows:\r\n feature = await row_to_geojson(row, db)\r\n stream.write(feature + \"\\n\") # just assuming newline mode for now\r\n```\r\n\r\nAlternately, that could be an async generator, like this:\r\n\r\n```python\r\nasync def stream_geojson(datasette, columns, rows, database):\r\n db = datasette.get_database(database)\r\n for row in rows:\r\n feature = await row_to_geojson(row, db)\r\n yield feature\r\n```\r\n\r\nNot sure which makes more sense, but I think this pattern would open up a lot of possibility. If you had your [stream_indented_json](https://til.simonwillison.net/python/output-json-array-streaming) function, you could do `yield from stream_indented_json(rows, 2)` and be one your way.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1193090967, "label": "Proposal: datasette query"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1699#issuecomment-1094453751", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1699", "id": 1094453751, "node_id": "IC_kwDOBm6k_c5BPAn3", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-04-11T01:32:12Z", "updated_at": "2022-04-11T01:32:12Z", "author_association": "CONTRIBUTOR", "body": "Was looking through old issues and realized a bunch of this got discussed in #1101 (including by me!), so sorry to rehash all this. Happy to help with whatever piece of it I can. Would be very excited to be able to use format plugins with exports.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1193090967, "label": "Proposal: datasette query"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1713#issuecomment-1099540225", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1713", "id": 1099540225, "node_id": "IC_kwDOBm6k_c5BiacB", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-04-14T19:09:57Z", "updated_at": "2022-04-14T19:09:57Z", "author_association": "CONTRIBUTOR", "body": "I wonder if this overlaps with what I outlined in #1605. You could run something like this:\r\n\r\n```sh\r\ndatasette freeze -d exports/\r\naws s3 cp exports/ s3://my-export-bucket/$(date)\r\n```\r\n\r\nAnd maybe that does what you need. Of course, that plugin isn't built yet. But that's the idea.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1203943272, "label": "Datasette feature for publishing snapshots of query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1713#issuecomment-1103312860", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1713", "id": 1103312860, "node_id": "IC_kwDOBm6k_c5Bwzfc", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-04-20T00:52:19Z", "updated_at": "2022-04-20T00:52:19Z", "author_association": "CONTRIBUTOR", "body": "feels related to #1402 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1203943272, "label": "Datasette feature for publishing snapshots of query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1728#issuecomment-1111705323", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1728", "id": 1111705323, "node_id": "IC_kwDOBm6k_c5CQ0br", "user": {"value": 127565, "label": "wragge"}, "created_at": "2022-04-28T03:32:06Z", "updated_at": "2022-04-28T03:32:06Z", "author_association": "CONTRIBUTOR", "body": "Ah, that would be it! I have a core set of data which doesn't change to which I want authorised users to be able to submit corrections. I was going to deal with the persistence issue by just grabbing the user corrections at regular intervals and saving to GitHub. I might need to rethink. Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1218133366, "label": "Writable canned queries fail with useless non-error against immutable databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1728#issuecomment-1111712953", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1728", "id": 1111712953, "node_id": "IC_kwDOBm6k_c5CQ2S5", "user": {"value": 127565, "label": "wragge"}, "created_at": "2022-04-28T03:48:36Z", "updated_at": "2022-04-28T03:48:36Z", "author_association": "CONTRIBUTOR", "body": "I don't think that'd work for this project. The db is very big, and my aim was to have an environment where researchers could be making use of the data, but be easily able to add corrections to the HTR/OCR extracted data when they came across problems. It's in its immutable (!) form here: https://sydney-stock-exchange-xqtkxtd5za-ts.a.run.app/stock_exchange/stocks", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1218133366, "label": "Writable canned queries fail with useless non-error against immutable databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1728#issuecomment-1111752676", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1728", "id": 1111752676, "node_id": "IC_kwDOBm6k_c5CQ__k", "user": {"value": 127565, "label": "wragge"}, "created_at": "2022-04-28T05:11:54Z", "updated_at": "2022-04-28T05:11:54Z", "author_association": "CONTRIBUTOR", "body": "And in terms of the bug, yep I agree that option 2 would be the most useful and least frustrating.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1218133366, "label": "Writable canned queries fail with useless non-error against immutable databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1728#issuecomment-1111751734", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1728", "id": 1111751734, "node_id": "IC_kwDOBm6k_c5CQ_w2", "user": {"value": 127565, "label": "wragge"}, "created_at": "2022-04-28T05:09:59Z", "updated_at": "2022-04-28T05:09:59Z", "author_association": "CONTRIBUTOR", "body": "Thanks, I'll give it a try!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1218133366, "label": "Writable canned queries fail with useless non-error against immutable databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/741#issuecomment-1125342229", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/741", "id": 1125342229, "node_id": "IC_kwDOBm6k_c5DE1wV", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-05-12T19:21:16Z", "updated_at": "2022-05-12T19:21:16Z", "author_association": "CONTRIBUTOR", "body": "Came here to check if this had been flagged already. Was helping a colleague get something on Cloud Run and had to dig to find `--extra-options=\"--setting sql_time_limit_ms 2500\"`.\r\n\r\nIf I get some time next week, maybe I'll try to tackle it. Would definitely make things easier to be able to do something like this:\r\n\r\n```sh\r\ndatasette publish cloudrun something.db --setting sql_time_limit_ms 2500\r\n```\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 607223136, "label": "Replace \"datasette publish --extra-options\" with \"--setting\""}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1742#issuecomment-1128049716", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1742", "id": 1128049716, "node_id": "IC_kwDOBm6k_c5DPKw0", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-05-16T19:24:44Z", "updated_at": "2022-05-16T19:24:44Z", "author_association": "CONTRIBUTOR", "body": "Where is `_trace` getting injected? And is it something a plugin should be able to handle? (If it is, I guess I should handle it in this case.)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1237586379, "label": "?_trace=1 fails with datasette-geojson for some reason"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1742#issuecomment-1128064864", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1742", "id": 1128064864, "node_id": "IC_kwDOBm6k_c5DPOdg", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-05-16T19:42:13Z", "updated_at": "2022-05-16T19:42:13Z", "author_association": "CONTRIBUTOR", "body": "Just to add a wrinkle here, this loads fine: https://alltheplaces-datasette.fly.dev/alltheplaces/places.geojson?_trace=1\r\n\r\nBut also, this doesn't add any trace data: https://alltheplaces-datasette.fly.dev/alltheplaces/places.json?_trace=1\r\n\r\nWhat am I missing?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1237586379, "label": "?_trace=1 fails with datasette-geojson for some reason"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1528#issuecomment-1151887842", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1528", "id": 1151887842, "node_id": "IC_kwDOBm6k_c5EqGni", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-06-10T03:23:08Z", "updated_at": "2022-06-10T03:23:08Z", "author_association": "CONTRIBUTOR", "body": "I just put together a version of this in a plugin: https://github.com/eyeseast/datasette-query-files. Happy to have any feedback.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1060631257, "label": "Add new `\"sql_file\"` key to Canned Queries in metadata?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1713#issuecomment-1173358747", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1713", "id": 1173358747, "node_id": "IC_kwDOBm6k_c5F8Aib", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2022-07-04T05:16:35Z", "updated_at": "2022-07-04T05:16:35Z", "author_association": "CONTRIBUTOR", "body": "This feature is pretty important and would be nice if it would be all within Datasette (no separate CLI/deploy required). My workflow now is to basically just copy the result and paste into a Google Sheet, which works, but then it's not discoverable to other journalists browsing the Datasette instance. I started building a plugin similar to [datasette-saved-queries](https://datasette.io/plugins/datasette-saved-queries) but one that maintains its own DB (required if you're working with all immutable DBs), but got bogged down in details.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1203943272, "label": "Datasette feature for publishing snapshots of query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1753#issuecomment-1163091750", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1753", "id": 1163091750, "node_id": "IC_kwDOBm6k_c5FU18m", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2022-06-22T13:22:34Z", "updated_at": "2022-06-22T13:22:34Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #1760.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1261826957, "label": "Bump furo from 2022.4.7 to 2022.6.4.1"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1693#issuecomment-1168704157", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1693", "id": 1168704157, "node_id": "IC_kwDOBm6k_c5FqQKd", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2022-06-28T13:11:36Z", "updated_at": "2022-06-28T13:11:36Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #1763.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1184850337, "label": "Bump black from 22.1.0 to 22.3.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1191#issuecomment-1200732975", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1191", "id": 1200732975, "node_id": "IC_kwDOBm6k_c5Hkbsv", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2022-08-01T05:39:27Z", "updated_at": "2022-08-01T05:39:27Z", "author_association": "CONTRIBUTOR", "body": "I've got a URL shortening plugin that I would like to embed on the query page but I'd like avoid capturing the entire `query.html` template. A feature like this would solve it. Where's this at and how can I help?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 787098345, "label": "Ability for plugins to collaborate when adding extra HTML to blocks in default templates"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1789#issuecomment-1223347322", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1789", "id": 1223347322, "node_id": "IC_kwDOBm6k_c5I6sx6", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2022-08-23T00:03:20Z", "updated_at": "2022-08-23T00:03:20Z", "author_association": "CONTRIBUTOR", "body": "@simonw to build the extension on ubuntu, you can run:\r\n\r\n```\r\napt-get update && apt-get install libsqlite3-dev gcc\r\ngcc ext.c -fPIC -shared -o ext.so\r\n```\r\n\r\nI'm not the best with Actions, but if you set the cache key to `ext.c`, run those two commands to download dependencies + compile to `ext.so`, then the unit test should pick it up and run it correctly. Let me know if you want me to update the PR with that added", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1344823170, "label": "Add new entrypoint option to `--load-extension`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1779#issuecomment-1210675046", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1779", "id": 1210675046, "node_id": "IC_kwDOBm6k_c5IKW9m", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-08-10T13:28:37Z", "updated_at": "2022-08-10T13:28:37Z", "author_association": "CONTRIBUTOR", "body": "maybe a simpler solution is to set the maxscale to like 2? since datasette is not set up to make use of container scaling anyway?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1334628400, "label": "google cloudrun updated their limits on maxscale based on memory and cpu count"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1779#issuecomment-1214437408", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1779", "id": 1214437408, "node_id": "IC_kwDOBm6k_c5IYtgg", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-08-14T19:42:58Z", "updated_at": "2022-08-14T19:42:58Z", "author_association": "CONTRIBUTOR", "body": "thanks @simonw!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1334628400, "label": "google cloudrun updated their limits on maxscale based on memory and cpu count"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1789#issuecomment-1221576460", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1789", "id": 1221576460, "node_id": "IC_kwDOBm6k_c5Iz8cM", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2022-08-21T16:16:42Z", "updated_at": "2022-08-21T16:16:42Z", "author_association": "CONTRIBUTOR", "body": "Rebased, Read the docs failure should now now fixed\r\n\r\nRe docs - ya that's a pretty ambitious page, I'm still not 100% sure what the best practices are/should be... Would be happy to make that page in a future PR", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1344823170, "label": "Add new entrypoint option to `--load-extension`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1685#issuecomment-1237381620", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1685", "id": 1237381620, "node_id": "IC_kwDOBm6k_c5JwPH0", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2022-09-05T18:36:47Z", "updated_at": "2022-09-05T18:36:47Z", "author_association": "CONTRIBUTOR", "body": "Looks like jinja2 is no longer updatable, so this is no longer needed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1180778860, "label": "Update jinja2 requirement from <3.1.0,>=2.10.3 to >=2.10.3,<3.2.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1799#issuecomment-1237381569", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1799", "id": 1237381569, "node_id": "IC_kwDOBm6k_c5JwPHB", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2022-09-05T18:36:42Z", "updated_at": "2022-09-05T18:36:42Z", "author_association": "CONTRIBUTOR", "body": "Looks like aiofiles is no longer updatable, so this is no longer needed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1362242558, "label": "Update aiofiles requirement from <0.9,>=0.4 to >=0.4,<22.2"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1817#issuecomment-1256781274", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1817", "id": 1256781274, "node_id": "IC_kwDOBm6k_c5K6PXa", "user": {"value": 50527, "label": "jefftriplett"}, "created_at": "2022-09-23T22:59:46Z", "updated_at": "2022-09-23T22:59:46Z", "author_association": "CONTRIBUTOR", "body": "While you are adding features, would you be future-proofing your APIs if you switched over some arguments over to keyword-only arguments or would that be too disruptive?\r\n\r\nThinking out loud:\r\n\r\n```\r\nasync def render_template( \r\n self, templates, *, context=None, plugin_context=None, request=None, view_name=None \r\n ): \r\n```\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384273985, "label": "Expose `sql` and `params` arguments to various plugin hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1810#issuecomment-1248204219", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1810", "id": 1248204219, "node_id": "IC_kwDOBm6k_c5KZhW7", "user": {"value": 82988, "label": "psychemedia"}, "created_at": "2022-09-15T14:44:47Z", "updated_at": "2022-09-15T14:46:26Z", "author_association": "CONTRIBUTOR", "body": "A couple+ of possible use case examples:\r\n\r\n- someone has a collection of articles indexed with FTS; they want to publish a simple search tool over the results;\r\n- someone has an image collection and they want to be able to search over description text to return images;\r\n- someone has a set of locations with descriptions, and wants to run a query over places and descriptions and get results as a listing or on a map;\r\n- someone has a set of audio or video files with titles, descriptions and/or transcripts, and wants to be able to search over them and return playable versions of returned items.\r\n\r\nIn many cases, I suspect the raw content will be in one table, but the search table will be a second (eg FTS) table. Generally, the search may be over one or more joined tables, and the results constructed from one or more tables (which may or may not be distinct from the search tables).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1374626873, "label": "Featured table(s) on the homepage"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1258129113", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1258129113, "node_id": "IC_kwDOBm6k_c5K_YbZ", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-09-26T14:30:11Z", "updated_at": "2022-09-26T14:48:31Z", "author_association": "CONTRIBUTOR", "body": "from your analysis, it seems like the GIL is blocking on loading of the data from sqlite to python, (particularly in the `fetchmany` call)\r\n\r\nthis is probably a simplistic idea, but what if you had the python code in the `execute` method iterate over the cursor and yield out rows or small chunks of rows.\r\n\r\nsomething like: \r\n```python\r\n with sqlite_timelimit(conn, time_limit_ms):\r\n try:\r\n cursor = conn.cursor()\r\n cursor.execute(sql, params if params is not None else {})\r\n except:\r\n ...\r\n max_returned_rows = self.ds.max_returned_rows\r\n if max_returned_rows == page_size:\r\n max_returned_rows += 1\r\n if max_returned_rows and truncate:\r\n for i, row in enumerate(cursor):\r\n yield row\r\n if i == max_returned_rows - 1:\r\n break\r\n else:\r\n for row in cursor:\r\n yield row\r\n truncated = False \r\n```\r\n\r\nthis kind of thing works well with a postgres server side cursor, but i'm not sure if it will hold for sqlite. \r\n\r\nyou would still spend about the same amount of time in python and would be contending for the gil, but it would be could be non blocking.\r\n\r\ndepending on the data flow, this could also some benefit for memory. (data stays in more compact sqlite-land until you need it)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-1258167564", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 1258167564, "node_id": "IC_kwDOBm6k_c5K_h0M", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-09-26T14:57:44Z", "updated_at": "2022-09-26T15:08:36Z", "author_association": "CONTRIBUTOR", "body": "reading the database execute method i have a few questions.\r\n\r\nhttps://github.com/simonw/datasette/blob/cb1e093fd361b758120aefc1a444df02462389a3/datasette/database.py#L229-L242\r\n\r\n---\r\nunless i'm missing something (which is very likely!!), the `max_returned_rows` argument doesn't actually offer any protections against running very expensive queries. \r\n\r\nIt's not like adding a `LIMIT max_rows` argument. it make sense that it isn't because, the query could already have an `LIMIT` argument. Doing something like `select * from (query) limit {max_returned_rows}` **might** be protective but wouldn't always.\r\n\r\nInstead the code executes the full original query, and if still has time it fetches out the first `max_rows + 1` rows. \r\n\r\nthis *does* offer some protection of memory exhaustion, as you won't hydrate a huge result set into python (however, there are [data flow patterns](https://github.com/simonw/datasette/issues/1727#issuecomment-1258129113) that could avoid that too)\r\n\r\ngiven the current architecture, i don't see how creating a new connection would be use?\r\n\r\n---\r\n\r\nIf we just removed the `max_return_rows` limitation, then i think most things would be fine **except** for the QueryViews. Right now rendering, just [5000 rows takes a lot of client-side memory](https://github.com/simonw/datasette/issues/1655) so some form of pagination would be required.\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1655#issuecomment-1258166572", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1655", "id": 1258166572, "node_id": "IC_kwDOBm6k_c5K_hks", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-09-26T14:57:04Z", "updated_at": "2022-09-26T14:57:04Z", "author_association": "CONTRIBUTOR", "body": "I think that paginating, even in javascript, could be very helpful. Maybe render json or csv into the page and let javascript loading that into the dom?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1163369515, "label": "query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1813#issuecomment-1250901367", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1813", "id": 1250901367, "node_id": "IC_kwDOBm6k_c5Kjz13", "user": {"value": 883348, "label": "adipasquale"}, "created_at": "2022-09-19T11:34:45Z", "updated_at": "2022-09-19T11:34:45Z", "author_association": "CONTRIBUTOR", "body": "oh and by writing this I just realized the difference: the URL on fly.io is with a custom SQL command whereas the local one is without. \r\nIt seems that there is no pagination when using custom SQL commands which makes sense\r\n\r\nSorry for this useless issue, maybe this can be useful for someone else / me in the future.\r\n\r\nThanks again for this wonderful project !", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1377811868, "label": "missing next and next_url in JSON responses from an instance deployed on Fly "}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-1254064260", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 1254064260, "node_id": "IC_kwDOBm6k_c5Kv4CE", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-09-21T18:17:04Z", "updated_at": "2022-09-21T18:18:01Z", "author_association": "CONTRIBUTOR", "body": "hi @simonw, this is becoming more of a bother for my [labor data warehouse](https://labordata.bunkum.us/). Is there any research or a spike i could do that would help you investigate this issue?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}