{"html_url": "https://github.com/simonw/datasette/issues/1421#issuecomment-894929080", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1421", "id": 894929080, "node_id": "IC_kwDOBm6k_c41V4i4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T03:33:02Z", "updated_at": "2021-08-09T03:33:02Z", "author_association": "OWNER", "body": "Fixed! Fantastic, this one has been bothering me for *years*.\r\n\r\nhttps://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27\r\n\r\n\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959999095, "label": "\"Query parameters\" form shows wrong input fields if query contains \"03:31\" style times"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1421#issuecomment-894929769", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1421", "id": 894929769, "node_id": "IC_kwDOBm6k_c41V4tp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T03:36:49Z", "updated_at": "2021-08-09T03:36:49Z", "author_association": "OWNER", "body": "SQLite carries a warning about using `EXPLAIN` like this: https://www.sqlite.org/lang_explain.html\r\n\r\n> The output from EXPLAIN and EXPLAIN QUERY PLAN is intended for interactive analysis and troubleshooting only. The details of the output format are subject to change from one release of SQLite to the next. Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.\r\n\r\nI think that's OK here, because of the regular expression fallback. If the format changes in the future in a way that breaks the query the error should be caught and the regex-captured parameters should be returned instead.\r\n\r\nHmmm... actually that's not entirely true:\r\n\r\nhttps://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L1084-L1091\r\n\r\nIf the format changes such that the same columns are returned but the `[row[\"p4\"].lstrip(\":\") for row in results if row[\"opcode\"] == \"Variable\"]` list comprehension returns an empty array it will break Datasette!\r\n\r\nI'm going to take that risk for the moment, but I'll actively watch out for problems in the future. If this does turn out to be bad I can always go back to the pure regular expression mechanism.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959999095, "label": "\"Query parameters\" form shows wrong input fields if query contains \"03:31\" style times"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1421#issuecomment-894930013", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1421", "id": 894930013, "node_id": "IC_kwDOBm6k_c41V4xd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T03:38:06Z", "updated_at": "2021-08-09T03:38:06Z", "author_association": "OWNER", "body": "Amusing edge-case: if you run this against a `explain ...` query it falls back to using regular expressions, because `explain explain select ...` is invalid SQL. https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959999095, "label": "\"Query parameters\" form shows wrong input fields if query contains \"03:31\" style times"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1425#issuecomment-894865323", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1425", "id": 894865323, "node_id": "IC_kwDOBm6k_c41Vo-r", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-08T22:33:19Z", "updated_at": "2021-08-08T22:33:19Z", "author_association": "OWNER", "body": "I can do this with the `await_me_maybe()` function, as seen here: https://github.com/simonw/datasette/blob/a21853c9dade240734abc6b4f750fae09a3e840a/datasette/app.py#L864-L873", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963528457, "label": "render_cell() hook should support returning an awaitable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1424#issuecomment-894864682", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1424", "id": 894864682, "node_id": "IC_kwDOBm6k_c41Vo0q", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-08T22:26:46Z", "updated_at": "2021-08-08T22:26:46Z", "author_association": "OWNER", "body": "Note that the `sqlite3` exceptions are in `sqlite3` if using the Python standard library but are in `pysqlite3` if that module is being used instead.\r\n\r\nSo maybe encourage people to use them from `datasette.sqlite.sqlite3` instead, which will point to the correct package.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963527045, "label": "Document exceptions that can be raised by db.execute() and friends"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1424#issuecomment-894864744", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1424", "id": 894864744, "node_id": "IC_kwDOBm6k_c41Vo1o", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-08T22:27:31Z", "updated_at": "2021-08-08T22:27:31Z", "author_association": "OWNER", "body": "https://docs.python.org/3/library/sqlite3.html#exceptions is useful - it looks like `sqlite3.DatabaseError` is the super-class of all of the other exceptions that we might see.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963527045, "label": "Document exceptions that can be raised by db.execute() and friends"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1424#issuecomment-894864404", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1424", "id": 894864404, "node_id": "IC_kwDOBm6k_c41VowU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-08T22:24:06Z", "updated_at": "2021-08-08T22:24:06Z", "author_association": "OWNER", "body": "Relevant code: https://github.com/simonw/datasette/blob/de5ce2e56339ad8966f417a4758f7c210c017dec/datasette/database.py#L176-L200", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963527045, "label": "Document exceptions that can be raised by db.execute() and friends"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1424#issuecomment-894864616", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1424", "id": 894864616, "node_id": "IC_kwDOBm6k_c41Vozo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-08T22:26:08Z", "updated_at": "2021-08-08T22:26:08Z", "author_association": "OWNER", "body": "- `datasette.database.QueryInterrupted` for queries that were interrupted\r\n- `sqlite3.OperationalError`\r\n- `sqlite3.DatabaseError` and more", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963527045, "label": "Document exceptions that can be raised by db.execute() and friends"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1425#issuecomment-894869692", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1425", "id": 894869692, "node_id": "IC_kwDOBm6k_c41VqC8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-08T23:08:29Z", "updated_at": "2021-08-08T23:08:29Z", "author_association": "OWNER", "body": "Updated documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-value-column-table-database-datasette", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963528457, "label": "render_cell() hook should support returning an awaitable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1425#issuecomment-894881448", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1425", "id": 894881448, "node_id": "IC_kwDOBm6k_c41Vs6o", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T00:24:25Z", "updated_at": "2021-08-09T00:24:39Z", "author_association": "OWNER", "body": "My hunch is that the \"skip this `render_cell()` result if it returns `None`\" logic isn't working correctly, ever since I added the `await_me_maybe` line.\r\n\r\nCould that be because Pluggy handles the \"do the next if `None` is returned\" logic itself, but I'm no-longer returning `None`, I'm returning an awaitable which when awaited returns `None`.\r\n\r\nThis would suggest that all of the `await_me_maybe()` plugin hooks have the same bug. That's definitely possible - it may well be that no-one has yet stumbled across a bug caused by a plugin returning an awaitable and hence not being skipped, because plugin hooks that return awaitable are rare enough that no-one has tried two plugins which both use that trick.\r\n\r\nStill don't see why it would pass on my laptop but fail in CI though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963528457, "label": "render_cell() hook should support returning an awaitable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1425#issuecomment-894881016", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1425", "id": 894881016, "node_id": "IC_kwDOBm6k_c41Vsz4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T00:21:53Z", "updated_at": "2021-08-09T00:21:53Z", "author_association": "OWNER", "body": "Still one test failure:\r\n```\r\n def test_hook_render_cell_link_from_json(app_client):\r\n sql = \"\"\"\r\n select '{\"href\": \"http://example.com/\", \"label\":\"Example\"}'\r\n \"\"\".strip()\r\n path = \"/fixtures?\" + urllib.parse.urlencode({\"sql\": sql})\r\n response = app_client.get(path)\r\n td = Soup(response.body, \"html.parser\").find(\"table\").find(\"tbody\").find(\"td\")\r\n a = td.find(\"a\")\r\n> assert a is not None, str(a)\r\nE AssertionError: None\r\nE assert None is not None\r\n```\r\nThe weird thing about this one is that I can't replicate it on my laptop - but it happens in CI every time, including when I shell in and try to run that single test.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963528457, "label": "render_cell() hook should support returning an awaitable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1425#issuecomment-894882123", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1425", "id": 894882123, "node_id": "IC_kwDOBm6k_c41VtFL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T00:27:43Z", "updated_at": "2021-08-09T00:27:43Z", "author_association": "OWNER", "body": "Good news: `render_cell()` is the only hook to use `firstresult=True`:\r\n\r\nhttps://github.com/simonw/datasette/blob/f3c9edb376a13c09b5ecf97c7390f4e49efaadf2/datasette/hookspecs.py#L62-L64\r\n\r\nhttps://pluggy.readthedocs.io/en/latest/#first-result-only", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963528457, "label": "render_cell() hook should support returning an awaitable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1425#issuecomment-894882642", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1425", "id": 894882642, "node_id": "IC_kwDOBm6k_c41VtNS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T00:29:57Z", "updated_at": "2021-08-09T00:29:57Z", "author_association": "OWNER", "body": "Here's the code in `pluggy` that implements this: https://github.com/pytest-dev/pluggy/blob/0a064fe275060dbdb1fe6e10c888e72bc400fb33/src/pluggy/callers.py#L31-L43\r\n\r\n```python\r\n if hook_impl.hookwrapper:\r\n try:\r\n gen = hook_impl.function(*args)\r\n next(gen) # first yield\r\n teardowns.append(gen)\r\n except StopIteration:\r\n _raise_wrapfail(gen, \"did not yield\")\r\n else:\r\n res = hook_impl.function(*args)\r\n if res is not None:\r\n results.append(res)\r\n if firstresult: # halt further impl calls\r\n break\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963528457, "label": "render_cell() hook should support returning an awaitable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1425#issuecomment-894883664", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1425", "id": 894883664, "node_id": "IC_kwDOBm6k_c41VtdQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T00:33:56Z", "updated_at": "2021-08-09T00:33:56Z", "author_association": "OWNER", "body": "I could extract that code out and write my own function which implements the equivalent of calling `pm.hook.render_cell(...)` but runs `await_me_maybe()` before checking if `res is not None`.\r\n\r\nThat's pretty nasty.\r\n\r\nCould I instead call the plugin hook normally, but then have additional logic which says \"if I await it and it returns `None` then try calling the hook again but skip this one\" - not sure if there's a way to do that either.\r\n\r\nI could remove the `firstresult=True` from the hookspec - which would cause it to call and return ALL hooks - but then in my own code use only the first one. This is slightly less efficient (since it calls all the hooks and then discards all-but-one value) but it's the least unpleasant in terms of the code I would have to write - plus I don't think it's going to be THAT common for someone to have multiple expensive `render_cell()` hooks installed at once (they are usually pretty cheap).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963528457, "label": "render_cell() hook should support returning an awaitable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1425#issuecomment-894884874", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1425", "id": 894884874, "node_id": "IC_kwDOBm6k_c41VtwK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T00:38:20Z", "updated_at": "2021-08-09T00:38:20Z", "author_association": "OWNER", "body": "I'm trying the version where I remove `firstresult=True`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963528457, "label": "render_cell() hook should support returning an awaitable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1425#issuecomment-894893319", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1425", "id": 894893319, "node_id": "IC_kwDOBm6k_c41Vv0H", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T01:08:56Z", "updated_at": "2021-08-09T01:09:12Z", "author_association": "OWNER", "body": "Demo: https://latest.datasette.io/fixtures/simple_primary_key shows `RENDER_CELL_ASYNC_RESULT` where the CSV version shows `RENDER_CELL_ASYNC`: https://latest.datasette.io/fixtures/simple_primary_key.csv - because of this test plugin code: https://github.com/simonw/datasette/blob/a390bdf9cef01d8723d025fc3348e81345ff4856/tests/plugins/my_plugin.py#L98-L122", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963528457, "label": "render_cell() hook should support returning an awaitable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1425#issuecomment-894900267", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1425", "id": 894900267, "node_id": "IC_kwDOBm6k_c41Vxgr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T01:31:22Z", "updated_at": "2021-08-09T01:31:22Z", "author_association": "OWNER", "body": "I used this to build a new plugin: https://github.com/simonw/datasette-query-links\r\n\r\nDemo here: https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++%27select+*+from+[facetable]%27+as+query%0D%0Aunion%0D%0Aselect%0D%0A++%27select+sqlite_version()%27%0D%0Aunion%0D%0Aselect%0D%0A++%27select+this+is+invalid+SQL+so+will+not+be+linked%27", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 963528457, "label": "render_cell() hook should support returning an awaitable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1426#issuecomment-895500565", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1426", "id": 895500565, "node_id": "IC_kwDOBm6k_c41YEEV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T20:00:04Z", "updated_at": "2021-08-09T20:00:04Z", "author_association": "OWNER", "body": "A few options for how this would work:\r\n\r\n- `datasette ... --robots allow`\r\n- `datasette ... --setting robots allow`\r\n\r\nOptions could be:\r\n\r\n- `allow` - allow all crawling\r\n- `deny` - deny all crawling\r\n- `limited` - allow access to the homepage and the index pages for each database and each table, but disallow crawling any further than that\r\n\r\nThe \"limited\" mode is particularly interesting. Could even make it the default, but I think that may be a bit too confusing. Idea would be to get the key pages indexed but use `nofollow` to discourage crawlers from indexing individual row pages or deep pages like `https://datasette.io/content/repos?_facet=owner&_facet=language&_facet_array=topics&topics__arraycontains=sqlite#facet-owner`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 964322136, "label": "Manage /robots.txt in Datasette core, block robots by default"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1426#issuecomment-895509536", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1426", "id": 895509536, "node_id": "IC_kwDOBm6k_c41YGQg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T20:12:57Z", "updated_at": "2021-08-09T20:12:57Z", "author_association": "OWNER", "body": "I could try out the `X-Robots` HTTP header too: https://developers.google.com/search/docs/advanced/robots/robots_meta_tag#xrobotstag", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 964322136, "label": "Manage /robots.txt in Datasette core, block robots by default"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1426#issuecomment-895510773", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1426", "id": 895510773, "node_id": "IC_kwDOBm6k_c41YGj1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T20:14:50Z", "updated_at": "2021-08-09T20:19:22Z", "author_association": "OWNER", "body": "https://twitter.com/mal/status/1424825895139876870\r\n\r\n> True pinging google should be part of the build process on a static site :)\r\n\r\nThat's another aspect of this: if you DO want your site crawled, teaching the `datasette publish` command how to ping Google when a deploy has gone out could be a nice improvement.\r\n\r\nAnnoyingly it looks like you need to configure an auth token of some sort in order to use their API though, which is likely too much hassle to be worth building into Datasette itself: https://developers.google.com/search/apis/indexing-api/v3/using-api\r\n\r\n```\r\ncurl -X POST https://indexing.googleapis.com/v3/urlNotifications:publish -d '{\r\n \"url\": \"https://careers.google.com/jobs/google/technical-writer\",\r\n \"type\": \"URL_UPDATED\"\r\n}' -H \"Content-Type: application/json\"\r\n\r\n{\r\n \"error\": {\r\n \"code\": 401,\r\n \"message\": \"Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.\",\r\n \"status\": \"UNAUTHENTICATED\"\r\n }\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 964322136, "label": "Manage /robots.txt in Datasette core, block robots by default"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1426#issuecomment-895522818", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1426", "id": 895522818, "node_id": "IC_kwDOBm6k_c41YJgC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-09T20:34:10Z", "updated_at": "2021-08-09T20:34:10Z", "author_association": "OWNER", "body": "At the very least Datasette should serve a blank `/robots.txt` by default - I'm seeing a ton of 404s for it in the logs.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 964322136, "label": "Manage /robots.txt in Datasette core, block robots by default"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-905900807", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 905900807, "node_id": "IC_kwDOBm6k_c41_vMH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-25T21:51:10Z", "updated_at": "2021-08-25T21:51:10Z", "author_association": "OWNER", "body": "10-20 minutes to populate `_internal`! How many databases and tables is that for?\r\n\r\nI may have to rethink the `_internal` mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed.\r\n\r\nI need to get my own stress testing rig setup for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898063815", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898063815, "node_id": "IC_kwDOBm6k_c41h13H", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-13T00:33:17Z", "updated_at": "2021-08-13T00:33:17Z", "author_association": "OWNER", "body": "Improved version of that function:\r\n```python\r\ndef columns_for_query(conn, sql):\r\n \"\"\"\r\n Given a SQLite connection ``conn`` and a SQL query ``sql``,\r\n returns a list of ``(table_name, column_name)`` pairs, one\r\n per returned column. ``(None, None)`` if no table and column\r\n could be derived.\r\n \"\"\"\r\n rows = conn.execute('explain ' + sql).fetchall()\r\n table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'}\r\n names_by_rootpage = dict(\r\n conn.execute(\r\n 'select rootpage, name from sqlite_master where rootpage in ({})'.format(\r\n ', '.join(map(str, table_rootpage_by_register.values()))\r\n )\r\n )\r\n )\r\n columns_by_column_register = {}\r\n for row in rows:\r\n if row['opcode'] in ('Rowid', 'Column'):\r\n addr, opcode, table_id, cid, column_register, p4, p5, comment = row\r\n table = names_by_rootpage[table_rootpage_by_register[table_id]]\r\n columns_by_column_register[column_register] = (table, cid)\r\n result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0]\r\n registers = list(range(result_row[\"p1\"], result_row[\"p1\"] + result_row[\"p2\"]))\r\n all_column_names = {}\r\n for table in names_by_rootpage.values():\r\n table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall()\r\n for row in table_xinfo:\r\n all_column_names[(table, row[\"cid\"])] = row[\"name\"]\r\n final_output = []\r\n for r in registers:\r\n try:\r\n table, cid = columns_by_column_register[r]\r\n final_output.append((table, all_column_names[table, cid]))\r\n except KeyError:\r\n final_output.append((None, None))\r\n return final_output\r\n```\r\nIt works!\r\n\r\n\r\n\r\n```diff\r\ndiff --git a/datasette/templates/query.html b/datasette/templates/query.html\r\nindex 75f7f1b..9fe1d4f 100644\r\n--- a/datasette/templates/query.html\r\n+++ b/datasette/templates/query.html\r\n@@ -67,6 +67,8 @@\r\n
\r\n \r\n \r\n+extra_column_info: {{ extra_column_info }}\r\n+\r\n {% if display_rows %}\r\nThis data as {% for name, url in renderers.items() %}{{ name }}{{ \", \" if not loop.last }}{% endfor %}, CSV
\r\n