{"html_url": "https://github.com/simonw/datasette/issues/498#issuecomment-504793988", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/498", "id": 504793988, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDc5Mzk4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T22:40:25Z", "updated_at": "2019-06-23T22:41:25Z", "author_association": "OWNER", "body": "You can use a database query against `sqlite_master` to find all the FTS tables:\r\n\r\nhttp://sf-trees.datasettes.com/trees-b64f0cb?sql=select+name+from+sqlite_master+where+name+like+%22%25_fts%22+and+type+%3D+%22table%22\r\n\r\n`select name from sqlite_master where name like \"%_fts\" and type = \"table\"`\r\n\r\nYou could then construct a crafty UNION query to get results back from all of those tables at once:\r\n\r\nhttp://sf-trees.datasettes.com/trees-b64f0cb?sql=select+%27PlantType_value_fts%27%2C+rowid%2C+value+from+PlantType_value_fts+where+PlantType_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qCareAssistant_value_fts%27%2C+rowid%2C+value+from+qCareAssistant_value_fts+where+qCareAssistant_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qCaretaker_value_fts%27%2C+rowid%2C+value+from+qCaretaker_value_fts+where+qCaretaker_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qLegalStatus_value_fts%27%2C+rowid%2C+value+from+qLegalStatus_value_fts+where+qLegalStatus_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qSiteInfo_value_fts%27%2C+rowid%2C+value+from+qSiteInfo_value_fts+where+qSiteInfo_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qSpecies_value_fts%27%2C+rowid%2C+value+from+qSpecies_value_fts+where+qSpecies_value_fts+match+%3Asearch&search=fi%2A\r\n\r\n(I'm searching for `fi*` here to demonstrate wildcards)\r\n\r\nThe problem, as discussed earlier, is relevance: there's no way to compare the scores you're getting across different tables, so you won't be able to order by anything.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 451513541, "label": "Full text search of all tables at once?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/518#issuecomment-504765018", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/518", "id": 504765018, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDc2NTAxOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T16:03:20Z", "updated_at": "2019-06-23T16:03:20Z", "author_association": "OWNER", "body": "Weird new bug: http://127.0.0.1:8001/fixtures/table%2Fwith%2Fslashes.csv?_format=json is downloading CSV for me now.\r\n\r\nhttps://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=json does the right thing.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459587155, "label": "Port Datasette from Sanic to ASGI + Uvicorn"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/498#issuecomment-504785662", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/498", "id": 504785662, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDc4NTY2Mg==", "user": {"value": 7936571, "label": "chrismp"}, "created_at": "2019-06-23T20:47:37Z", "updated_at": "2019-06-23T20:47:37Z", "author_association": "NONE", "body": "Very cool, thank you.\r\n\r\nUsing http://search-24ways.herokuapp.com as an example, let's say I want to search all FTS columns in all tables in all databases for the word \"web.\" \r\n\r\n[Here's a link](http://search-24ways.herokuapp.com/24ways-f8f455f?sql=select+count%28*%29from+articles+where+rowid+in+%28select+rowid+from+articles_fts+where+articles_fts+match+%3Asearch%29&search=web) to the query I'd need to run to search \"web\" on FTS columns in `articles` table of the `24ways` database. \r\n\r\nAnd [here's a link](http://search-24ways.herokuapp.com/24ways-f8f455f.json?sql=select+count%28*%29from+articles+where+rowid+in+%28select+rowid+from+articles_fts+where+articles_fts+match+%3Asearch%29&search=web) to the JSON version of the above result. I'd like to get the JSON result of that query for each FTS table of each database in my datasette project. \r\n\r\nIs it possible in Javascript to automate the construction of query URLs like the one I linked, but for every FTS table in my datasette project?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 451513541, "label": "Full text search of all tables at once?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/518#issuecomment-504762769", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/518", "id": 504762769, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDc2Mjc2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T15:37:26Z", "updated_at": "2019-06-23T15:37:26Z", "author_association": "OWNER", "body": "This is strange: on my local machine `http://127.0.0.1:8001/fixtures/table%2Fwith%2Fslashes.csv` is returning a 404 BUT there's a test for that which is passing under pytest:\r\n\r\nhttps://github.com/simonw/datasette/blob/d60fbfcae2658e71cab6d7b3b9f53f8d895064ef/tests/test_api.py#L721-L727", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459587155, "label": "Port Datasette from Sanic to ASGI + Uvicorn"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/520#issuecomment-504772599", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/520", "id": 504772599, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDc3MjU5OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T17:44:36Z", "updated_at": "2019-06-23T17:44:36Z", "author_association": "OWNER", "body": "These plugins will need access to configuration and the ability to execute SQL - which means we need to make the `datasette` instance available to them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459598080, "label": "asgi_wrapper plugin hook"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/226#issuecomment-504720379", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/226", "id": 504720379, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDcyMDM3OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T05:05:32Z", "updated_at": "2019-06-23T05:05:32Z", "author_association": "OWNER", "body": "The mechanism I described here - having a `tests/example_plugin` folder - is probably the right solution for #517 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 315738696, "label": "Unit tests for installable plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/518#issuecomment-504790825", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/518", "id": 504790825, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDc5MDgyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T21:57:58Z", "updated_at": "2019-06-23T21:57:58Z", "author_association": "OWNER", "body": "The big one: **Replace Sanic request and response objects with my own classes, so I can remove Sanic dependency**", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459587155, "label": "Port Datasette from Sanic to ASGI + Uvicorn"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/518#issuecomment-504791053", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/518", "id": 504791053, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDc5MTA1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T22:00:56Z", "updated_at": "2019-06-23T22:00:56Z", "author_association": "OWNER", "body": "The `InvalidUsage` exception is thrown by Sanic when it gets an unhandled request - usually a HEAD. It was added in efbb4e83374a2c795e436c72fa79f70da72309b8\r\n\r\nI'm going to replace it with specific handling for HEAD requests plus a unit test.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459587155, "label": "Port Datasette from Sanic to ASGI + Uvicorn"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/516#issuecomment-504768147", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/516", "id": 504768147, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDc2ODE0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T16:43:23Z", "updated_at": "2019-06-23T16:43:23Z", "author_association": "OWNER", "body": "The Starlette lint and test scripts do this, and also apply autoflake to remove any unnecessary imports: https://github.com/encode/starlette/tree/8c8cc2ec0a5cb834a9a15b871ae8b480503abb67/scripts", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459509126, "label": "Enforce import sort order with isort"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/518#issuecomment-504791796", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/518", "id": 504791796, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDc5MTc5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T22:10:02Z", "updated_at": "2019-06-24T13:42:50Z", "author_association": "OWNER", "body": "The Sanic stuff I import at the moment is:\r\n\r\n- [x] `from sanic.views import HTTPMethodView`\r\n- [x] `from sanic.request import Request` https://github.com/huge-success/sanic/blob/0.7.0/sanic/request.py#L44\r\n- [x] `from sanic.exceptions import NotFound`\r\n- [x] `from sanic import response` - used for `response.text(\"\")`, `response.redirect(path)`, `response.HTTPResponse` and `response.html()` https://github.com/huge-success/sanic/blob/0.7.0/sanic/response.py\r\n- [x] `from sanic.request import RequestParameters` https://github.com/huge-success/sanic/blob/0.7.0/sanic/request.py#L30", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459587155, "label": "Port Datasette from Sanic to ASGI + Uvicorn"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/272#issuecomment-504760061", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/272", "id": 504760061, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDc2MDA2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T15:02:52Z", "updated_at": "2019-06-23T15:02:52Z", "author_association": "OWNER", "body": "Tests are failing on Python 3.5: https://travis-ci.org/simonw/datasette/jobs/549380098 - error is `TypeError: the JSON object must be str, not 'bytes'`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 324188953, "label": "Port Datasette to ASGI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/517#issuecomment-504720326", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/517", "id": 504720326, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDcyMDMyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T05:04:26Z", "updated_at": "2019-06-23T05:04:42Z", "author_association": "OWNER", "body": "See also #226 - \"Unit tests for installable plugins\"", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459537047, "label": "Add unit test for \"static\" mechanism in plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/272#issuecomment-504711468", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/272", "id": 504711468, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDcxMTQ2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T01:36:33Z", "updated_at": "2019-06-23T01:36:33Z", "author_association": "OWNER", "body": "Published an in-progress demo:\r\n\r\n datasette publish now fixtures.db -n datasette-asgi-early-demo --branch=asgi\r\n\r\nHere it is: https://datasette-asgi-early-demo-qahhxctqpw.now.sh/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 324188953, "label": "Port Datasette to ASGI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/498#issuecomment-504763919", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/498", "id": 504763919, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDc2MzkxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T15:50:49Z", "updated_at": "2019-06-23T15:50:49Z", "author_association": "OWNER", "body": "One interesting way to approach this could be to do it entirely in JavaScript. I've had a lot of success building small apps on top of Datasette's JavaScript API - I wrote up one example here: https://24ways.org/2018/fast-autocomplete-search-for-your-website/\r\n\r\nOnce #272 is done I'll be adding a plugin hook that allows plugins to define entirely new pages within the Datasette application, which may also be a good way to work on this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 451513541, "label": "Full text search of all tables at once?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/272#issuecomment-504716988", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/272", "id": 504716988, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDcxNjk4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T03:43:46Z", "updated_at": "2019-06-23T15:15:26Z", "author_association": "OWNER", "body": "OK, it's beginning to shape up now. Next steps:\r\n\r\n- [x] Static file support (including for plugins) - plus tests\r\n- [x] Streaming support so the CSV tests will pass\r\n- [x] Ability to download the database file\r\n- [x] Implement missing-slash redirects\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 324188953, "label": "Port Datasette to ASGI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/518#issuecomment-504765738", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/518", "id": 504765738, "node_id": "MDEyOklzc3VlQ29tbWVudDUwNDc2NTczOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-06-23T16:11:49Z", "updated_at": "2019-06-23T16:20:44Z", "author_association": "OWNER", "body": "OK, for **Get ?_trace=1 working again**. The old code lives in two places:\r\n\r\nhttps://github.com/simonw/datasette/blob/35429f90894321eda7f2db31b9ea7976f31f73ac/datasette/app.py#L546-L560\r\n\r\nAnd then:\r\n\r\nhttps://github.com/simonw/datasette/blob/35429f90894321eda7f2db31b9ea7976f31f73ac/datasette/app.py#L653-L672\r\n\r\nSo it's stashing something on the request to tell the rest of the code it should be tracing, then using that collected data from the request to add information to the final body.\r\n\r\nOne possible shape for the replacement is a new ASGI middleware that wraps everything else. We don't have a mutable request object here though, so we will need to untangle this entirely from the request object.\r\n\r\nAlso tricky is that in ASGI land we handle streams - we don't usually wait around for the entire response body to be compiled for us. This means the code that modifies the response (adding to the JSON or appending inside the `