{"html_url": "https://github.com/simonw/datasette/issues/1101#issuecomment-755128038", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1101", "id": 755128038, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTEyODAzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T07:10:22Z", "updated_at": "2021-01-06T07:10:22Z", "author_association": "OWNER", "body": "Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:\r\n\r\n    pandas.read_json(\"https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on\", lines=True)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 749283032, "label": "register_output_renderer() should support streaming data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1101#issuecomment-755133937", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1101", "id": 755133937, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTEzMzkzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T07:25:48Z", "updated_at": "2021-01-06T07:26:43Z", "author_association": "OWNER", "body": "Idea: instead of returning a dictionary, `register_output_renderer` could return an object. The object could have the following properties:\r\n\r\n- `.extension` - the extension to use\r\n- `.can_render(...)` - says if it can render this\r\n- `.can_stream(...)` - says if streaming is supported\r\n- `async .stream_rows(rows_iterator, send)` - method that loops through all rows and uses `send` to send them to the response in the correct format\r\n\r\nI can then deprecate the existing `dict` return type for 1.0.", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 749283032, "label": "register_output_renderer() should support streaming data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1101#issuecomment-755134771", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1101", "id": 755134771, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTEzNDc3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T07:28:01Z", "updated_at": "2021-01-06T07:28:01Z", "author_association": "OWNER", "body": "With this structure it will become possible to stream non-newline-delimited JSON array-of-objects too - the `stream_rows()` method could output `[` first, then each row followed by a comma, then `]` after the very last row.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 749283032, "label": "register_output_renderer() should support streaming data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1176#issuecomment-755159583", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1176", "id": 755159583, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTE1OTU4Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T08:28:20Z", "updated_at": "2021-01-06T08:28:20Z", "author_association": "OWNER", "body": "I used `from datasette.utils import path_with_format` in https://github.com/simonw/datasette-export-notebook/blob/0.1/datasette_export_notebook/__init__.py just now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 779691739, "label": "Policy on documenting \"public\" datasette.utils functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1178#issuecomment-755156606", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1178", "id": 755156606, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTE1NjYwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T08:21:49Z", "updated_at": "2021-01-06T08:21:49Z", "author_association": "OWNER", "body": "https://github.com/simonw/datasette-export-notebook/blob/aec398eab4f34791d240d7bc47b6eec575b357be/datasette_export_notebook/__init__.py#L18-L23\r\n\r\nMaybe this is a bug in `datasette.absolute_url`? Perhaps it doesn't take the scheme into account.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780267857, "label": "Use force_https_urls on when deploying with Cloud Run"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1178#issuecomment-755157066", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1178", "id": 755157066, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTE1NzA2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T08:22:47Z", "updated_at": "2021-01-06T08:22:47Z", "author_association": "OWNER", "body": "Weird... https://github.com/simonw/datasette/blob/a882d679626438ba0d809944f06f239bcba8ee96/datasette/app.py#L609-L613\r\n\r\n```python\r\n    def absolute_url(self, request, path):\r\n        url = urllib.parse.urljoin(request.url, path)\r\n        if url.startswith(\"http://\") and self.setting(\"force_https_urls\"):\r\n            url = \"https://\" + url[len(\"http://\") :]\r\n        return url\r\n```\r\nThat looks like it should work. Needs more digging.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780267857, "label": "Use force_https_urls on when deploying with Cloud Run"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1178#issuecomment-755157281", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1178", "id": 755157281, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTE1NzI4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T08:23:14Z", "updated_at": "2021-01-06T08:23:14Z", "author_association": "OWNER", "body": "https://latest-with-plugins.datasette.io/-/settings says `\"force_https_urls\": false`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780267857, "label": "Use force_https_urls on when deploying with Cloud Run"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1178#issuecomment-755157732", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1178", "id": 755157732, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTE1NzczMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T08:24:12Z", "updated_at": "2021-01-06T08:24:12Z", "author_association": "OWNER", "body": "https://latest-with-plugins.datasette.io/fixtures/sortable.json has the bug too - the `next_url` is `http://` when it should be `https://`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780267857, "label": "Use force_https_urls on when deploying with Cloud Run"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1178#issuecomment-755158310", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1178", "id": 755158310, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTE1ODMxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T08:25:31Z", "updated_at": "2021-01-06T08:25:31Z", "author_association": "OWNER", "body": "Moving this to the Datasette repo.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780267857, "label": "Use force_https_urls on when deploying with Cloud Run"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1178#issuecomment-755160187", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1178", "id": 755160187, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTE2MDE4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T08:29:35Z", "updated_at": "2021-01-06T08:29:35Z", "author_association": "OWNER", "body": "https://latest-with-plugins.datasette.io/-/asgi-scope\r\n\r\n```\r\n{'asgi': {'spec_version': '2.1', 'version': '3.0'},\r\n 'client': ('169.254.8.129', 54971),\r\n 'headers': [(b'host', b'latest-with-plugins.datasette.io'),\r\n             (b'user-agent',\r\n              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko'\r\n              b'/20100101 Firefox/84.0'),\r\n             (b'accept',\r\n              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'\r\n              b'webp,*/*;q=0.8'),\r\n             (b'accept-language', b'en-US,en;q=0.5'),\r\n             (b'dnt', b'1'),\r\n             (b'cookie',\r\n              b'_ga_LL6M7BK6D4=GS1.1.1609886546.49.1.1609886923.0; _ga=GA1.1'\r\n              b'.894633707.1607575712'),\r\n             (b'upgrade-insecure-requests', b'1'),\r\n             (b'x-client-data', b'CgSL6ZsV'),\r\n             (b'x-cloud-trace-context',\r\n              b'e776af843c657d2a3da28a73b726e6fe/14187666787557102189;o=1'),\r\n             (b'x-forwarded-for', b'148.64.98.14'),\r\n             (b'x-forwarded-proto', b'https'),\r\n             (b'forwarded', b'for=\"148.64.98.14\";proto=https'),\r\n             (b'accept-encoding', b'gzip, deflate, br'),\r\n             (b'content-length', b'0')],\r\n 'http_version': '1.1',\r\n 'method': 'GET',\r\n 'path': '/-/asgi-scope',\r\n 'query_string': b'',\r\n 'raw_path': b'/-/asgi-scope',\r\n 'root_path': '',\r\n 'scheme': 'http',\r\n 'server': ('169.254.8.130', 8080),\r\n 'type': 'http'}\r\n```\r\nNote the `'scheme': 'http'` but also the `(b'x-forwarded-proto', b'https')`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780267857, "label": "Use force_https_urls on when deploying with Cloud Run"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1178#issuecomment-755163886", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1178", "id": 755163886, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTE2Mzg4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T08:37:51Z", "updated_at": "2021-01-06T08:37:51Z", "author_association": "OWNER", "body": "Easiest fix would be for `publish cloudrun` to set `force_https_urls`:\r\n\r\n`datasette publish now` used to do this: https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/publish/now.py#L59-L63", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780267857, "label": "Use force_https_urls on when deploying with Cloud Run"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1178#issuecomment-755468795", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1178", "id": 755468795, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTQ2ODc5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T18:14:35Z", "updated_at": "2021-01-06T18:14:35Z", "author_association": "OWNER", "body": "Deploying that change now to test it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780267857, "label": "Use force_https_urls on when deploying with Cloud Run"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1178#issuecomment-755476820", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1178", "id": 755476820, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTQ3NjgyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T18:24:47Z", "updated_at": "2021-01-06T18:24:47Z", "author_association": "OWNER", "body": "Issue fixed - https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on displays the correct schemes now.\r\n\r\nI can't think of a reason anyone on Cloud Run would ever NOT want the `force_https_urls` option, but just in case I've made it so if you pass `--extra-options --setting force_https_urls off` to `publish cloudrun` your setting will be respected.\r\n\r\nhttps://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/publish/cloudrun.py#L105-L110", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780267857, "label": "Use force_https_urls on when deploying with Cloud Run"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1179#issuecomment-755161574", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1179", "id": 755161574, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTE2MTU3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T08:32:31Z", "updated_at": "2021-01-06T08:32:31Z", "author_association": "OWNER", "body": "An optional `path` argument to https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette which shows the path WITHOUT the `.Notebook` extension would be useful here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780278550, "label": "Make original path available to render hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1179#issuecomment-755486103", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1179", "id": 755486103, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTQ4NjEwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T18:32:41Z", "updated_at": "2021-01-06T18:34:11Z", "author_association": "OWNER", "body": "This parameter will return the URL path, with querystring arguments, to the HTML version of the page - e.g. `/github/issue_comments` or `/github/issue_comments?_sort_desc=created_at`\r\n\r\nOpen questions:\r\n\r\n- What should it be called? `path` could be misleading since it also includes the querystring.\r\n- Should I provide a `url` or `full_url` version which includes `https://blah.com/...`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780278550, "label": "Make original path available to render hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1179#issuecomment-755489974", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1179", "id": 755489974, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTQ4OTk3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T18:35:24Z", "updated_at": "2021-01-06T18:35:24Z", "author_association": "OWNER", "body": "Django calls this ` HttpRequest.get_full_path()` - for the path plus the querystring.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780278550, "label": "Make original path available to render hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1179#issuecomment-755492945", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1179", "id": 755492945, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTQ5Mjk0NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T18:37:39Z", "updated_at": "2021-01-06T18:37:39Z", "author_association": "OWNER", "body": "I think I'll call this `full_path` for consistency with Django.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780278550, "label": "Make original path available to render hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1179#issuecomment-755495387", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1179", "id": 755495387, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTQ5NTM4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T18:39:23Z", "updated_at": "2021-01-06T18:39:23Z", "author_association": "OWNER", "body": "In that case maybe there are three new arguments: `path`, `full_path` and `url`.\r\n\r\nI'll also add `request.full_path` for consistency with these: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/asgi.py#L77-L90", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780278550, "label": "Make original path available to render hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1180#issuecomment-755500475", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1180", "id": 755500475, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTUwMDQ3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T18:43:41Z", "updated_at": "2021-01-06T18:43:41Z", "author_association": "OWNER", "body": "Relevant code: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/__init__.py#L919-L940", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 780767542, "label": "Lazily evaluated arguments for call_with_supported_arguments"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-755484384", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 755484384, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTQ4NDM4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T18:31:14Z", "updated_at": "2021-01-06T18:31:57Z", "author_association": "OWNER", "body": "In building https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on I discovered the following patterns for importing data into both Pandas and Observable/d3:\r\n```python\r\nimport pandas\r\ndf = pandas.read_json(\r\n  \"https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array\"\r\n)\r\n```\r\nAnd:\r\n```javascript\r\nd3 = require(\"d3@5\")\r\nrows = d3.json(\r\n    \"https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array\"\r\n)\r\n```\r\nOnce again I find myself torn on the best possible default. A list of JSON objects is instantly compatible with both `pandas.read_json()` and `d3.json()` - but it leaves nowhere to put the extra information like pagination and suchlike!\r\n\r\nEven given this I still think the correct default is an object with `\"rows\"`, `\"total\"` and `\"next_url\"` keys. I should commit to that and implement it - this thought exercise has been running for far too long.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}