{"html_url": "https://github.com/simonw/datasette/pull/416#issuecomment-473154643", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/416", "id": 473154643, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzE1NDY0Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T04:27:47Z", "updated_at": "2019-03-15T04:28:00Z", "author_association": "OWNER", "body": "Deployed a demo: https://datasette-optional-hash-demo.now.sh/\r\n\r\n    datasette publish now \\\r\n        ../demo-databses/russian-ads.db \\\r\n        ../demo-databses/polar-bears.db \\\r\n        --branch=optional-hash \\\r\n        -n datasette-optional-hash \\\r\n        --alias datasette-optional-hash-demo \\\r\n        --install=datasette-cluster-map \\\r\n        --install=datasette-json-html\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421348146, "label": "URL hashing now optional: turn on with --config hash_urls:1 (#418)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/416#issuecomment-473156513", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/416", "id": 473156513, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzE1NjUxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T04:40:29Z", "updated_at": "2019-03-15T04:40:29Z", "author_association": "OWNER", "body": "Still TODO: need to figure out what to do about cache TTL. Defaulting to 365 days no longer makes sense without the hash_urls setting.\r\n\r\nMaybe drop that setting default to 0?\r\n\r\nHere's the setting:\r\n\r\nhttps://github.com/simonw/datasette/blob/9743e1d91b5f0a2b3c1c0bd6ffce8739341f43c4/datasette/app.py#L84-L86\r\n\r\nAnd here's where it takes affect:\r\n\r\nhttps://github.com/simonw/datasette/blob/4462a5ab2817ac0d9ffe20dafbbf27c5c5b81466/datasette/views/base.py#L491-L501", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421348146, "label": "URL hashing now optional: turn on with --config hash_urls:1 (#418)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/414#issuecomment-473156774", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/414", "id": 473156774, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzE1Njc3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T04:42:06Z", "updated_at": "2019-03-15T04:42:06Z", "author_association": "OWNER", "body": "This has been bothering me as well, especially when I try to install `datasette` and `sqlite-utils` at the same time.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 415575624, "label": "datasette requires specific version of Click"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/411#issuecomment-473156905", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/411", "id": 473156905, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzE1NjkwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T04:42:58Z", "updated_at": "2019-03-15T04:42:58Z", "author_association": "OWNER", "body": "Have you tried this?\r\n\r\n    MakePoint(:Long || \", \" || :Lat)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 410384988, "label": "How to pass named parameter into spatialite MakePoint() function"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/415#issuecomment-473157770", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/415", "id": 473157770, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzE1Nzc3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T04:49:03Z", "updated_at": "2019-03-15T04:49:03Z", "author_association": "OWNER", "body": "Interesting idea. I can see how this would make sense if you are dealing with really long SQL queries.\r\n\r\nMy own example of a long query that might benefit from this: https://russian-ads-demo.herokuapp.com/russian-ads-a42c4e8?sql=select%0D%0A++++target_id%2C%0D%0A++++targets.name%2C%0D%0A++++count(*)+as+n%2C%0D%0A++++json_object(%0D%0A++++++++%22href%22%2C+%22%2Frussian-ads%2Ffaceted-targets%3Ftargets%3D%22+||+%0D%0A++++++++++++json_insert(%3Atargets%2C+%27%24[%27+||+json_array_length(%3Atargets)+||+%27]%27%2C+target_id)%0D%0A++++++++%2C%0D%0A++++++++%22label%22%2C+json_insert(%3Atargets%2C+%27%24[%27+||+json_array_length(%3Atargets)+||+%27]%27%2C+target_id)%0D%0A++++)+as+apply_this_facet%2C%0D%0A++++json_object(%0D%0A++++++++%22href%22%2C+%22%2Frussian-ads%2Fdisplay_ads%3F_targets_json%3D%22+||+%0D%0A++++++++++++json_insert(%3Atargets%2C+%27%24[%27+||+json_array_length(%3Atargets)+||+%27]%27%2C+target_id)%0D%0A++++++++%2C%0D%0A++++++++%22label%22%2C+%22See+%22+||+count(*)+||+%22+ads+matching+%22+||+json_insert(%3Atargets%2C+%27%24[%27+||+json_array_length(%3Atargets)+||+%27]%27%2C+target_id)%0D%0A++++)+as+browse_these_ads%0D%0Afrom+ad_targets%0D%0Ajoin+targets+on+ad_targets.target_id+%3D+targets.id%0D%0Awhere%0D%0A++++json_array_length(%3Atargets)+%3D%3D+0+or%0D%0A++++ad_id+in+(%0D%0A++++++++select+ad_id%0D%0A++++++++from+%22ad_targets%22%0D%0A++++++++where+%22ad_targets%22.target_id+in+(select+value+from+json_each(%3Atargets))%0D%0A++++++++group+by+%22ad_targets%22.ad_id%0D%0A++++++++having+count(distinct+%22ad_targets%22.target_id)+%3D+json_array_length(%3Atargets)%0D%0A++++)%0D%0A++++and+target_id+not+in+(select+value+from+json_each(%3Atargets))%0D%0Agroup+by%0D%0A++++target_id+order+by+n+desc%0D%0A&targets=[%22e6200%22]\r\n\r\nHaving a `show/hide` link would be an easy way to support this in the UI, and those could add/remove a `_hide_sql=1` parameter.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 418329842, "label": "Add query parameter to hide SQL textarea"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/412#issuecomment-473158506", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/412", "id": 473158506, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzE1ODUwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T04:53:53Z", "updated_at": "2019-03-15T04:53:53Z", "author_association": "OWNER", "body": "I've been thinking about how Datasette instances could query each other for a while - it's a really interesting direction.\r\n\r\nThere are some tricky problems to solve to get this to work. There's a SQLite mechanism called \"virtual table functions\" which can implement things like this, but it's not supported by Python's `sqlite3` module out of the box.\r\n\r\nhttps://github.com/coleifer/sqlite-vtfunc is a library that enables this feature. I experimented with using that to implement a function that scrapes HTML content (with an eye to accessing data from other APIs and Datasette instances) a while ago: https://github.com/coleifer/sqlite-vtfunc/issues/6\r\n\r\nThe bigger challenge is how to get this kind of thing to behave well within a Python 3 async environment. I have some ideas here but they're going to require some very crafty engineering.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 411257981, "label": "Linked Data(sette)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/416#issuecomment-473159679", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/416", "id": 473159679, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzE1OTY3OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T05:01:27Z", "updated_at": "2019-03-15T05:01:27Z", "author_association": "OWNER", "body": "Also: if the option is False and the user visits a URL with a hash in it, should we redirect them?\r\n\r\nI'm inclined to say no: furthermore, I'd be OK continuing to serve a far-future cache header for that case.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421348146, "label": "URL hashing now optional: turn on with --config hash_urls:1 (#418)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/413#issuecomment-473160476", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/413", "id": 473160476, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzE2MDQ3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T05:06:37Z", "updated_at": "2019-03-15T05:06:37Z", "author_association": "OWNER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 413887019, "label": "Update spatialite.rst"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/416#issuecomment-473160702", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/416", "id": 473160702, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzE2MDcwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T05:08:13Z", "updated_at": "2019-03-15T05:08:13Z", "author_association": "OWNER", "body": "This also needs extensive tests to ensure that with the option turned on all of the redirects behave as they should.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421348146, "label": "URL hashing now optional: turn on with --config hash_urls:1 (#418)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/415#issuecomment-473164038", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/415", "id": 473164038, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzE2NDAzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T05:31:21Z", "updated_at": "2019-03-15T05:31:21Z", "author_association": "OWNER", "body": "Demo: https://latest.datasette.io/fixtures-dd88475?sql=select+%2A+from+sortable+order+by+pk1%2C+pk2+limit+101\r\n\r\n<img width=\"761\" alt=\"Screen Shot 2019-03-14 at 10 30 47 PM\" src=\"https://user-images.githubusercontent.com/9599/54410469-d5631000-46a8-11e9-8f77-d8fb5dd8fd23.png\">\r\n\r\nv.s. https://latest.datasette.io/fixtures-dd88475?sql=select+%2A+from+sortable+order+by+pk1%2C+pk2+limit+101&_hide_sql=1\r\n\r\n<img width=\"761\" alt=\"Screen Shot 2019-03-14 at 10 31 05 PM\" src=\"https://user-images.githubusercontent.com/9599/54410489-e27fff00-46a8-11e9-9207-8245b62f1cf4.png\">\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 418329842, "label": "Add query parameter to hide SQL textarea"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/415#issuecomment-473217334", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/415", "id": 473217334, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzIxNzMzNA==", "user": {"value": 36796532, "label": "ad-si"}, "created_at": "2019-03-15T09:30:57Z", "updated_at": "2019-03-15T09:30:57Z", "author_association": "NONE", "body": "Awesome, thanks! \ud83d\ude01 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 418329842, "label": "Add query parameter to hide SQL textarea"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/417#issuecomment-473308631", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/417", "id": 473308631, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzMwODYzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T14:32:13Z", "updated_at": "2019-03-15T14:32:13Z", "author_association": "OWNER", "body": "This would allow Datasette to be easily used as a \"data library\" (like a data warehouse but less expectation of big data querying technology such as Presto).\r\n\r\nOne of the things I learned at the NICAR CAR 2019 conference in Newport Beach is that there is a very real need for some kind of easily accessible data library at most newsrooms.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421546944, "label": "Datasette Library"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/416#issuecomment-473310026", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/416", "id": 473310026, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzMxMDAyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T14:35:53Z", "updated_at": "2019-03-15T14:35:53Z", "author_association": "OWNER", "body": "See #418 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421348146, "label": "URL hashing now optional: turn on with --config hash_urls:1 (#418)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/417#issuecomment-473312514", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/417", "id": 473312514, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzMxMjUxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T14:42:07Z", "updated_at": "2019-03-17T22:12:30Z", "author_association": "OWNER", "body": "A neat ability of Datasette Library would be if it can work against other files that have been dropped into the folder. In particular: if a user drops a CSV file into the folder, how about automatically converting that CSV file to SQLite using [sqlite-utils](https://github.com/simonw/sqlite-utils)?", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421546944, "label": "Datasette Library"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/123#issuecomment-473313975", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/123", "id": 473313975, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzMxMzk3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T14:45:46Z", "updated_at": "2019-03-15T14:45:46Z", "author_association": "OWNER", "body": "I'm reopening this one as part of #417.\r\n\r\nFurther experience with Python's CSV standard library module has convinced me that pandas is not a required dependency for this. My [sqlite-utils](https://github.com/simonw/sqlite-utils) package can do most of the work here with very few dependencies.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 275125561, "label": "Datasette serve should accept paths/URLs to CSVs and other file formats"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/123#issuecomment-473323329", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/123", "id": 473323329, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3MzMyMzMyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-15T15:09:15Z", "updated_at": "2019-05-14T15:53:05Z", "author_association": "OWNER", "body": "How would Datasette accepting URLs work?\r\n\r\nI want to support not just SQLite files and CSVs but other extensible formats (geojson, Atom, shapefiles etc) as well.\r\n\r\nSo `datasette serve` needs to be able to take filepaths or URLs to a variety of different content types.\r\n\r\nIf it's a URL, we can use the first 200 downloaded bytes to decide which type of file it is. This is likely more reliable than hoping the web server provided the correct content-type.\r\n\r\nAlso: let's have a threshold for downloading to disk. We will start downloading to a temp file (location controlled by an environment variable) if either the content length header is above that threshold OR we hit that much data cached in memory already and don't know how much more is still to come.\r\n\r\nThere needs to be a command line option for saying \"grab from this URL but force treat it as CSV\" - same thing for files on disk.\r\n\r\n    datasette mydb.db --type=db http://blah/blah --type=csv\r\n\r\nIf you provide less `--type` options thatn you did URLs then the default behavior is used for all of the subsequent URLs.\r\n\r\nAuto detection could be tricky. Probably do this with a plugin hook.\r\n\r\nhttps://github.com/h2non/filetype.py is interesting but deals with images video etc so not right for this purpose. \r\n\r\nI think we need our own simple content sniffing code via a plugin hook.\r\n\r\nWhat if two plugin type hooks can both potentially handle a sniffed file? The CLI can quit and return an error saying content is ambiguous and you need to specify a `--type`, picking from the following list.\r\n\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 275125561, "label": "Datasette serve should accept paths/URLs to CSVs and other file formats"}, "performed_via_github_app": null}