{"html_url": "https://github.com/simonw/datasette/issues/1576#issuecomment-1017112543", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1576", "id": 1017112543, "node_id": "IC_kwDOBm6k_c48n-ff", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-20T04:35:00Z", "updated_at": "2022-02-05T04:33:46Z", "author_association": "OWNER", "body": "I dropped support for Python 3.6 in fae3983c51f4a3aca8335f3e01ff85ef27076fbf so now free to use `contextvars` for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1087181951, "label": "Traces should include SQL executed by subtasks created with `asyncio.gather`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030454114", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/398", "id": 1030454114, "node_id": "IC_kwDOCGYnMM49a3ti", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T00:14:47Z", "updated_at": "2022-02-05T00:14:47Z", "author_association": "OWNER", "body": "I like these designs a lot. I would suggest `sqlite-utils create database.db --init-spatialite` there for consistency with the `sqlite-utils init-spatialite database.db` command.\r\n\r\nThe other part of this story is how we support actually inserting spatial data from the command-line. I opened an issue about the challenges in doing that for the Python API here - #399 - but we need a good answer for the CLI too.\r\n\r\nI don't yet have any good ideas here. The `conversions=` option in the Python library was designed to cover these kinds of cases but it's pretty clunky and I don't think it's very widely used: https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124237013, "label": "Add SpatiaLite helpers to CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030455715", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030455715, "node_id": "IC_kwDOCGYnMM49a4Gj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T00:15:28Z", "updated_at": "2022-02-05T00:15:28Z", "author_association": "OWNER", "body": "The `conversions=` argument to `.insert()` and friends is designed to handle this case, but I don't think it's very elegant: https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions\r\n\r\n```python\r\ndb[\"places\"].insert(\r\n {\"name\": \"Wales\", \"geometry\": wkt},\r\n conversions={\"geometry\": \"GeomFromText(?, 4326)\"},\r\n)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030456717", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/398", "id": 1030456717, "node_id": "IC_kwDOCGYnMM49a4WN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T00:16:42Z", "updated_at": "2022-02-05T00:16:42Z", "author_association": "OWNER", "body": "> The one thing worth highlighting in docs is that geometry columns can only be added to existing tables. Trying to add a geometry column to a table that doesn't exist yet might mean you have a schema like `{\"rowid\": int, \"geometry\": bytes}`. Might be worth nudging people to explicitly create a table first, then add geometry columns.\r\n\r\nThat's a good call. I'm happy for `sqlite-utils add-geometry-column` to throw an error if the table doesn't exist yet.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124237013, "label": "Add SpatiaLite helpers to CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030461163", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030461163, "node_id": "IC_kwDOCGYnMM49a5br", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T00:30:18Z", "updated_at": "2022-02-05T00:30:18Z", "author_association": "OWNER", "body": "I wonder what the most developer-friendly way to insert geometry data into SpatiaLite is?\r\n\r\nFrom https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html it looks like these are the main options:\r\n\r\n- `GeomFromText( wkt String [ , SRID Integer] )`\r\n- `GeomFromWKB( wkbGeometry Binary [ , SRID Integer] )`\r\n- `GeomFromKml( KmlGeometry String )`\r\n- `GeomFromGML( gmlGeometry String )`\r\n- `GeomFromGeoJSON( geoJSONGeometry String )`\r\n- `GeomFromEWKB( ewkbGeometry String )`\r\n- `GeomFromEWKT( ewktGeometry String )`\r\n- `GeomFromFGF( fgfGeometry Binary [ , SRID Integer] )`\r\n- `GeomFromTWKB( twkbGeometry BLOB [ , SRID Integer] )`\r\n- `GeomFromGPB( geom GPKG Blob Geometry )` - GeoPackage format\r\n- `GeomFromExifGpsBlob( image BLOB )`\r\n\r\nInteresting that some accept an SRID and others do not - presumably `GeomFromGeoJSON()` always uses SRID=4326?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030465557", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030465557, "node_id": "IC_kwDOCGYnMM49a6gV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T00:39:09Z", "updated_at": "2022-02-05T00:39:09Z", "author_association": "OWNER", "body": "I can't seem to get `GeomFromGeoJSON()` to work - example:\r\n\r\nhttps://calands.datasettes.com/calands?sql=select+IsValid%28SetSRID%28GeomFromGeoJSON%28%27%7B%0D%0A++++++++%22type%22%3A+%22Point%22%2C%0D%0A++++++++%22coordinates%22%3A+%5B%0D%0A++++++++++-94.921875%2C%0D%0A++++++++++45.460130637921004%0D%0A++++++++%5D%0D%0A++++++%7D%27%29%2C+4326%29%29\r\n\r\n```sql\r\nselect IsValid(SetSRID(GeomFromGeoJSON('{\r\n \"type\": \"Point\",\r\n \"coordinates\": [\r\n -94.921875,\r\n 45.460130637921004\r\n ]\r\n }'), 4326))\r\n```\r\nReturns `-1` suggesting the geometry is not valid. Just doing this (with or without that `SetSRID()` function) returns null:\r\n\r\n```sql\r\nselect SetSRID(GeomFromGeoJSON('{\r\n \"type\": \"Point\",\r\n \"coordinates\": [\r\n -94.921875,\r\n 45.460130637921004\r\n ]\r\n }'), 4326)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030466255", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030466255, "node_id": "IC_kwDOCGYnMM49a6rP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T00:41:35Z", "updated_at": "2022-02-05T00:42:23Z", "author_association": "OWNER", "body": "Wow, it was the newlines that broke it! This works fine: \r\n```sql\r\nselect AsWKT(SetSRID(GeomFromGeoJSON('{\"type\": \"Point\",\"coordinates\": [-94.921875,45.460130637921004]}'), 4326))\r\n```\r\nhttps://calands.datasettes.com/calands?sql=select+AsWKT%28SetSRID%28GeomFromGeoJSON%28%27%7B%22type%22%3A+%22Point%22%2C%22coordinates%22%3A+%5B-94.921875%2C45.460130637921004%5D%7D%27%29%2C+4326%29%29\r\n\r\nAnd removing `SetSRID()` returns exactly the same result:\r\n\r\nhttps://calands.datasettes.com/calands?sql=select+AsWKT%28GeomFromGeoJSON%28%27%7B%22type%22%3A+%22Point%22%2C%22coordinates%22%3A+%5B-94.921875%2C45.460130637921004%5D%7D%27%29%29", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030468418", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030468418, "node_id": "IC_kwDOCGYnMM49a7NC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T00:49:08Z", "updated_at": "2022-02-05T22:59:06Z", "author_association": "OWNER", "body": "I'm trying to think of ways to make this nicer from the perspective of someone calling the `.insert()` or `.insert_all()` methods against a table that has geometry columns.\r\n\r\nOne option would be for the code to introspect the table (if it exists) before running the insert, looking for any geometry columns.\r\n\r\nThis introspection isn't easy! The table schema just gives you `\"name_of_column\" point` or similar - to figure out the SRID and suchlike you need to consult the `geometry_columns` table, I think - which throws a 500 error on https://calands.datasettes.com/calands/geometry_columns for some reason. Also does the shape of that table change between SpatiaLite versions?\r\n\r\nAssuming we can introspect the table, what would we do with that information? We could add code that detects if the user attempted to pass GeoJSON objects and automatically inserts a `GeomFromGeoJSON()` function call - but detecting GeoJSON is a bit weird, and GeoJSON also isn't necessarily the nicest format for populating e.g. latitude/longitude points.\r\n\r\nMaybe we just support the simplest possible case: a tuple of floats, which we assume is `latitude, longitude` (or should we expect `longitude, latitude`, the eternal debate?) - if those are used against a geometry table (especially a point table) we assume they are coordinates that need to be converted using `GeomFromText('POINT(...`.\r\n\r\nNot crazy about either of these ideas. Is there something better?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030521533", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/398", "id": 1030521533, "node_id": "IC_kwDOCGYnMM49bIK9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T04:25:49Z", "updated_at": "2022-02-05T04:25:49Z", "author_association": "OWNER", "body": "For ingesting geometry data from the command-line maybe GeoJSON would be the best route?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124237013, "label": "Add SpatiaLite helpers to CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1576#issuecomment-1030525218", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1576", "id": 1030525218, "node_id": "IC_kwDOBm6k_c49bJEi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T04:45:11Z", "updated_at": "2022-02-05T04:45:11Z", "author_association": "OWNER", "body": "Got a prototype working with `contextvars` - it identified two parallel executing queries using the patch from above:\r\n\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1087181951, "label": "Traces should include SQL executed by subtasks created with `asyncio.gather`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1576#issuecomment-1030528532", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1576", "id": 1030528532, "node_id": "IC_kwDOBm6k_c49bJ4U", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T05:09:57Z", "updated_at": "2022-02-05T05:09:57Z", "author_association": "OWNER", "body": "Needs documentation. I'll document `from datasette.tracer import trace` too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1087181951, "label": "Traces should include SQL executed by subtasks created with `asyncio.gather`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1576#issuecomment-1030530071", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1576", "id": 1030530071, "node_id": "IC_kwDOBm6k_c49bKQX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T05:21:35Z", "updated_at": "2022-02-05T05:21:35Z", "author_association": "OWNER", "body": "New documentation section: https://docs.datasette.io/en/latest/internals.html#datasette-tracer", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1087181951, "label": "Traces should include SQL executed by subtasks created with `asyncio.gather`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030534868", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/398", "id": 1030534868, "node_id": "IC_kwDOCGYnMM49bLbU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T06:03:38Z", "updated_at": "2022-02-05T06:03:38Z", "author_association": "OWNER", "body": "@eyeseast how do you usually insert geometries at the moment?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124237013, "label": "Add SpatiaLite helpers to CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030629879", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/398", "id": 1030629879, "node_id": "IC_kwDOCGYnMM49bin3", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-02-05T13:57:33Z", "updated_at": "2022-02-05T19:49:38Z", "author_association": "CONTRIBUTOR", "body": "I'm mostly using [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite) at the moment. Even with shapefiles, I'm usually converting to GeoJSON and projecting to EPSG:4326 (with [ogr2ogr](https://gdal.org/programs/ogr2ogr.html)) first. \r\n\r\nI think an open question here is how much you want to leave to external libraries and how much you want here. My thinking has been that adding Spatialite helpers here would make external stuff easier, but it would be nice to have some standard way to insert geometries.\r\n\r\nI'm in the middle of adding GeoJSON and Spatialite support to [geocode-sqlite](https://github.com/eyeseast/geocode-sqlite), and that will probably use WKT. Since that's all points, I think I can just make the string inline. But for polygons, I'd generally use Shapely, which probably isn't a dependency you want to add to sqlite-utils.\r\n\r\nI've also been trying to get some of the approaches [here](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=Supporting+GeoJSON) to work, but haven't had any success so far.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124237013, "label": "Add SpatiaLite helpers to CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030712129", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030712129, "node_id": "IC_kwDOCGYnMM49b2tB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-05T23:08:45Z", "updated_at": "2022-02-05T23:08:45Z", "author_association": "OWNER", "body": "Useful thoughts on Twitter regarding making coordinate pairs easy and more complex shapes possible:\r\n\r\nhttps://twitter.com/dbreunig/status/1490099303888547843\r\n> That is exactly where I was going: two modes.\r\n>\r\n> 1. Heuristics and assumptions to get coordinates as a pair (in tuple) or as columns (look for lat, lon, latitude, longitude, etc).\r\n> 2. GIS mode with projections, polys, etc\r\n>\r\n> Make it easy for people with csvs of coordinates. If you're using Geojson or shp files, you have to specify.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}