{"html_url": "https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030726991", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/397", "id": 1030726991, "node_id": "IC_kwDOCGYnMM49b6VP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T01:13:58Z", "updated_at": "2022-02-06T01:13:58Z", "author_association": "OWNER", "body": "This is a good idea. We already have that parameter for the `table.create_index()` method: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite_utils.db.Table.create_index", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1123903919, "label": "Support IF NOT EXISTS for table creation"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030727979", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/397", "id": 1030727979, "node_id": "IC_kwDOCGYnMM49b6kr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T01:19:21Z", "updated_at": "2022-02-06T01:19:21Z", "author_association": "OWNER", "body": "Just noticed there's no explicit test coverage for the `db[\"table\"].create(...)` method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1123903919, "label": "Support IF NOT EXISTS for table creation"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030730108", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/397", "id": 1030730108, "node_id": "IC_kwDOCGYnMM49b7F8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T01:30:46Z", "updated_at": "2022-02-06T01:30:46Z", "author_association": "OWNER", "body": "Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/python-api.html#explicitly-creating-a-table", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1123903919, "label": "Support IF NOT EXISTS for table creation"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/400#issuecomment-1030730748", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/400", "id": 1030730748, "node_id": "IC_kwDOCGYnMM49b7P8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T01:34:46Z", "updated_at": "2022-02-06T01:34:46Z", "author_association": "OWNER", "body": "Actually this is not needed - there is already an option that does this, it's just called `--ignore` rather than `--if-not-exists`.\r\n\r\nThe lack of consistency here is a little annoying, but not annoying enough to justify making a backwards incompatible change.\r\n\r\n```\r\n% sqlite-utils create-table --help\r\nUsage: sqlite-utils create-table [OPTIONS] PATH TABLE COLUMNS...\r\n\r\n Add a table with the specified columns. Columns should be specified using\r\n name, type pairs, for example:\r\n\r\n sqlite-utils create-table my.db people \\\r\n id integer \\\r\n name text \\\r\n height float \\\r\n photo blob --pk id\r\n\r\nOptions:\r\n --pk TEXT Column to use as primary key\r\n --not-null TEXT Columns that should be created as NOT NULL\r\n --default ... Default value that should be set for a column\r\n --fk ... Column, other table, other column to set as a\r\n foreign key\r\n --ignore If table already exists, do nothing\r\n --replace If table already exists, replace it\r\n --load-extension TEXT SQLite extensions to load\r\n -h, --help Show this message and exit.\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1125077063, "label": "`sqlite-utils create-table` ... `--if-not-exists`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030732093", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/398", "id": 1030732093, "node_id": "IC_kwDOCGYnMM49b7k9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T01:41:37Z", "updated_at": "2022-02-06T01:41:37Z", "author_association": "OWNER", "body": "Yeah I'd like to avoid adding any geo-dependencies to `sqlite-utils` if I can avoid it. I'm fine using stuff that's going to be available in SpatiaLite itself (provided it's available as a SQLite module) since then I don't need to add any extra Python dependencies.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124237013, "label": "Add SpatiaLite helpers to CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030732222", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/398", "id": 1030732222, "node_id": "IC_kwDOCGYnMM49b7m-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T01:42:19Z", "updated_at": "2022-02-06T01:42:28Z", "author_association": "OWNER", "body": "Adding some thoughts to:\r\n- #399 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124237013, "label": "Add SpatiaLite helpers to CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030732909", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030732909, "node_id": "IC_kwDOCGYnMM49b7xt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T01:47:06Z", "updated_at": "2022-02-06T01:47:06Z", "author_association": "OWNER", "body": "Here's an idea for an API design:\r\n```python\r\ngeojson_geometry = {} # ... GeoJSON goes here\r\ndb[\"places\"].insert(\r\n {\"name\": \"Wales\", \"geometry\": geojson_geometry},\r\n geojson=\"geometry\"\r\n)\r\n```\r\nThat `geojson=` parameter takes either a single column name or an iterable of column names. Any column in that list is expected to be a compatible `geometry` and the correct conversion functions will be applied.\r\n\r\nThat solves for GeoJSON, but it's a bit ugly. Should I add `wkt=` and maybe even `kml=` and `gml=` and so-on too? Definitely not, that's way too many ugly and inscrutable new parameters.\r\n\r\nMore importantly: if I want to support the following how would I do it?\r\n\r\n```python\r\ndb[\"places\"].insert(\r\n {\"name\": \"London\", \"point\": (51.509865, -0.118092)}\r\n)\r\n```\r\nHere I want to provide a `(latitude, longitude)` pair and have it inserted correctly into a `point` column.\r\n\r\nCould do this, but again it's messy:\r\n```python\r\ndb[\"places\"].insert(\r\n {\"name\": \"London\", \"point\": (51.509865, -0.118092)},\r\n point=\"point\"\r\n)\r\n```\r\nAnd again, what about those `(longitude, latitude)` people?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/401#issuecomment-1030734937", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/401", "id": 1030734937, "node_id": "IC_kwDOCGYnMM49b8RZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:02:24Z", "updated_at": "2022-02-06T02:02:24Z", "author_association": "OWNER", "body": "The example also doesn't work right now - the code that fetches from Who's On First gets a 403 forbidden error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1125081640, "label": "Update SpatiaLite example in the documentation"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/401#issuecomment-1030735372", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/401", "id": 1030735372, "node_id": "IC_kwDOCGYnMM49b8YM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:05:03Z", "updated_at": "2022-02-06T02:05:03Z", "author_association": "OWNER", "body": "Improved version: https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-column-values-using-sql-functions", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1125081640, "label": "Update SpatiaLite example in the documentation"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030735774", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030735774, "node_id": "IC_kwDOCGYnMM49b8ee", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:08:19Z", "updated_at": "2022-02-06T02:08:59Z", "author_association": "OWNER", "body": "Maybe I should leave this entirely up to documented patterns in the `conversions={}` dictionary?\r\n\r\nBut even that's not ideal for the co-ordinate case. Consider the following:\r\n\r\n```python\r\ndb[\"places\"].insert(\r\n {\"name\": \"London\", \"point\": (51.509865, -0.118092)},\r\n conversions={\"point\": \"GeomFromText(?, 4326)\"},\r\n)\r\n```\r\nThe challenge here is that the SpatiaLite function `GeomFromText()` expects a WKT string, which looks like this:\r\n\r\n POINT(-0.118092 51.509865)\r\n\r\nThe existing `conversions=` mechanism doesn't support applying Python code to convert the `(lat, lon)` tuple to that value.\r\n\r\nIt doesn't even support passing a Python tuple as a `?` parameter - so I don't think I could come up with a SQL string that would do the right thing here either.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736047", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030736047, "node_id": "IC_kwDOCGYnMM49b8iv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:10:18Z", "updated_at": "2022-02-06T02:10:18Z", "author_association": "OWNER", "body": "So maybe back to that earlier idea where the code introspects the table, figures out that `\"point\"` is a geometry table of type POINT, then applies the necessary conversions to the raw Python data?\r\n\r\nThat feels overly-complicated to me, especially since nothing else in the `.insert()` method currently relies on table introspection.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736589", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030736589, "node_id": "IC_kwDOCGYnMM49b8rN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:14:52Z", "updated_at": "2022-02-06T02:14:52Z", "author_association": "OWNER", "body": "Another idea: introduce a helper function transform pattern, something a bit like this:\r\n\r\n```python\r\n\r\ntransformer = make_transformer({\r\n \"point\": lambda pair: \"POINT({} {})\".format(pair[1], pair[0])\r\n})\r\n\r\ndb[\"places\"].insert_all(\r\n transformer([{\"name\": \"London\", \"point\": (51.509865, -0.118092)}])\r\n conversions={\"point\": \"GeomFromText(?, 4326)\"},\r\n)\r\n```\r\nThe `make_transformer(...)` function builds an object that can work as a wrapping iterator, applying those transform functions to everything in the sequence that it wraps.\r\n\r\nSo the above code would handle converting `(lat, lon)` to `POINT(lon lat)` - then the `conversions=` applies `GeomFromText`.\r\n\r\nNaming is a challenge here: `.transform()` and `.convert()` and `conversions=` all have existing meanings within the `sqlite-utils` Python library.\r\n\r\nIt's also a bit of a messy way of solving this. It's not exactly a smooth API for inserting a bunch of lat/lon coordinate pairs!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736848", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030736848, "node_id": "IC_kwDOCGYnMM49b8vQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:17:35Z", "updated_at": "2022-02-06T02:17:35Z", "author_association": "OWNER", "body": "Note that GeoJSON itself uses `(longitude, latitude)` so I should probably stick to that order here too. https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.1", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030738023", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030738023, "node_id": "IC_kwDOCGYnMM49b9Bn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:28:05Z", "updated_at": "2022-02-06T02:29:24Z", "author_association": "OWNER", "body": "Here's the definitive guide to `latitude, longitude` v.s. `longitude, latitude`: https://macwright.com/lonlat/\r\n\r\n> Which is right?\r\n>\r\n> Neither. This is an opinion with no right answer. Geographical tradition favors lat, lon. Math and software prefer lon, lat.\r\n\r\nI asked on Twitter here: https://twitter.com/simonw/status/1490148001569906688", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030739566, "node_id": "IC_kwDOCGYnMM49b9Zu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:45:25Z", "updated_at": "2022-02-06T02:50:27Z", "author_association": "OWNER", "body": "Another idea - my favourite option so far:\r\n```python\r\nfrom sqlite_utils.utils import LongitudeLatitude\r\n\r\ndb[\"places\"].insert(\r\n {\r\n \"name\": \"London\",\r\n \"point\": (-0.118092, 51.509865)\r\n },\r\n conversions={\"point\": LongitudeLatitude},\r\n)\r\n```\r\nHere `LongitudeLatitude` is a magical value which does TWO things: it sets up the `GeomFromText(?, 4326)` SQL function, and it handles converting the `(51.509865, -0.118092)` tuple into a `POINT({} {})` string.\r\n\r\nThis would involve a change to the `conversions=` contract - where it usually expects a SQL string fragment, but it can also take an object which combines that SQL string fragment with a Python conversion function.\r\n\r\nBest of all... this resolves the `lat, lon` v.s. `lon, lat` dilemma because you can use `from sqlite_utils.utils import LongitudeLatitude` OR `from sqlite_utils.utils import LatitudeLongitude` depending on which you prefer!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740570", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030740570, "node_id": "IC_kwDOCGYnMM49b9pa", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:56:17Z", "updated_at": "2022-02-06T02:57:00Z", "author_association": "OWNER", "body": "Thinking about types. The type of the `conversions` parameter right now is a bit lazy:\r\n\r\n```python\r\nconversions: Optional[dict] = None,\r\n```\r\nThat becomes:\r\n```python\r\nOptional[Dict[str, Union[str, Conversion]]]\r\n```\r\nWhere `Conversion` is an abstract base class which expects implementations to have a `.sql() -> str` and a `.convert(value) -> str` method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740653", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030740653, "node_id": "IC_kwDOCGYnMM49b9qt", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-02-06T02:57:17Z", "updated_at": "2022-02-06T02:57:17Z", "author_association": "CONTRIBUTOR", "body": "I like the idea of having stock conversions you could import. I'd actually move them to a dedicated module (call it `sqlite_utils.conversions` or something), because it's different from other utilities. Maybe they even take configuration, or they're composable.\r\n\r\n```python\r\nfrom sqlite_utils.conversions import LongitudeLatitude\r\n\r\ndb[\"places\"].insert(\r\n {\r\n \"name\": \"London\",\r\n \"lng\": -0.118092,\r\n \"lat\": 51.509865,\r\n },\r\n conversions={\"point\": LongitudeLatitude(\"lng\", \"lat\")},\r\n)\r\n```\r\n\r\nI would definitely use that for every CSV I get with lat/lng columns where I actually need GeoJSON.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740771", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030740771, "node_id": "IC_kwDOCGYnMM49b9sj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:58:29Z", "updated_at": "2022-02-06T02:58:29Z", "author_association": "OWNER", "body": "That example you have there is really neat - I like the idea that they can also be used to populate completely new columns that are derived from the other column inputs.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740826", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030740826, "node_id": "IC_kwDOCGYnMM49b9ta", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-02-06T02:59:10Z", "updated_at": "2022-02-06T02:59:10Z", "author_association": "CONTRIBUTOR", "body": "All this said, I don't think it's unreasonable to point people to dedicated tools like `geojson-to-sqlite`. If I'm dealing with a bunch of GeoJSON or Shapefiles, I need to something to read those anyway (or I need to figure out virtual tables). But something like this might make it easier to build those libraries, or standardize the underlying parts.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740846", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030740846, "node_id": "IC_kwDOCGYnMM49b9tu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T02:59:21Z", "updated_at": "2022-02-06T02:59:21Z", "author_association": "OWNER", "body": "I wonder if there are any interesting non-geospatial canned conversions that it would be worth including?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740963", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030740963, "node_id": "IC_kwDOCGYnMM49b9vj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T03:00:33Z", "updated_at": "2022-02-06T03:00:33Z", "author_association": "OWNER", "body": "Yeah, having this be a general purpose mechanism which has a few canned examples for handling geospatial stuff is a lot neater than having a mechanism for this that's exclusive to SpatiaLite.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030741289", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030741289, "node_id": "IC_kwDOCGYnMM49b90p", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-02-06T03:03:43Z", "updated_at": "2022-02-06T03:03:43Z", "author_association": "CONTRIBUTOR", "body": "> I wonder if there are any interesting non-geospatial canned conversions that it would be worth including?\r\n\r\nOff the top of my head:\r\n\r\n- Un-nesting JSON objects into columns\r\n- Splitting arrays\r\n- Normalizing dates and times\r\n- URL munging with `urlparse`\r\n- Converting strings to numbers\r\n\r\nSome of this is easy enough with SQL functions, some is easier in Python. Maybe that's where having pre-built classes gets really handy, because it saves you from thinking about which way it's implemented.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/957#issuecomment-1030761625", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/957", "id": 1030761625, "node_id": "IC_kwDOBm6k_c49cCyZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T06:30:32Z", "updated_at": "2022-02-06T06:31:44Z", "author_association": "OWNER", "body": "I'm just going with:\r\n\r\n```python\r\nfrom datasette import Response\r\nfrom datasette import Forbidden\r\nfrom datasette import NotFound\r\nfrom datasette import hookimpl\r\nfrom datasette import actor_matches_allow\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688622148, "label": "Simplify imports of common classes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/957#issuecomment-1030762140", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/957", "id": 1030762140, "node_id": "IC_kwDOBm6k_c49cC6c", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T06:36:41Z", "updated_at": "2022-02-06T06:36:41Z", "author_association": "OWNER", "body": "Documented here: https://docs.datasette.io/en/latest/internals.html#import-shortcuts", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688622148, "label": "Simplify imports of common classes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1176#issuecomment-1030762279", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1176", "id": 1030762279, "node_id": "IC_kwDOBm6k_c49cC8n", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T06:38:08Z", "updated_at": "2022-02-06T06:41:37Z", "author_association": "OWNER", "body": "Might do this using Sphinx auto-generated function and class documentation hooks, as seen here in `sqlite-utils`: https://sqlite-utils.datasette.io/en/stable/python-api.html#spatialite-helpers\r\n\r\nThis would encourage me to add really good docstrings.\r\n\r\n```\r\n.. _python_api_gis_find_spatialite:\r\n\r\nFinding SpatiaLite\r\n------------------\r\n\r\n.. autofunction:: sqlite_utils.utils.find_spatialite\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 779691739, "label": "Policy on documenting \"public\" datasette.utils functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030807433", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030807433, "node_id": "IC_kwDOCGYnMM49cN-J", "user": {"value": 6025893, "label": "chris48s"}, "created_at": "2022-02-06T10:54:09Z", "updated_at": "2022-02-06T10:54:09Z", "author_association": "NONE", "body": "> Interesting that some accept an SRID and others do not - presumably GeomFromGeoJSON() always uses SRID=4326?\r\n\r\nThe ewtk/ewkb ones don't accept an SRID is because ewkt encodes the SRID in the string, so you would do this with a wkt string:\r\n\r\n`GeomFromText('POINT(529090 179645)', 27700)`\r\n\r\nbut for ewkt it would be\r\n\r\n`GeomFromEWKT('SRID=27700;POINT(529090 179645)')`\r\n\r\nThe specs for KML and GeoJSON specify a Coordinate Reference System for the format\r\n\r\n- https://datatracker.ietf.org/doc/html/rfc7946#section-4\r\n- https://docs.opengeospatial.org/is/12-007r2/12-007r2.html#1274\r\n\r\nGML can specify the SRID in the XML at feature level e.g:\r\n\r\n```\r\n\r\n 529090, 179645\r\n\r\n```\r\n\r\nThere's a few more obscure formats in there, but broadly I think it is safe to assume an SRID param exists on the function for cases where the SRID is not implied by or specified in the input format.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030871591", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030871591, "node_id": "IC_kwDOCGYnMM49cdon", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T16:57:22Z", "updated_at": "2022-02-06T16:57:22Z", "author_association": "OWNER", "body": "I wonder if I could implement the above such that this *also* works:\r\n```python\r\ndb[\"places\"].insert(\r\n {\r\n \"name\": \"London\",\r\n \"point\": LongitudeLatitude(-0.118092, 51.509865)\r\n }\r\n)\r\n```\r\nThis feels like a very natural way to work with single inserts.\r\n\r\nThe challenge is writing the code inside `.insert_all()` such that it can handle these special objects in the input column values in addition to them being passed in `conversions=`.\r\n\r\nI'm feeling very good about this direction in general though, it feels like it takes the existing but not particularly elegant `conversions=` mechanism and upgrades it to be far more useful, while maintaining backwards compatibility.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030901189", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/402", "id": 1030901189, "node_id": "IC_kwDOCGYnMM49ck3F", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T19:48:36Z", "updated_at": "2022-02-06T19:48:52Z", "author_association": "OWNER", "body": "From [that thread](https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566), two extra ideas which it may be possible to support in a single implementation:\r\n\r\n```python\r\nfrom sqlite_utils.conversions import LongitudeLatitude\r\n\r\ndb[\"places\"].insert(\r\n {\r\n \"name\": \"London\",\r\n \"lng\": -0.118092,\r\n \"lat\": 51.509865,\r\n },\r\n conversions={\"point\": LongitudeLatitude(\"lng\", \"lat\")},\r\n)\r\n```\r\nAnd\r\n```python\r\ndb[\"places\"].insert(\r\n {\r\n \"name\": \"London\",\r\n \"point\": LongitudeLatitude(-0.118092, 51.509865)\r\n }\r\n)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1125297737, "label": "Advanced class-based `conversions=` mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030901853", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/402", "id": 1030901853, "node_id": "IC_kwDOCGYnMM49clBd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T19:52:10Z", "updated_at": "2022-02-06T19:52:10Z", "author_association": "OWNER", "body": "So the key idea here is to introduce a new abstract base class, `Conversion`, which has the following abilities:\r\n\r\n- Can wrap one or more Python values (if called using the constructor) such that the `.insert_all()` method knows how to transform those into a format that can be included in an insert - something like `GeomFromText(?, 4326)` with input `POINT(-0.118092 51.509865)`\r\n- Can be passed to `conversions={\"point\": LongitudeLatitude}` in a way that then knows to apply that conversion to every value in the `\"point\"` key of the data being inserted.\r\n- Maybe also extend `conversions=` to allow the definition of additional keys that use as input other rows? That's the `conversions={\"point\": LongitudeLatitude(\"lng\", \"lat\")}` example above - it may not be possible to get this working with the rest of the design though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1125297737, "label": "Advanced class-based `conversions=` mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030902158", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/399", "id": 1030902158, "node_id": "IC_kwDOCGYnMM49clGO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T19:53:54Z", "updated_at": "2022-02-06T19:53:54Z", "author_association": "OWNER", "body": "Moving the design of this new `Conversion` subclass mechanism to:\r\n- https://github.com/simonw/sqlite-utils/issues/402", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1124731464, "label": "Make it easier to insert geometries, with documentation and maybe code"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/932#issuecomment-1030940407", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/932", "id": 1030940407, "node_id": "IC_kwDOBm6k_c49cub3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T23:31:22Z", "updated_at": "2022-02-06T23:31:22Z", "author_association": "OWNER", "body": "Great argument for doing this from a conversation on Twitter about documentation-driven development:\r\n\r\n> Long ago, when the majority of commercial programs were desktop apps, I've read a very wise advice: The user manual should be written first, before even a single line if code.\r\n\r\nhttps://twitter.com/b11c/status/1490466703175823362", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 678760988, "label": "End-user documentation"}, "performed_via_github_app": null}