{"html_url": "https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030902102", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/402", "id": 1030902102, "node_id": "IC_kwDOCGYnMM49clFW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T19:53:34Z", "updated_at": "2022-02-08T07:40:34Z", "author_association": "OWNER", "body": "I like the idea that the contract for `Conversion` (or rather for its subclasses) is that it can wrap a Python value and then return both the SQL fragment - e.g. `GeomFromText(?, 4326)` - and the values that should be used as the SQL parameters.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1125297737, "label": "Advanced class-based `conversions=` mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030904948", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/402", "id": 1030904948, "node_id": "IC_kwDOCGYnMM49clx0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-06T20:09:42Z", "updated_at": "2022-02-08T07:40:44Z", "author_association": "OWNER", "body": "I think this is the code that needs to become aware of this system: https://github.com/simonw/sqlite-utils/blob/fea8c9bcc509bcae75e99ae8870f520103b9aa58/sqlite_utils/db.py#L2453-L2469\r\n\r\nThere's an earlier branch that runs for upserts which needs to be modified too: https://github.com/simonw/sqlite-utils/blob/fea8c9bcc509bcae75e99ae8870f520103b9aa58/sqlite_utils/db.py#L2417-L2440", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1125297737, "label": "Advanced class-based `conversions=` mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/26#issuecomment-1032120014", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/26", "id": 1032120014, "node_id": "IC_kwDOCGYnMM49hObO", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-02-08T01:32:34Z", "updated_at": "2022-02-08T01:32:34Z", "author_association": "CONTRIBUTOR", "body": "if you are curious about prior art, https://github.com/jsnell/json-to-multicsv is really good!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 455486286, "label": "Mechanism for turning nested JSON into foreign keys / many-to-many"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032126353", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/403", "id": 1032126353, "node_id": "IC_kwDOCGYnMM49hP-R", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-02-08T01:45:15Z", "updated_at": "2022-02-08T01:45:31Z", "author_association": "CONTRIBUTOR", "body": "you can hack something like this to achieve this result:\r\n\r\n`sqlite-utils convert my_database my_table rowid \"{'id': value}\" --multi`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1126692066, "label": "Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032294365", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/402", "id": 1032294365, "node_id": "IC_kwDOCGYnMM49h4_d", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-08T07:32:09Z", "updated_at": "2022-02-08T07:34:41Z", "author_association": "OWNER", "body": "I have an idea for how that third option could work - the one that creates a new column using values from the existing ones:\r\n```python\r\ndb[\"places\"].insert(\r\n    {\r\n        \"name\": \"London\",\r\n        \"lng\": -0.118092,\r\n        \"lat\": 51.509865,\r\n    },\r\n    conversions={\"point\": LongitudeLatitude(\"lng\", \"lat\")},\r\n)\r\n```\r\nHow about specifying that the values in that `conversion=` dictionary can be:\r\n\r\n- A SQL string fragment (as currently implemented)\r\n- A subclass of `Conversion` as described above\r\n- Or... a callable function that takes the row as an argument and returns either a `Conversion` subclass instance or a literal value to be jnserted into the database (a string, int or float)\r\n\r\nThen you could do this:\r\n\r\n```python\r\ndb[\"places\"].insert(\r\n    {\r\n        \"name\": \"London\",\r\n        \"lng\": -0.118092,\r\n        \"lat\": 51.509865,\r\n    },\r\n    conversions={\r\n        \"point\": lambda row: LongitudeLatitude(\r\n            row[\"lng\"], row[\"lat\"]\r\n        )\r\n    }\r\n)\r\n```\r\nSomething I really like about this is that it expands the abilities of `conversions=` beyond the slightly obscure need to customize the SQL fragment into something that can solve other data insertion cleanup problems too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1125297737, "label": "Advanced class-based `conversions=` mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032296717", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/402", "id": 1032296717, "node_id": "IC_kwDOCGYnMM49h5kN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-08T07:35:46Z", "updated_at": "2022-02-08T07:35:46Z", "author_association": "OWNER", "body": "I'm going to write the documentation for this first, before the implementation, so I can see if it explains cleanly enough that the design appears to be sound.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1125297737, "label": "Advanced class-based `conversions=` mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032732242", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/402", "id": 1032732242, "node_id": "IC_kwDOCGYnMM49jj5S", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-02-08T15:26:59Z", "updated_at": "2022-02-08T15:26:59Z", "author_association": "CONTRIBUTOR", "body": "What if you did something like this:\r\n\r\n```python\r\n\r\nclass Conversion:\r\n    def __init__(self, *args, **kwargs):\r\n        \"Put whatever settings you need here\"\r\n\r\n    def python(self, row, column, value): # not sure on args here\r\n        \"Python step to transform value\"\r\n        return value\r\n\r\n    def sql(self, row, column, value):\r\n        \"Return the actual sql that goes in the insert/update step, and maybe params\"\r\n        # value is the return of self.python()\r\n        return value, []\r\n```\r\n\r\nThis way, you're always passing an instance, which has methods that do the conversion. (Or you're passing a SQL string, as you would now.) The `__init__` could take column names, or SRID, or whatever other setup state you need per row, but the row is getting processed with the `python` and `sql` methods (or whatever you want to call them). This is pretty rough, so do what you will with names and args and such.\r\n\r\nYou'd then use it like this:\r\n\r\n```python\r\n# subclass might be unneeded here, if methods are present\r\nclass LngLatConversion(Conversion):\r\n    def __init__(self, x=\"longitude\", y=\"latitude\"):\r\n        self.x = x\r\n        self.y = y\r\n\r\n    def python(self, row, column, value):\r\n        x = row[self.x]\r\n        y = row[self.y]\r\n        return x, y\r\n\r\n    def sql(self, row, column, value):\r\n        # value is now a tuple, returned above\r\n        s = \"GeomFromText(POINT(? ?))\"\r\n        return s, value\r\n\r\ntable.insert_all(rows, conversions={\"point\": LngLatConversion(\"lng\", \"lat\"))}\r\n```\r\n\r\nI haven't thought through all the implementation details here, and it'll probably break in ways I haven't foreseen, but wanted to get this idea out of my head. Hope it helps.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1125297737, "label": "Advanced class-based `conversions=` mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032976720", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/403", "id": 1032976720, "node_id": "IC_kwDOCGYnMM49kflQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-08T19:23:05Z", "updated_at": "2022-02-08T19:23:27Z", "author_association": "OWNER", "body": "This is already possible using `sqlite-utils transform` like so:\r\n\r\n```\r\n% echo '[{\"name\": \"Barry\"}, {\"name\": \"Sandra\"}]' | sqlite-utils insert rowid.db records -\r\n% sqlite-utils schema rowid.db                                                           \r\nCREATE TABLE [records] (\r\n   [name] TEXT\r\n);\r\n% sqlite-utils rows rowid.db records\r\n[{\"name\": \"Barry\"},\r\n {\"name\": \"Sandra\"}]\r\n% sqlite-utils transform rowid.db records --pk id\r\n% sqlite-utils rows rowid.db records\r\n[{\"id\": 1, \"name\": \"Barry\"},\r\n {\"id\": 2, \"name\": \"Sandra\"}]\r\n% sqlite-utils schema rowid.db\r\nCREATE TABLE \"records\" (\r\n   [id] INTEGER PRIMARY KEY,\r\n   [name] TEXT\r\n);\r\n% echo '[{\"name\": \"Barry 2\"}, {\"name\": \"Sandra 2\"}]' | sqlite-utils insert rowid.db records -\r\n% sqlite-utils rows rowid.db records                                                         \r\n[{\"id\": 1, \"name\": \"Barry\"},\r\n {\"id\": 2, \"name\": \"Sandra\"},\r\n {\"id\": 3, \"name\": \"Barry 2\"},\r\n {\"id\": 4, \"name\": \"Sandra 2\"}]\r\n```\r\nIt's not covered in the documentation though: https://sqlite-utils.datasette.io/en/3.23/cli.html#transforming-tables", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1126692066, "label": "Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032987901", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/403", "id": 1032987901, "node_id": "IC_kwDOCGYnMM49kiT9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-08T19:36:06Z", "updated_at": "2022-02-08T19:36:06Z", "author_association": "OWNER", "body": "New documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#adding-a-primary-key-to-a-rowid-table", "reactions": "{\"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1126692066, "label": "Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`"}, "performed_via_github_app": null}