{"html_url": "https://github.com/simonw/datasette/issues/1927#issuecomment-1335984268", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1927", "id": 1335984268, "node_id": "IC_kwDOBm6k_c5PoYCM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-03T00:26:26Z", "updated_at": "2022-12-03T00:26:26Z", "author_association": "OWNER", "body": "Also: the documentation should clarify that you can call this API multiple times when using the `rows` option.\r\n\r\n(It will probably grow `\"alter\": true` soon too).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1473411197, "label": "ignore:true/replace:true options for /db/-/create API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1930#issuecomment-1336017976", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1930", "id": 1336017976, "node_id": "IC_kwDOBm6k_c5PogQ4", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-12-03T02:30:21Z", "updated_at": "2022-12-03T02:30:21Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1930?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nBase: **90.42**% // Head: **90.42**% // No change to project coverage :thumbsup:\n> Coverage data is based on head [(`9928ff1`)](https://codecov.io/gh/simonw/datasette/pull/1930?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`cab5b60`)](https://codecov.io/gh/simonw/datasette/commit/cab5b60e09e94aca820dbec5308446a88c99ea3d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> Patch has no changes to coverable lines.\n\nAdditional details and impacted files
\n\n\n```diff\n@@ Coverage Diff @@\n## main #1930 +/- ##\n=======================================\n Coverage 90.42% 90.42% \n=======================================\n Files 36 36 \n Lines 5057 5057 \n=======================================\n Hits 4573 4573 \n Misses 484 484 \n```\n\n\n\nHelp us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n[^\\/\\.]+)/-/insert$\",\r\n )\r\n+ add_route(\r\n+ TableUpsertView.as_view(self),\r\n+ r\"/(?P[^\\/\\.]+)/(?P[^\\/\\.]+)/-/upsert$\",\r\n+ )\r\n add_route(\r\n TableDropView.as_view(self),\r\n r\"/(?P[^\\/\\.]+)/(?P[^\\/\\.]+)/-/drop$\",\r\ndiff --git a/datasette/views/table.py b/datasette/views/table.py\r\nindex 7ba78c11..ae0d6366 100644\r\n--- a/datasette/views/table.py\r\n+++ b/datasette/views/table.py\r\n@@ -1074,9 +1074,15 @@ class TableInsertView(BaseView):\r\n def __init__(self, datasette):\r\n self.ds = datasette\r\n \r\n- async def _validate_data(self, request, db, table_name):\r\n+ async def _validate_data(self, request, db, table_name, pks, upsert):\r\n errors = []\r\n \r\n+ pks_list = []\r\n+ if isinstance(pks, str):\r\n+ pks_list = [pks]\r\n+ else:\r\n+ pks_list = list(pks)\r\n+\r\n def _errors(errors):\r\n return None, errors, {}\r\n \r\n@@ -1135,6 +1141,15 @@ class TableInsertView(BaseView):\r\n # Validate columns of each row\r\n columns = set(await db.table_columns(table_name))\r\n for i, row in enumerate(rows):\r\n+ if upsert:\r\n+ # It MUST have the primary key\r\n+ missing_pks = [pk for pk in pks_list if pk not in row]\r\n+ if missing_pks:\r\n+ errors.append(\r\n+ 'Row {} is missing primary key column(s): \"{}\"'.format(\r\n+ i, '\", \"'.join(missing_pks)\r\n+ )\r\n+ )\r\n invalid_columns = set(row.keys()) - columns\r\n if invalid_columns:\r\n errors.append(\r\n@@ -1146,7 +1161,7 @@ class TableInsertView(BaseView):\r\n return _errors(errors)\r\n return rows, errors, extras\r\n \r\n- async def post(self, request):\r\n+ async def post(self, request, upsert=False):\r\n try:\r\n resolved = await self.ds.resolve_table(request)\r\n except NotFound as e:\r\n@@ -1164,7 +1179,12 @@ class TableInsertView(BaseView):\r\n request.actor, \"insert-row\", resource=(database_name, table_name)\r\n ):\r\n return _error([\"Permission denied\"], 403)\r\n- rows, errors, extras = await self._validate_data(request, db, table_name)\r\n+\r\n+ pks = await db.primary_keys(table_name)\r\n+\r\n+ rows, errors, extras = await self._validate_data(\r\n+ request, db, table_name, pks, upsert\r\n+ )\r\n if errors:\r\n return _error(errors, 400)\r\n \r\n@@ -1172,15 +1192,19 @@ class TableInsertView(BaseView):\r\n replace = extras.get(\"replace\")\r\n \r\n should_return = bool(extras.get(\"return\", False))\r\n- # Insert rows\r\n- def insert_rows(conn):\r\n+\r\n+ def insert_or_upsert_rows(conn):\r\n table = sqlite_utils.Database(conn)[table_name]\r\n+ kwargs = {}\r\n+ if upsert:\r\n+ kwargs[\"pk\"] = pks[0] if len(pks) == 1 else pks\r\n+ else:\r\n+ kwargs = {\"ignore\": ignore, \"replace\": replace}\r\n if should_return:\r\n rowids = []\r\n+ method = table.upsert if upsert else table.insert\r\n for row in rows:\r\n- rowids.append(\r\n- table.insert(row, ignore=ignore, replace=replace).last_rowid\r\n- )\r\n+ rowids.append(method(row, **kwargs).last_rowid)\r\n return list(\r\n table.rows_where(\r\n \"rowid in ({})\".format(\",\".join(\"?\" for _ in rowids)),\r\n@@ -1188,10 +1212,11 @@ class TableInsertView(BaseView):\r\n )\r\n )\r\n else:\r\n- table.insert_all(rows, ignore=ignore, replace=replace)\r\n+ method_all = table.upsert_all if upsert else table.insert_all\r\n+ method_all(rows, **kwargs)\r\n \r\n try:\r\n- rows = await db.execute_write_fn(insert_rows)\r\n+ rows = await db.execute_write_fn(insert_or_upsert_rows)\r\n except Exception as e:\r\n return _error([str(e)])\r\n result = {\"ok\": True}\r\n@@ -1200,6 +1225,13 @@ class TableInsertView(BaseView):\r\n return Response.json(result, status=201)\r\n \r\n \r\n+class TableUpsertView(TableInsertView):\r\n+ name = \"table-upsert\"\r\n+\r\n+ async def post(self, request):\r\n+ return await super().post(request, upsert=True)\r\n+\r\n+\r\n class TableDropView(BaseView):\r\n name = \"table-drop\"\r\n ```\r\nManual testing reveals that this mostly works... but it's not doing the right thing for `\"return\": true` - it always returns an empty list.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1432013704, "label": "/db/table/-/upsert API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1878#issuecomment-1336094470", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1878", "id": 1336094470, "node_id": "IC_kwDOBm6k_c5Poy8G", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-03T06:27:13Z", "updated_at": "2022-12-03T06:27:13Z", "author_association": "OWNER", "body": "Tests are going to need to cover both rowid-only and compound primary key tables, including all of the error states.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1432013704, "label": "/db/table/-/upsert API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1878#issuecomment-1336094562", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1878", "id": 1336094562, "node_id": "IC_kwDOBm6k_c5Poy9i", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-03T06:27:50Z", "updated_at": "2022-12-03T06:29:06Z", "author_association": "OWNER", "body": "This adds it to the API explorer:\r\n\r\n```diff\r\ndiff --git a/datasette/views/special.py b/datasette/views/special.py\r\nindex 1f84b094..1b4a9d3c 100644\r\n--- a/datasette/views/special.py\r\n+++ b/datasette/views/special.py\r\n@@ -316,21 +316,37 @@ class ApiExplorerView(BaseView):\r\n request.actor, \"insert-row\", (name, table)\r\n ):\r\n pks = await db.primary_keys(table)\r\n- table_links.append(\r\n- {\r\n- \"path\": self.ds.urls.table(name, table) + \"/-/insert\",\r\n- \"method\": \"POST\",\r\n- \"label\": \"Insert rows into {}\".format(table),\r\n- \"json\": {\r\n- \"rows\": [\r\n- {\r\n- column: None\r\n- for column in await db.table_columns(table)\r\n- if column not in pks\r\n- }\r\n- ]\r\n+ table_links.extend(\r\n+ [\r\n+ {\r\n+ \"path\": self.ds.urls.table(name, table) + \"/-/insert\",\r\n+ \"method\": \"POST\",\r\n+ \"label\": \"Insert rows into {}\".format(table),\r\n+ \"json\": {\r\n+ \"rows\": [\r\n+ {\r\n+ column: None\r\n+ for column in await db.table_columns(table)\r\n+ if column not in pks\r\n+ }\r\n+ ]\r\n+ },\r\n },\r\n- }\r\n+ {\r\n+ \"path\": self.ds.urls.table(name, table) + \"/-/upsert\",\r\n+ \"method\": \"POST\",\r\n+ \"label\": \"Upsert rows into {}\".format(table),\r\n+ \"json\": {\r\n+ \"rows\": [\r\n+ {\r\n+ column: None\r\n+ for column in await db.table_columns(table)\r\n+ if column not in pks\r\n+ }\r\n+ ]\r\n+ },\r\n+ },\r\n+ ]\r\n )\r\n if await self.ds.permission_allowed(\r\n request.actor, \"drop-table\", (name, table)\r\n```\r\nExcept it doesn't quite, because the example JSON this generates is invalid as it does not include the primary key column.\r\n\r\n(Made me notice that the way example columns are created for `/-/insert` will fail for tables that don't have an auto-incrementing primary key)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1432013704, "label": "/db/table/-/upsert API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1927#issuecomment-1336099368", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1927", "id": 1336099368, "node_id": "IC_kwDOBm6k_c5Po0Io", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-03T06:56:36Z", "updated_at": "2022-12-03T06:56:36Z", "author_association": "OWNER", "body": "Neither of these options make sense if you didn't pass a `\"pk\"`.\r\n\r\nMy initial implementation spotted if the `pk` was missing and looked it up from the table, but actually I don't think that makes sense - if you know the table exists and hence don't pass the `pk` you should be using `/-/insert` or `/-/upsert` instead.\r\n\r\nSo maybe this work should expanded to include validation that checks if the table exists already - and if it does, confirms that the primary key (and maybe even the columns) are the same as for that existing table.\r\n\r\nOf course if you only send `row` or `rows` then checking `columns` doesn't completely make sense - but we could check that the rows you have sent are equal to or a subset of the columns in the table. We could even check the column types as well, as seen in:\r\n- #1910 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1473411197, "label": "ignore:true/replace:true options for /db/-/create API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1927#issuecomment-1336099533", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1927", "id": 1336099533, "node_id": "IC_kwDOBm6k_c5Po0LN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-03T06:57:52Z", "updated_at": "2022-12-03T06:57:52Z", "author_association": "OWNER", "body": "I'm going to push what I have anyway. I'll keep this issue open while I think through the above comment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1473411197, "label": "ignore:true/replace:true options for /db/-/create API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1927#issuecomment-1336099588", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1927", "id": 1336099588, "node_id": "IC_kwDOBm6k_c5Po0ME", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-03T06:58:14Z", "updated_at": "2022-12-03T06:58:14Z", "author_association": "OWNER", "body": "I have not yet documented the new `insert` and `replace` options.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1473411197, "label": "ignore:true/replace:true options for /db/-/create API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1878#issuecomment-1336100218", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1878", "id": 1336100218, "node_id": "IC_kwDOBm6k_c5Po0V6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-03T07:02:15Z", "updated_at": "2022-12-03T07:02:15Z", "author_association": "OWNER", "body": "Moved this work to a PR:\r\n- #1931", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1432013704, "label": "/db/table/-/upsert API"}, "performed_via_github_app": null}