{"html_url": "https://github.com/simonw/datasette/issues/1878#issuecomment-1312534826", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1878", "id": 1312534826, "node_id": "IC_kwDOBm6k_c5OO7Eq", "user": {"value": 18738650, "label": "stevecrawshaw"}, "created_at": "2022-11-12T17:34:58Z", "updated_at": "2022-11-12T17:34:58Z", "author_association": "NONE", "body": "Hi Simon. I have just started experimenting with datasette in earnest, looking at it's suitability for air quality open data. A bulk upsert \\ upsert_all would be very useful for me in enabling real time data to be pushed from a sql server database with FME server to a datasette db. \r\n\r\nAn hourly process queries the last 2 hours of data and pushes that to my database, inserting new data and updating existing combinations of pk siteid and date_time. This is already implemented on our current [open data portal](https://opendata.bristol.gov.uk/explore/dataset/air-quality-data-continuous/table/?disjunctive.location&sort=date_time). Excited to see your progress with this! Thank you for this amazing software.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1432013704, "label": "/db/table/-/upsert API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1312556044", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1312556044, "node_id": "IC_kwDOBm6k_c5OPAQM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-12T19:29:11Z", "updated_at": "2022-11-12T19:29:11Z", "author_association": "OWNER", "body": "Thought of an edge-case: with `sqlite-utils` one feature I really like is that I can pipe data into it without caring if the table already exists or not:\r\n\r\n cat data.json | sqlite-utils insert my.db mytable -\r\n\r\nHow could this new API support that?\r\n\r\nI thought about adding a `\"create\": true` option to `/db/table/-/insert` which creates the table if it doesn't already exist, but if I do that I'll need to start adding other options to that endpoint - to set the primary key, add foreign keys and suchlike - which would be ignored except for the cases where the table was being created from scratch.\r\n\r\nThis doesn't feel right to me - I want to keep those options here, on `/db/-/create`.\r\n\r\nOne idea I had was to implement it such that you can call `/db/-/create` multiple times for the same table, but only if you are using the `\"rows\"` option. If so, and if the rows can be safely inserted, it would let you do that.\r\n\r\nBut instead, I'm going to outsource this to the CLI tool I plan to write that feeds data into this API. I'm already planning to use that tool for CSV inserts (so the API doesn't need to accept CSV directly). I think it's a good place for other usability enhancements like \"insert this, creating the table if it does not exist\" as well.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1312575048", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1312575048, "node_id": "IC_kwDOBm6k_c5OPE5I", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-12T21:22:58Z", "updated_at": "2022-11-12T21:22:58Z", "author_association": "OWNER", "body": "Need to validate the table name. SQLite supports almost any table name - but they can't contain a newline character and cannot start with `sqlite_` - according to https://stackoverflow.com/questions/3694276/what-are-valid-table-names-in-sqlite/43049720#43049720", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1312580348", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1312580348, "node_id": "IC_kwDOBm6k_c5OPGL8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-12T21:55:54Z", "updated_at": "2022-11-12T21:56:45Z", "author_association": "OWNER", "body": "What should this API return?\r\n\r\nI think the name of the table (`name`), the URL to that table (`table_url` - for consistency with how faceting API works already) and the schema of the table (`schema`).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1312581008", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1312581008, "node_id": "IC_kwDOBm6k_c5OPGWQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-12T22:00:52Z", "updated_at": "2022-11-12T22:00:52Z", "author_association": "OWNER", "body": "Tried out my prototype in the API explorer:\r\n\r\n\"image\"\r\n\r\nThe `\"name\"` on the output is bothering me a bit - should it be `table_name` or `table` instead?\r\n\r\nProblem is I really like `name` for the input, so should it be consistent to have the same name on the output here, or should I aim for consistency with other endpoints that might return `table` rather than the ambiguous `name` elsewhere?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1312581121", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1312581121, "node_id": "IC_kwDOBm6k_c5OPGYB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-12T22:01:32Z", "updated_at": "2022-11-12T22:01:32Z", "author_association": "OWNER", "body": "I'm going to change it to `table` in the output AND the input.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1312582512", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1312582512, "node_id": "IC_kwDOBm6k_c5OPGtw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-12T22:11:18Z", "updated_at": "2022-11-12T22:11:18Z", "author_association": "OWNER", "body": "I like this:\r\n\r\n\"image\"\r\n\r\n```json\r\n{\r\n \"ok\": true,\r\n \"database\": \"data\",\r\n \"table\": \"agai2n\",\r\n \"table_url\": \"http://127.0.0.1:8001/data/agai2n\",\r\n \"schema\": \"CREATE TABLE [agai2n] (\\n [hello] INTEGER\\n)\"\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null}