{"html_url": "https://github.com/simonw/sqlite-utils/issues/507#issuecomment-1297859539", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/507", "id": 1297859539, "node_id": "IC_kwDOCGYnMM5NW8PT", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-11-01T00:40:16Z", "updated_at": "2022-11-01T00:40:16Z", "author_association": "CONTRIBUTOR", "body": "Ideally people could fix their data if they run into this issue.\r\n\r\nIf you are using filenames try [convmv](https://linux.die.net/man/1/convmv)\r\n\r\n```\r\nconvmv --preserve-mtimes -f utf8 -t utf8 --notest -i -r .\r\n```\r\n\r\nmaybe this script will also help: \r\n\r\n```py\r\nimport argparse, shutil\r\nfrom pathlib import Path\r\n\r\nimport ftfy\r\n\r\nfrom xklb import utils\r\nfrom xklb.utils import log\r\n\r\n\r\ndef parse_args() -> argparse.Namespace:\r\n    parser = argparse.ArgumentParser()\r\n    parser.add_argument(\"paths\", nargs='*')\r\n    parser.add_argument(\"--verbose\", \"-v\", action=\"count\", default=0)\r\n    args = parser.parse_args()\r\n\r\n    log.info(utils.dict_filter_bool(args.__dict__))\r\n    return args\r\n\r\n\r\ndef rename_invalid_paths() -> None:\r\n    args = parse_args()\r\n\r\n    for path in args.paths:\r\n        log.info(path)\r\n        for p in sorted([str(p) for p in Path(path).rglob(\"*\")], key=len):\r\n            fixed = ftfy.fix_text(p, uncurl_quotes=False).replace(\"\\r\\n\", \"\\n\").replace(\"\\r\", \"\\n\").replace(\"\\n\", \"\")\r\n            if p != fixed:\r\n                try:\r\n                    shutil.move(p, fixed)\r\n                except FileNotFoundError:\r\n                    log.warning(\"FileNotFound. %s\", p)\r\n                else:\r\n                    log.info(fixed)\r\n\r\n\r\nif __name__ == \"__main__\":\r\n    rename_invalid_paths()\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1430325103, "label": "conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1876#issuecomment-1298854321", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1876", "id": 1298854321, "node_id": "IC_kwDOBm6k_c5NavGx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T17:14:33Z", "updated_at": "2022-11-01T17:14:33Z", "author_association": "OWNER", "body": "I could use a `textarea` here (would need to figure out a neat pattern to expand it to fit the query):\r\n\r\n<img width=\"426\" alt=\"image\" src=\"https://user-images.githubusercontent.com/9599/199295041-25abe0b9-f825-43a2-ae5a-face622e08bc.png\">\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1431786951, "label": "SQL query should wrap on SQL interrupted screen"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1876#issuecomment-1298856054", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1876", "id": 1298856054, "node_id": "IC_kwDOBm6k_c5Navh2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T17:16:01Z", "updated_at": "2022-11-01T17:16:01Z", "author_association": "OWNER", "body": "`ta.style.height = ta.scrollHeight + 'px'` is an easy way to do that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1431786951, "label": "SQL query should wrap on SQL interrupted screen"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298877872", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/506", "id": 1298877872, "node_id": "IC_kwDOCGYnMM5Na02w", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T17:35:30Z", "updated_at": "2022-11-01T17:35:30Z", "author_association": "OWNER", "body": "This may not make sense.\r\n\r\nFirst, `.last_rowid` is a property on table - but that doesn't make sense for `rowcount` since it should clearly be a property on the database itself (you can run a query directly using `db.execute()` without going through a `Table` object).\r\n\r\nSo I tried this prototype:\r\n\r\n```diff\r\ndiff --git a/docs/python-api.rst b/docs/python-api.rst\r\nindex 206e5e6..78d3a8d 100644\r\n--- a/docs/python-api.rst\r\n+++ b/docs/python-api.rst\r\n@@ -186,6 +186,15 @@ The ``db.query(sql)`` function executes a SQL query and returns an iterator over\r\n     # {'name': 'Cleo'}\r\n     # {'name': 'Pancakes'}\r\n \r\n+After executing a query the ``db.rowcount`` property on that database instance will reflect the number of rows affected by any insert, update or delete operations performed by that query:\r\n+\r\n+.. code-block:: python\r\n+\r\n+    db = Database(memory=True)\r\n+    db[\"dogs\"].insert_all([{\"name\": \"Cleo\"}, {\"name\": \"Pancakes\"}])\r\n+    print(db.rowcount)\r\n+    # Outputs: 2\r\n+\r\n .. _python_api_execute:\r\n \r\n db.execute(sql, params)\r\ndiff --git a/sqlite_utils/db.py b/sqlite_utils/db.py\r\nindex a06f4b7..c19c2dd 100644\r\n--- a/sqlite_utils/db.py\r\n+++ b/sqlite_utils/db.py\r\n@@ -294,6 +294,8 @@ class Database:\r\n \r\n     _counts_table_name = \"_counts\"\r\n     use_counts_table = False\r\n+    # Number of rows inserted, updated or deleted\r\n+    rowcount: Optional[int] = None\r\n \r\n     def __init__(\r\n         self,\r\n@@ -480,9 +482,11 @@ class Database:\r\n         if self._tracer:\r\n             self._tracer(sql, parameters)\r\n         if parameters is not None:\r\n-            return self.conn.execute(sql, parameters)\r\n+            cursor = self.conn.execute(sql, parameters)\r\n         else:\r\n-            return self.conn.execute(sql)\r\n+            cursor = self.conn.execute(sql)\r\n+        self.rowcount = cursor.rowcount\r\n+        return cursor\r\n \r\n     def executescript(self, sql: str) -> sqlite3.Cursor:\r\n         \"\"\"\r\n```\r\nBut this happens:\r\n```pycon\r\n>>> from sqlite_utils import Database\r\n>>> db = Database(memory=True)\r\n>>> db[\"dogs\"].insert_all([{\"name\": \"Cleo\"}, {\"name\": \"Pancakes\"}])\r\n<Table dogs (name)>\r\n>>> db.rowcount\r\n-1\r\n```\r\nTurning on query tracing demonstrates why:\r\n```pycon\r\n>>> db = Database(memory=True, tracer=print)\r\nPRAGMA recursive_triggers=on; None\r\n>>> db[\"dogs\"].insert_all([{\"name\": \"Cleo\"}, {\"name\": \"Pancakes\"}])\r\nselect name from sqlite_master where type = 'view' None\r\nselect name from sqlite_master where type = 'table' None\r\nselect name from sqlite_master where type = 'view' None\r\nCREATE TABLE [dogs] (\r\n   [name] TEXT\r\n);\r\n         None\r\nselect name from sqlite_master where type = 'view' None\r\nINSERT INTO [dogs] ([name]) VALUES (?), (?); ['Cleo', 'Pancakes']\r\nselect name from sqlite_master where type = 'table' None\r\nselect name from sqlite_master where type = 'table' None\r\nPRAGMA table_info([dogs]) None\r\n<Table dogs (name)>\r\n>>>\r\n```\r\nThe `.insert_all()` function does a bunch of other queries too, so `.rowcount` is quickly over-ridden by the same result from extra queries that it executed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1429029604, "label": "Make `cursor.rowcount` accessible (wontfix)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298879701", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/506", "id": 1298879701, "node_id": "IC_kwDOCGYnMM5Na1TV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T17:37:13Z", "updated_at": "2022-11-01T17:37:13Z", "author_association": "OWNER", "body": "The question I was originally trying to answer here was this: how many rows were actually inserted by that call to `.insert_all()`?\r\n\r\nI don't know that `.rowcount` would ever be useful here, since the \"correct\" answer depends on other factors - had I determined to ignore or replace records with a primary key that matches an existing record for example?\r\n\r\nSo I think if people need `rowcount` they can get it by using a `cursor` directly.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1429029604, "label": "Make `cursor.rowcount` accessible (wontfix)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1873", "id": 1298885451, "node_id": "IC_kwDOBm6k_c5Na2tL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T17:42:20Z", "updated_at": "2022-11-01T17:42:20Z", "author_association": "OWNER", "body": "Design decision:\r\n```json\r\n{\r\n  \"rows\": [{\"id\": 1, \"title\": \"The title\"}],\r\n  \"ignore\": true\r\n}\r\n```\r\nOr `\"replace\": true`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1428630253, "label": "Ensure insert API has good tests for rowid and compound primark key tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1873#issuecomment-1298905135", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1873", "id": 1298905135, "node_id": "IC_kwDOBm6k_c5Na7gv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T17:59:59Z", "updated_at": "2022-11-01T17:59:59Z", "author_association": "OWNER", "body": "It's a bit surprising that you can send `\"ignore\": true, \"return_rows\": true` and the returned `\"inserted\"` key will list rows that were NOT inserted (since they were ignored).\r\n\r\nThree options:\r\n\r\n1. Ignore that and document it\r\n2. Fix it so `\"inserted\"` only returns rows that were actually inserted (bit tricky)\r\n3. Change the name of `\"inserted\"` to something else\r\n\r\nI'm picking 3 - I'm going to change it to be called `\"rows\"` instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1428630253, "label": "Ensure insert API has good tests for rowid and compound primark key tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1873#issuecomment-1298919552", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1873", "id": 1298919552, "node_id": "IC_kwDOBm6k_c5Na_CA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T18:11:27Z", "updated_at": "2022-11-01T18:11:27Z", "author_association": "OWNER", "body": "I forgot to document `ignore` and `replace`. Also I need to add tests that cover:\r\n\r\n- Forgetting to include a primary key on a non-autoincrement table\r\n- Compound primary keys\r\n- Rowid only tables with and without rowid specified\r\n\r\nI think my validation logic here will get caught out by the fact that `rowid` does not show up as a valid column name: https://github.com/simonw/datasette/blob/9bec7c38eb93cde5afb16df9bdd96aea2a5b0459/datasette/views/table.py#L1151-L1160\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1428630253, "label": "Ensure insert API has good tests for rowid and compound primark key tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1878#issuecomment-1299071456", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1878", "id": 1299071456, "node_id": "IC_kwDOBm6k_c5NbkHg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T20:02:43Z", "updated_at": "2022-11-01T20:02:43Z", "author_association": "OWNER", "body": "Note that \"update\" is partially covered by the `replace` option to `/-/insert`, added here:\r\n- https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1432013704, "label": "/db/table/-/upsert API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1862#issuecomment-1299073433", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1862", "id": 1299073433, "node_id": "IC_kwDOBm6k_c5NbkmZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T20:04:31Z", "updated_at": "2022-11-01T20:04:31Z", "author_association": "OWNER", "body": "It really feels like this should be accompanied by a `/db/-/create` API for creating tables. I had to add that to `sqlite-utils` eventually (initially it only supported creating by passing in an example document):\r\n\r\nhttps://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425011030, "label": "Create a new table from one or more records, `sqlite-utils` style"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1879#issuecomment-1299090678", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1879", "id": 1299090678, "node_id": "IC_kwDOBm6k_c5Nboz2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T20:20:28Z", "updated_at": "2022-11-01T20:20:28Z", "author_association": "OWNER", "body": "My first step in debugging these is to install https://datasette.io/plugins/datasette-debug-asgi - but now I'm thinking maybe something like that should be part of core.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1432037325, "label": "Make it easier to fix URL proxy problems"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1879#issuecomment-1299096850", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1879", "id": 1299096850, "node_id": "IC_kwDOBm6k_c5NbqUS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T20:26:12Z", "updated_at": "2022-11-01T20:26:12Z", "author_association": "OWNER", "body": "The other relevant plugin here is https://datasette.io/plugins/datasette-x-forwarded-host\r\n\r\nMaybe that should be rolled into core too?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1432037325, "label": "Make it easier to fix URL proxy problems"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1879#issuecomment-1299098458", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1879", "id": 1299098458, "node_id": "IC_kwDOBm6k_c5Nbqta", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T20:27:40Z", "updated_at": "2022-11-01T20:33:52Z", "author_association": "OWNER", "body": "https://github.com/simonw/datasette-x-forwarded-host/blob/main/datasette_x_forwarded_host/__init__.py could happen in core controlled by:\r\n\r\n`--setting trust_forwarded_host 1`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1432037325, "label": "Make it easier to fix URL proxy problems"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1879#issuecomment-1299102108", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1879", "id": 1299102108, "node_id": "IC_kwDOBm6k_c5Nbrmc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T20:30:54Z", "updated_at": "2022-11-01T20:33:06Z", "author_association": "OWNER", "body": "One idea: add a `/-/debug` page (or `/-/tips` or `/-/checks`) which shows the incoming requests headers and could even detect if there's an `x-forwarded-host` header that isn't being repeated and show a tip on how to fix that.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1432037325, "label": "Make it easier to fix URL proxy problems"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1879#issuecomment-1299102755", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1879", "id": 1299102755, "node_id": "IC_kwDOBm6k_c5Nbrwj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T20:31:37Z", "updated_at": "2022-11-01T20:31:37Z", "author_association": "OWNER", "body": "And some JavaScript that can spot if Datasette thinks it is being served over HTTP when it's actually being served over HTTPS.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1432037325, "label": "Make it easier to fix URL proxy problems"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1871#issuecomment-1299349741", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1871", "id": 1299349741, "node_id": "IC_kwDOBm6k_c5NcoDt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-01T23:22:55Z", "updated_at": "2022-11-01T23:22:55Z", "author_association": "OWNER", "body": "It's weird that the API explorer only lets you explore POST APIs. It should probably also let you explore GET APIs, or be renamed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1427293909, "label": "API explorer tool"}, "performed_via_github_app": null}