{"html_url": "https://github.com/simonw/sqlite-utils/issues/170#issuecomment-697047591", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/170", "id": 697047591, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzA0NzU5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T00:14:52Z", "updated_at": "2020-09-23T00:14:52Z", "author_association": "OWNER", "body": "\r\n@simonw\r\n@db.register_function decorator, closes #162\r\n4824775\r\n@simonw\r\ntable.transform() method - closes #114\r\n987dd12\r\n@simonw\r\nKeyword only arguments for transform()\r\nf8e10df\r\n\r\nAlso renamed columns= to types=\r\n\r\nCloses #165\r\n\r\nCommits on Sep 22, 2020\r\n@simonw\r\nImplemented sqlite-utils transform command, closes #164\r\n752d261\r\n@simonw\r\nApplied Black\r\nf29f682\r\n@simonw\r\ntable.extract() method, refs #42\r\nf855379\r\n@simonw\r\nDocstring for sqlite-utils transform\r\nc755f28\r\n@simonw\r\nAdded table.extract(rename=) option, refs #42\r\nc3210f2\r\n@simonw\r\nApplied Black\r\n317071a\r\n@simonw\r\nNew .rows_where(select=) argument\r\n7178231\r\n@simonw\r\ntable.extract() now works with rowid tables, refs #42\r\n2db6c5b\r\n@simonw\r\nsqlite-utils extract, closes #42\r\n55cf928\r\n@simonw\r\nProgress bar for \"sqlite-utils extract\", closes #169\r\n5c4d58d\r\n@simonw\r\nFixed PRAGMA foreign_keys handling for .transform, closes #167 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 706768798, "label": "Release notes for 2.20"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/970#issuecomment-697073465", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/970", "id": 697073465, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzA3MzQ2NQ==", "user": {"value": 2861690, "label": "secretGeek"}, "created_at": "2020-09-23T01:49:05Z", "updated_at": "2020-09-23T01:49:05Z", "author_association": "NONE", "body": "Oh wow oh wow. Thanks so much Simon. In an astoundingly rough week, this is a shining jewel. \ud83e\udd23", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705108492, "label": "request an \"-o\" option on \"datasette server\" to open the default browser at the running url"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697466497", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/172", "id": 697466497, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzQ2NjQ5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T14:41:17Z", "updated_at": "2020-09-23T14:41:17Z", "author_association": "OWNER", "body": "Steps to produce that database:\r\n```\r\ncurl -o salaries.csv 'https://data.sfgov.org/api/views/88g8-5mnd/rows.csv?accessType=DOWNLOAD'\r\nsqlite-utils insert salaries.db salaries salaries.csv --csv\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707427200, "label": "Improve performance of extract operations"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697467833", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/172", "id": 697467833, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzQ2NzgzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T14:42:03Z", "updated_at": "2020-09-23T14:42:03Z", "author_association": "OWNER", "body": "Here's the loop that's taking the time: https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L892-L897", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707427200, "label": "Improve performance of extract operations"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697473247", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/172", "id": 697473247, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzQ3MzI0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T14:45:13Z", "updated_at": "2020-09-23T14:45:13Z", "author_association": "OWNER", "body": "`lookup_table.lookup(lookups)` is doing a SQL lookup. This could be cached in-memory, maybe with a LRU cache, to avoid looking up the primary key for records that we have recently used.\r\n\r\nThe `.update()` method it is calling first does a `get()` and then does a SQL `UPDATE ... WHERE`:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L1244-L1264\r\n\r\nBatching those updates may have an effect. Or finding a way to skip the `.get()` since we already know we have a valid record.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707427200, "label": "Improve performance of extract operations"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/111#issuecomment-697545290", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/111", "id": 697545290, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzU0NTI5MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T15:29:11Z", "updated_at": "2020-09-23T15:29:11Z", "author_association": "OWNER", "body": "This is still a good idea.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 274615452, "label": "Add \u201cupdated\u201d to metadata"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/173#issuecomment-697577646", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/173", "id": 697577646, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzU3NzY0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T15:48:51Z", "updated_at": "2020-09-23T15:48:51Z", "author_association": "OWNER", "body": "This can only work when it's reading from a file, not when it's reading from standard input.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707478649, "label": "Progress bar for sqlite-utils insert"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697835956", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/172", "id": 697835956, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzgzNTk1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T18:22:49Z", "updated_at": "2020-09-23T18:22:49Z", "author_association": "OWNER", "body": "I ran `sudo py-spy top -p 123` against the process while it was running and the most time is definitely spent in `.update()`:\r\n```\r\nTotal Samples 1000\r\nGIL: 0.00%, Active: 90.00%, Threads: 1\r\n\r\n  %Own   %Total  OwnTime  TotalTime  Function (filename:line)                                                                                                                                  \r\n 38.00%  38.00%    3.85s     3.85s   update (sqlite_utils/db.py:1283)\r\n 27.00%  27.00%    2.12s     2.12s   execute (sqlite_utils/db.py:161)\r\n 10.00%  10.00%   0.890s    0.890s   execute (sqlite_utils/db.py:163)\r\n 10.00%  17.00%   0.870s     1.54s   columns (sqlite_utils/db.py:553)\r\n  0.00%   0.00%   0.110s    0.210s   <listcomp> (sqlite_utils/db.py:554)\r\n  0.00%   3.00%   0.100s    0.320s   table_names (sqlite_utils/db.py:191)\r\n  0.00%   0.00%   0.100s    0.100s   __new__ (<string>:1)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707427200, "label": "Improve performance of extract operations"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697859772", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/172", "id": 697859772, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Nzg1OTc3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T18:38:43Z", "updated_at": "2020-09-23T18:38:52Z", "author_association": "OWNER", "body": "I wonder if I could make this faster by separating it out into a few steps:\r\n- Create the new lookup table with all of the distinct rows\r\n- Add the blank foreign key column\r\n- run a `UPDATE table SET blah_id = (select id from lookup where thang = table.thang)`\r\n- Drop the value columns", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707427200, "label": "Improve performance of extract operations"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697863116", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/172", "id": 697863116, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Nzg2MzExNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T18:41:06Z", "updated_at": "2020-09-23T18:41:06Z", "author_association": "OWNER", "body": "Problem with this approach is it's not compatible with progress bars - but if it's a multiple of times faster it's worth it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707427200, "label": "Improve performance of extract operations"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697866885", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/172", "id": 697866885, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Nzg2Njg4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T18:43:37Z", "updated_at": "2020-09-23T18:43:37Z", "author_association": "OWNER", "body": "Also what would happen if the table had new rows added to it while that command was running?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707427200, "label": "Improve performance of extract operations"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697869886", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/172", "id": 697869886, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Nzg2OTg4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T18:45:30Z", "updated_at": "2020-09-23T18:45:30Z", "author_association": "OWNER", "body": "There's something to be said for making this operation pausable and resumable, especially if I'm going to make it available in a Datasette plugin at some point.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707427200, "label": "Improve performance of extract operations"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/619#issuecomment-697973420", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/619", "id": 697973420, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Nzk3MzQyMA==", "user": {"value": 45416, "label": "obra"}, "created_at": "2020-09-23T21:07:58Z", "updated_at": "2020-09-23T21:07:58Z", "author_association": "NONE", "body": "I've just run into this after crafting a complex query and discovered that hitting back loses my query.\r\n\r\nEven showing me the whole bad query would be a huge improvement over the current status quo.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520655983, "label": "\"Invalid SQL\" page should let you edit the SQL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/619#issuecomment-697980061", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/619", "id": 697980061, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Nzk4MDA2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T21:22:42Z", "updated_at": "2020-09-23T21:22:42Z", "author_association": "OWNER", "body": "Yeah that sucks. Bumping this up the priority list.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520655983, "label": "\"Invalid SQL\" page should let you edit the SQL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/619#issuecomment-697995303", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/619", "id": 697995303, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Nzk5NTMwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T22:01:08Z", "updated_at": "2020-09-23T22:01:08Z", "author_association": "OWNER", "body": "This is a little tricky to solve, because of the location of the form and the need to return JSON as well as HTML. It would be weird if a JSON request came in and got back the standard output from https://latest.datasette.io/fixtures.json when they were expecting to get back JSON in the shape of https://latest.datasette.io/fixtures.json?sql=select%20*%20from%20sqlite_master\r\n\r\nI'm going to return the HTML view that you would get for 0 results for a query - https://latest.datasette.io/fixtures?sql=select%201%20limit%200 - but with an error message.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520655983, "label": "\"Invalid SQL\" page should let you edit the SQL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/619#issuecomment-697995885", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/619", "id": 697995885, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Nzk5NTg4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T22:02:44Z", "updated_at": "2020-09-23T22:08:28Z", "author_association": "OWNER", "body": "So the JSON (still served with a 500 code) will look something like this:\r\n```json\r\n{\r\n  \"ok\": false,\r\n  \"status\": 500,\r\n  \"database\": \"fixtures\",\r\n  \"query_name\": null,\r\n  \"rows\": [],\r\n  \"truncated\": false,\r\n  \"error\": \"Error message goes here\",\r\n  \"columns\": [],\r\n  \"query\": {\r\n    \"sql\": \"the query that broke goes here\",\r\n    \"params\": {}\r\n  },\r\n  \"private\": false,\r\n  \"allow_execute_sql\": true,\r\n  \"query_ms\": 0.8716583251953125,\r\n  \"source\": \"tests/fixtures.py\",\r\n  \"source_url\": \"https://github.com/simonw/datasette/blob/master/tests/fixtures.py\",\r\n  \"license\": \"Apache License 2.0\",\r\n  \"license_url\": \"https://github.com/simonw/datasette/blob/master/LICENSE\"\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520655983, "label": "\"Invalid SQL\" page should let you edit the SQL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/619#issuecomment-697998045", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/619", "id": 697998045, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Nzk5ODA0NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T22:09:06Z", "updated_at": "2020-09-23T22:09:06Z", "author_association": "OWNER", "body": "I'll add this to the succesful JSON format:\r\n```json\r\n{\r\n  \"ok\": true,\r\n  \"error\": null\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520655983, "label": "\"Invalid SQL\" page should let you edit the SQL"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/619#issuecomment-698024773", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/619", "id": 698024773, "node_id": "MDEyOklzc3VlQ29tbWVudDY5ODAyNDc3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T23:31:46Z", "updated_at": "2020-09-23T23:31:46Z", "author_association": "OWNER", "body": "I'm going to have to untangle Datasette's error handling a bit for this - currently the expectation is that exceptions will be handled at a higher level, but I need to rethink that to make it cleaner for views like the \"execute custom SQL\" view to add their own error handling (and still be able to return the correct HTTP status codes, even with custom pages).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520655983, "label": "\"Invalid SQL\" page should let you edit the SQL"}, "performed_via_github_app": null}