{"html_url": "https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1229206306", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/467", "id": 1229206306, "node_id": "IC_kwDOCGYnMM5JRDMi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T14:47:04Z", "updated_at": "2022-08-27T14:47:04Z", "author_association": "OWNER", "body": "I could add a `--transform` option to `sqlite-utils create-table` too.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "issue": {"value": 1348169997, "label": "Mechanism for ensuring a table has all the columns"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1229311612", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/467", "id": 1229311612, "node_id": "IC_kwDOCGYnMM5JRc58", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T23:19:41Z", "updated_at": "2022-08-27T23:19:41Z", "author_association": "OWNER", "body": "Documentation:\r\n\r\n- https://sqlite-utils.datasette.io/en/latest/python-api.html#explicitly-creating-a-table\r\n- https://sqlite-utils.datasette.io/en/latest/cli.html#creating-tables\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1348169997, "label": "Mechanism for ensuring a table has all the columns"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/469#issuecomment-1229120779", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/469", "id": 1229120779, "node_id": "IC_kwDOCGYnMM5JQuUL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:30:48Z", "updated_at": "2022-08-27T04:30:48Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352931464, "label": "sqlite-utils rows --order option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229115501", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/470", "id": 1229115501, "node_id": "IC_kwDOCGYnMM5JQtBt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T03:54:26Z", "updated_at": "2022-08-27T03:54:26Z", "author_association": "OWNER", "body": "I can borrow Alex's example extension for the tests too: https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/tests/ext.c", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932038, "label": "Upgrade `--load-extension` to accept entrypoints like Datasette"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229128617", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/470", "id": 1229128617, "node_id": "IC_kwDOCGYnMM5JQwOp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:39:06Z", "updated_at": "2022-08-27T05:39:06Z", "author_association": "OWNER", "body": "That didn't run the tests as expected on Ubuntu: https://github.com/simonw/sqlite-utils/runs/8048229213?check_suite_focus=true\r\n\r\n```\r\ntests/test_cli.py::test_load_extension[None-should_pass0-should_fail0] SKIPPED [ 19%]\r\ntests/test_cli.py::test_load_extension[sqlite3_ext_b_init-b-should_fail1] SKIPPED [ 19%]\r\ntests/test_cli.py::test_load_extension[sqlite3_ext_c_init-c-should_fail2] SKIPPED [ 19%]\r\n```\r\n\r\nThat should have worked because of this: https://github.com/simonw/sqlite-utils/blob/3f694e51a0212a297f663e2fc7f9cdad8775a550/.github/workflows/test.yml#L38-L41\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932038, "label": "Upgrade `--load-extension` to accept entrypoints like Datasette"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229128950", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/470", "id": 1229128950, "node_id": "IC_kwDOCGYnMM5JQwT2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:42:04Z", "updated_at": "2022-08-27T05:42:04Z", "author_association": "OWNER", "body": "Extension looks to be correctly compiled:\r\n\r\n```\r\nRun (cd tests && gcc ext.c -fPIC -shared -o ext.so && ls -lah)\r\ntotal 412K\r\ndrwxr-xr-x 3 runner docker 4.0K Aug 27 05:40 .\r\ndrwxr-xr-x 8 runner docker 4.0K Aug 27 05:40 ..\r\n-rw-r--r-- 1 runner docker 0 Aug 27 05:40 __init__.py\r\n-rw-r--r-- 1 runner docker 760 Aug 27 05:40 conftest.py\r\n-rw-r--r-- 1 runner docker 1.6K Aug 27 05:40 ext.c\r\n-rwxr-xr-x 1 runner docker 16K Aug 27 05:40 ext.so\r\ndrwxr-xr-x 2 runner docker 4.0K Aug 27 05:40 sniff\r\n```\r\nSo why doesn't this check pass?\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/3f694e51a0212a297f663e2fc7f9cdad8775a550/tests/test_cli.py#L25-L30", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932038, "label": "Upgrade `--load-extension` to accept entrypoints like Datasette"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229130158", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/470", "id": 1229130158, "node_id": "IC_kwDOCGYnMM5JQwmu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:52:46Z", "updated_at": "2022-08-27T05:52:46Z", "author_association": "OWNER", "body": "To run the tests locally on my Mac I needed to run Ubuntu in Docker (for the compilation step).\r\n\r\nThis worked:\r\n\r\n docker run -it -v \"$(pwd):/app\" ubuntu:latest /bin/bash\r\n\r\nThen:\r\n\r\n```\r\napt-get update && apt-get install python3-pip gcc libsqlite3-dev -y\r\ncd /app/tests\r\ngcc ext.c -fPIC -shared -o ext.so\r\ncd /app\r\npip3 install -e '.[test]'\r\npytest -k test_load_extension\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932038, "label": "Upgrade `--load-extension` to accept entrypoints like Datasette"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229130242", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/470", "id": 1229130242, "node_id": "IC_kwDOCGYnMM5JQwoC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:53:31Z", "updated_at": "2022-08-27T05:53:31Z", "author_association": "OWNER", "body": "Tests passed in CI against Ubuntu:\r\n```\r\ntests/test_cli.py::test_load_extension[None-should_pass0-should_fail0] PASSED [ 19%]\r\ntests/test_cli.py::test_load_extension[sqlite3_ext_b_init-b-should_fail1] PASSED [ 19%]\r\ntests/test_cli.py::test_load_extension[sqlite3_ext_c_init-c-should_fail2] PASSED [ 19%]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932038, "label": "Upgrade `--load-extension` to accept entrypoints like Datasette"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229130422", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/470", "id": 1229130422, "node_id": "IC_kwDOCGYnMM5JQwq2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:55:01Z", "updated_at": "2022-08-27T05:55:01Z", "author_association": "OWNER", "body": "cc @asg017 since this feature was inspired by his work on Datasette.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932038, "label": "Upgrade `--load-extension` to accept entrypoints like Datasette"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229116423", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229116423, "node_id": "IC_kwDOCGYnMM5JQtQH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:00:52Z", "updated_at": "2022-08-27T04:00:52Z", "author_association": "OWNER", "body": "Alternative design would be `--function name definition` - like this:\r\n\r\n```\r\nsqlite-utils data.db 'update images set domain = extract_domain(url)' --function extract_domain '\r\n from urllib.parse import urlparse\r\n return urlparse(url).netloc\r\n'\r\n```\r\nI like the `--functions` design better because it leaves space for import statements at the top of the code block, and allows more than one function to be defined in a single go.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229118619", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229118619, "node_id": "IC_kwDOCGYnMM5JQtyb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:14:52Z", "updated_at": "2022-08-27T04:14:52Z", "author_association": "OWNER", "body": "Quick prototype:\r\n```diff\r\ndiff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py\r\nindex 43e76fa..5dee4f6 100644\r\n--- a/sqlite_utils/cli.py\r\n+++ b/sqlite_utils/cli.py\r\n@@ -1633,6 +1633,9 @@ def drop_view(path, view, ignore, load_extension):\r\n type=(str, str),\r\n help=\"Named :parameters for SQL query\",\r\n )\r\n+@click.option(\r\n+ \"--functions\", help=\"Python code defining one or more custom SQL functions\"\r\n+)\r\n @load_extension_option\r\n def query(\r\n path,\r\n@@ -1649,6 +1652,7 @@ def query(\r\n raw,\r\n param,\r\n load_extension,\r\n+ functions,\r\n ):\r\n \"\"\"Execute SQL query and return the results as JSON\r\n \r\n@@ -1665,6 +1669,16 @@ def query(\r\n _load_extensions(db, load_extension)\r\n db.register_fts4_bm25()\r\n \r\n+ # Register any Python functions as SQL functions:\r\n+ if functions:\r\n+ locals = {}\r\n+ globals = {}\r\n+ exec(functions, globals, locals)\r\n+ # Register all callables in the locals dict:\r\n+ for name, value in locals.items():\r\n+ if callable(value):\r\n+ db.register_function(value, name=name)\r\n+\r\n _execute_query(\r\n db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols\r\n )\r\n```\r\nDemo:\r\n```bash\r\n % sqlite-utils :memory: 'select 1 + dog()' --functions '\r\nquote> def dog():\r\nquote> return 2\r\nquote> '\r\n[{\"1 + dog()\": 3}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229119171", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229119171, "node_id": "IC_kwDOCGYnMM5JQt7D", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:18:28Z", "updated_at": "2022-08-27T04:18:28Z", "author_association": "OWNER", "body": "I tried this:\r\n```\r\nsqlite-utils :memory: 'select extract_domain(\"https://www.google.com/blah\")' --functions '\r\nfrom urllib.parse import urlparse\r\n\r\ndef extract_domain(url):\r\n from urllib.parse import urlparse\r\n return urlparse(url).netloc\r\n'\r\n```\r\nAnd got:\r\n```\r\nNameError: name 'urlparse' is not defined\r\nError: user-defined function raised exception\r\n```\r\nBut this worked OK:\r\n```\r\n% sqlite-utils :memory: 'select extract_domain(\"https://www.google.com/blah\")' --functions '\r\ndef extract_domain(url):\r\n from urllib.parse import urlparse\r\n return urlparse(url).netloc\r\n'\r\n[{\"extract_domain(\\\"https://www.google.com/blah\\\")\": \"www.google.com\"}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229119999", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229119999, "node_id": "IC_kwDOCGYnMM5JQuH_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:24:58Z", "updated_at": "2022-08-27T04:24:58Z", "author_association": "OWNER", "body": "I've encountered this problem before: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-convert-complex\r\n\r\n> ```\r\n> $ sqlite-utils convert content.db articles score '\r\n> import random\r\n> random.seed(10)\r\n> \r\n> def convert(value):\r\n> global random\r\n> return random.random()\r\n> '\r\n> ```\r\n> Note the `global random` line here. Due to the way the tool compiles Python code, this is necessary to ensure the `random` module is available within the `convert()` function. If you were to omit this you would see a `NameError: name 'random' is not defined` error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120104", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229120104, "node_id": "IC_kwDOCGYnMM5JQuJo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:25:39Z", "updated_at": "2022-08-27T04:25:39Z", "author_association": "OWNER", "body": "This works:\r\n```\r\nsqlite-utils :memory: 'select extract_domain(\"https://www.google.com/blah\")' --functions '\r\nfrom urllib.parse import urlparse\r\n\r\ndef extract_domain(url):\r\n global urlparse\r\n return urlparse(url).netloc\r\n'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120653", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229120653, "node_id": "IC_kwDOCGYnMM5JQuSN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:29:49Z", "updated_at": "2022-08-27T04:30:03Z", "author_association": "OWNER", "body": "Found a fix for that!\r\n\r\nI replaced this:\r\n\r\n```python\r\n locals = {}\r\n globals = {}\r\n exec(functions, globals, locals)\r\n # Register all callables in the locals dict:\r\n for name, value in locals.items():\r\n if callable(value):\r\n db.register_function(value, name=name)\r\n```\r\nWith this:\r\n```python\r\n globals = {}\r\n exec(functions, globals)\r\n # Register all callables in the globals dict:\r\n for name, value in globals.items():\r\n if callable(value):\r\n db.register_function(value, name=name)\r\n```\r\nBecause https://docs.python.org/3/library/functions.html#exec says:\r\n\r\n> If only *globals* is provided, it must be a dictionary (and not a subclass of dictionary), which will be used for both the global and the local variables. If *globals* and *locals* are given, they are used for the global and local variables, respectively.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120899", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229120899, "node_id": "IC_kwDOCGYnMM5JQuWD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:31:35Z", "updated_at": "2022-08-27T04:32:38Z", "author_association": "OWNER", "body": "I should add this `--functions` feature to the `memory` and `bulk` commands too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229124379", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229124379, "node_id": "IC_kwDOCGYnMM5JQvMb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:02:21Z", "updated_at": "2022-08-27T05:02:21Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#defining-custom-sql-functions", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229124549", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229124549, "node_id": "IC_kwDOCGYnMM5JQvPF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:03:39Z", "updated_at": "2022-08-27T05:03:39Z", "author_association": "OWNER", "body": "I don't think I need separate documentation for `sqlite-utils memory` and `sqlite-tils bulk` since they work the same, and the `--help` text provides the necessary hints.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229125114", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229125114, "node_id": "IC_kwDOCGYnMM5JQvX6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:08:58Z", "updated_at": "2022-08-27T05:08:58Z", "author_association": "OWNER", "body": "Testing `bulk --functions`:\r\n\r\n```\r\n% sqlite-utils create-table chickens.db chickens id integer name text name_upper text \r\n% echo 'id,name\r\n1,Blue\r\n2,Snowy\r\n3,Azi\r\n4,Lila\r\n5,Suna\r\n6,Cardi' | sqlite-utils bulk chickens.db '\r\ninsert into chickens (id, name, name_upper) values (:id, :name, myupper(:name))\r\n' - --functions '\r\ndef myupper(s):\r\n return s.upper()\r\n' --csv\r\n% sqlite-utils rows chickens.db chickens\r\n[{\"id\": 1, \"name\": \"Blue\", \"name_upper\": \"BLUE\"},\r\n {\"id\": 2, \"name\": \"Snowy\", \"name_upper\": \"SNOWY\"},\r\n {\"id\": 3, \"name\": \"Azi\", \"name_upper\": \"AZI\"},\r\n {\"id\": 4, \"name\": \"Lila\", \"name_upper\": \"LILA\"},\r\n {\"id\": 5, \"name\": \"Suna\", \"name_upper\": \"SUNA\"},\r\n {\"id\": 6, \"name\": \"Cardi\", \"name_upper\": \"CARDI\"}]\r\n```", "reactions": "{\"total_count\": 2, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/472#issuecomment-1229125614", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/472", "id": 1229125614, "node_id": "IC_kwDOCGYnMM5JQvfu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:12:59Z", "updated_at": "2022-08-27T05:12:59Z", "author_association": "OWNER", "body": "Annoyingly this means the `--import` option is no longer required, but removing it would be a backwards incompatible breakage so I need to leave it in. I can minimize it in the documentation though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352946135, "label": "Reuse the locals/globals fix from --functions for other code accepting options"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/472#issuecomment-1229125890", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/472", "id": 1229125890, "node_id": "IC_kwDOCGYnMM5JQvkC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:15:26Z", "updated_at": "2022-08-27T05:15:26Z", "author_association": "OWNER", "body": "Here's the docs I get to update to remove the documented `global` hack: https://sqlite-utils.datasette.io/en/3.28/cli.html#cli-convert-complex", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352946135, "label": "Reuse the locals/globals fix from --functions for other code accepting options"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/474#issuecomment-1229265285", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/474", "id": 1229265285, "node_id": "IC_kwDOCGYnMM5JRRmF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T20:52:53Z", "updated_at": "2022-08-27T20:52:53Z", "author_association": "OWNER", "body": "A couple of tricks I use here. Firstly, I often create the table before the import using the `sqlite-utils create-table` command: https://sqlite-utils.datasette.io/en/stable/cli.html#creating-tables\r\n\r\nThe other current option is to use the `bulk` command, which lets you construct a custom SQL query to execute against every row from a CSV file: https://sqlite-utils.datasette.io/en/stable/cli.html#executing-sql-in-bulk\r\n\r\nDo either of those options work here or is there a useful new feature that would work better?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1353074021, "label": "Add an option for specifying column names when inserting CSV data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/475#issuecomment-1229283063", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/475", "id": 1229283063, "node_id": "IC_kwDOCGYnMM5JRV73", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T22:44:45Z", "updated_at": "2022-08-27T22:44:45Z", "author_association": "OWNER", "body": "Documentation:\r\n- https://sqlite-utils.datasette.io/en/latest/python-api.html#default-values\r\n- https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.default_values", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1353189941, "label": "table.default_values introspection property"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/476#issuecomment-1229320114", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/476", "id": 1229320114, "node_id": "IC_kwDOCGYnMM5JRe-y", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T23:26:48Z", "updated_at": "2022-08-27T23:26:48Z", "author_association": "OWNER", "body": "- #458 - the `register_function(name=...)` argument\r\n- New tutorial: [Cleaning data with sqlite-utils and Datasette](https://datasette.io/tutorials/clean-data) provides a tutorial introduction (and accompanying ten minute video) about using this tool.\r\n- New Discord community, https://discord.gg/Ass7bCAMDw\r\n- #469 `sqlite-utils rows --order` option\r\n- #471 `sqlite-utils query --functions` option\r\n- #472 Improved code compilation pattern\r\n- #473 Support entrypoints for `--load-extension`\r\n- #455\r\n- #475 `table.default_values` property\r\n- #467", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1353196970, "label": "Release notes for 3.29"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/455#issuecomment-1229205990", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/455", "id": 1229205990, "node_id": "IC_kwDOCGYnMM5JRDHm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T14:44:45Z", "updated_at": "2022-08-27T14:44:45Z", "author_association": "OWNER", "body": "I did not know `is` worked against strings and integers and booleans!\r\n\r\nhttps://latest.datasette.io/_memory?sql=select+1+is+1%2C+2.3+is+2.3%2C+true+is+true%2C+true+is+false%2C+%27dog%27+is+%27dog%27%2C+%27dog%27+is+%27cat%27%2C+null+is+null%2C+null+is+not+null\r\n\r\n```sql\r\nselect 1 is 1, 2.3 is 2.3, true is true, true is false, 'dog' is 'dog', 'dog' is 'cat', null is null, null is not null\r\n```\r\n\r\n| 1 is 1 | 2.3 is 2.3 | true is true | true is false | 'dog' is 'dog' | 'dog' is 'cat' | null is null | null is not null |\r\n|----------|--------------|----------------|-----------------|------------------|------------------|----------------|--------------------|\r\n| 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1309542173, "label": "in extract code, check equality with IS instead of = for nulls"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1224412060", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468", "id": 1224412060, "node_id": "IC_kwDOCGYnMM5I-wuc", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-08-23T17:30:24Z", "updated_at": "2022-08-27T23:11:44Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/468?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#468](https://codecov.io/gh/simonw/sqlite-utils/pull/468?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fc38480) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/c5f8a2eb1a81a18b52825cc649112f71fe419b12?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c5f8a2e) will **decrease** coverage by `0.12%`.\n> The diff coverage is `88.88%`.\n\n> :exclamation: Current head fc38480 differs from pull request most recent head 2f6a64f. Consider uploading reports for the commit 2f6a64f to get more accurate results\n\n```diff\n@@ Coverage Diff @@\n## main #468 +/- ##\n==========================================\n- Coverage 96.60% 96.47% -0.13% \n==========================================\n Files 6 6 \n Lines 2590 2640 +50 \n==========================================\n+ Hits 2502 2547 +45 \n- Misses 88 93 +5 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/468?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/468/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.86% <66.66%> (-0.09%)` | :arrow_down: |\n| [sqlite\\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/468/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.31% <90.19%> (-0.23%)` | :arrow_down: |\n\nHelp us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1348294436, "label": "db[table].create(..., transform=True) and create-table --transform"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229207725", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468", "id": 1229207725, "node_id": "IC_kwDOCGYnMM5JRDit", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T14:55:32Z", "updated_at": "2022-08-27T22:03:50Z", "author_association": "OWNER", "body": "The main challenge here is coming up with comprehensive tests. The cases I need to cover are from this block of code:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/c5f8a2eb1a81a18b52825cc649112f71fe419b12/sqlite_utils/db.py#L1468-L1480", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1348294436, "label": "db[table].create(..., transform=True) and create-table --transform"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229276554", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468", "id": 1229276554, "node_id": "IC_kwDOCGYnMM5JRUWK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T22:05:40Z", "updated_at": "2022-08-27T22:05:51Z", "author_association": "OWNER", "body": "For the moment I'm not going to pay attention to `foreign_keys` changes - I will note that these are not modified in the documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1348294436, "label": "db[table].create(..., transform=True) and create-table --transform"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229279539", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468", "id": 1229279539, "node_id": "IC_kwDOCGYnMM5JRVEz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T22:24:33Z", "updated_at": "2022-08-27T22:24:33Z", "author_association": "OWNER", "body": "Interesting challenge with `default_value`: I need to be able to tell if the default values passed to `.create()` differ from those in the database already.\r\n\r\nIntrospecting that is a bit tricky:\r\n\r\n```pycon\r\n>>> import sqlite_utils\r\n>>> db = sqlite_utils.Database(memory=True)\r\n>>> db[\"blah\"].create({\"id\": int, \"name\": str}, not_null=(\"name\",), defaults={\"name\": \"bob\"})\r\n\r\n>>> db[\"blah\"].columns\r\n[Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value=\"'bob'\", is_pk=0)]\r\n```\r\nNote how a default value of the Python string `bob` is represented in the results of `PRAGMA table_info()` as `default_value=\"'bob'\"` - it's got single quotes added to it!\r\n\r\nSo comparing default values from introspecting the database needs me to first parse that syntax. This may require a new table introspection method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1348294436, "label": "db[table].create(..., transform=True) and create-table --transform"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229284539", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468", "id": 1229284539, "node_id": "IC_kwDOCGYnMM5JRWS7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T22:54:07Z", "updated_at": "2022-08-27T22:55:01Z", "author_association": "OWNER", "body": "Just needs documentation now, which can go here: https://sqlite-utils--468.org.readthedocs.build/en/468/python-api.html#explicitly-creating-a-table", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1348294436, "label": "db[table].create(..., transform=True) and create-table --transform"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229303691", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468", "id": 1229303691, "node_id": "IC_kwDOCGYnMM5JRa-L", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T23:13:06Z", "updated_at": "2022-08-27T23:13:06Z", "author_association": "OWNER", "body": "Documentation:\r\n- https://sqlite-utils--468.org.readthedocs.build/en/468/python-api.html#explicitly-creating-a-table\r\n- https://sqlite-utils--468.org.readthedocs.build/en/468/cli.html#creating-tables", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1348294436, "label": "db[table].create(..., transform=True) and create-table --transform"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/473#issuecomment-1229130453", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/473", "id": 1229130453, "node_id": "IC_kwDOCGYnMM5JQwrV", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-08-27T05:55:12Z", "updated_at": "2022-08-27T05:55:18Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/473?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#473](https://codecov.io/gh/simonw/sqlite-utils/pull/473?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (72db599) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a46a5e3a9e03dcdd8c84a92e4a5dbfa02ba461fa?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a46a5e3) will **decrease** coverage by `0.07%`.\n> The diff coverage is `50.00%`.\n\n```diff\n@@ Coverage Diff @@\n## main #473 +/- ##\n==========================================\n- Coverage 96.67% 96.60% -0.08% \n==========================================\n Files 6 6 \n Lines 2587 2590 +3 \n==========================================\n+ Hits 2501 2502 +1 \n- Misses 86 88 +2 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/473?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/473/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.94% <50.00%> (-0.18%)` | :arrow_down: |\n\nHelp us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352953535, "label": "Support entrypoints for `--load-extension`"}, "performed_via_github_app": null}