{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513262013", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 513262013, "node_id": "MDEyOklzc3VlQ29tbWVudDUxMzI2MjAxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-19T14:58:23Z", "updated_at": "2020-09-22T18:12:11Z", "author_association": "OWNER", "body": "CLI design idea:\r\n\r\n    $ sqlite-utils extract my.db \\\r\n        dea_sales company_name\r\n\r\nHere we just specify the original table and column - the new extracted table will automatically be called \"company_name\" and will have \"id\" and \"value\" columns, by default.\r\n\r\nTo set a custom extract table:\r\n\r\n    $ sqlite-utils extract my.db \\\r\n        dea_sales company_name \\\r\n        --table companies\r\n\r\nAnd for extracting multiple columns and renaming them on the created table, maybe something like this:\r\n\r\n    $ sqlite-utils extract my.db \\\r\n        dea_sales company_name company_address \\\r\n        --table companies \\\r\n        --column company_name name \\\r\n        --column company_address address\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-693009048", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 693009048, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MzAwOTA0OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T22:17:30Z", "updated_at": "2020-09-22T14:37:00Z", "author_association": "OWNER", "body": "Maybe instead of implementing `datasette.get()` and `datasette.post()` and `datasette.request()` and `datasette.stream()` I could instead have a nested object called `datasette.client` which is a preconfigured `AsyncClient` instance.\r\n\r\n```python\r\nresponse = await datasette.client.get(\"/\")\r\n```\r\nOr perhaps this should be a method in case I ever need to be able to `await` it:\r\n```python\r\nresponse = await (await datasette.client()).get(\"/\")\r\n```\r\nThis is a bit cosmetically ugly though, I'd rather avoid that if possible.\r\n\r\nMaybe I could get this working by returning an object from `.client()` which provides a `await obj.get()` method:\r\n```python\r\nresponse = await datasette.client().get(\"/\")\r\n```\r\nI don't think there's any benefit to that over `await datasette.client.get()` though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696442621", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696442621, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ0MjYyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T00:00:23Z", "updated_at": "2020-09-22T00:00:23Z", "author_association": "OWNER", "body": "I still need to figure out what to do about these various other table properties: https://github.com/simonw/sqlite-utils/blob/b34c9b40c206d7a9d7ee57a8c1f198ff1f522735/sqlite_utils/db.py#L775-L787", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443042", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696443042, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ0MzA0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T00:01:50Z", "updated_at": "2020-09-22T00:01:50Z", "author_association": "OWNER", "body": "When you transform a table, it should keep its primary key, foreign keys, not_null and defaults. I don't think it needs to care about `hash_id` or `extracts=` since those don't affect the structure of the table as it is being created - well, `hash_id` does but if we are transforming an existing table we will get the `hash_id` column for free.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443190", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696443190, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ0MzE5MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T00:02:22Z", "updated_at": "2020-09-22T00:02:22Z", "author_association": "OWNER", "body": "How would I detect which columns are `not_null` and what their defaults are? I don`t think my introspection logic handles that yet.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443845", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696443845, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ0Mzg0NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T00:04:31Z", "updated_at": "2020-09-22T00:04:44Z", "author_association": "OWNER", "body": "Good news: the `.columns` introspection does tell me those things:\r\n```\r\n>>> import sqlite_utils\r\n>>> db = sqlite_utils.Database(memory=True)\r\n>>> db.create_table(\"foo\", {\"id\": int, \"name\": str, \"age\": int}, defaults={\"age\": 1}, not_null={\"name\", \"age\"})\r\n<Table foo (id, name, age)>\r\n>>> db[\"foo\"]\r\n<Table foo (id, name, age)>\r\n>>> print(db[\"foo\"].schema)\r\nCREATE TABLE [foo] (\r\n   [id] INTEGER,\r\n   [name] TEXT NOT NULL,\r\n   [age] INTEGER NOT NULL DEFAULT 1\r\n)\r\n>>> db[\"foo\"].columns\r\n[Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0),\r\n Column(cid=1, name='name', type='TEXT', notnull=1, default_value=None, is_pk=0),\r\n Column(cid=2, name='age', type='INTEGER', notnull=1, default_value='1', is_pk=0)]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444353", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696444353, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ0NDM1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T00:06:12Z", "updated_at": "2020-09-22T00:06:12Z", "author_association": "OWNER", "body": "I should support `not_null=` and `default=` arguments to the `.transform()` method because it looks like you can't use `ALTER TABLE` to change those.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444842", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696444842, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ0NDg0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T00:07:43Z", "updated_at": "2020-09-22T00:09:05Z", "author_association": "OWNER", "body": "Syntax challenge: I could use `.transform(defaults={\"age\": None})` to indicate that the `age` column should have its default removed, but how would I tell `.transform()` that the `age` column, currently `not null`, should have the `not null` removed from it?\r\n\r\nI could do this: `.transform(not_not_null={\"age\"})` - it's a bit gross but it's also kind of funny. I actually like it!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696445766", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696445766, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ0NTc2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T00:10:50Z", "updated_at": "2020-09-22T00:11:12Z", "author_association": "OWNER", "body": "A less horrible interface might be the following:\r\n```python\r\n# Ensure the 'age' column is not null:\r\ntable.transform(not_null={\"age\"})\r\n# The 'age' column is not null but I don't want it to be:\r\ntable.transform(not_null={\"age\": False})\r\n```\r\nSo if the argument is a set it means \"make sure these are all not null\" - if the argument is a dictionary it means \"set these to be null or not null depending on if their dictionary value is true or false\".", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696446658", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696446658, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ0NjY1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T00:13:55Z", "updated_at": "2020-09-22T00:14:21Z", "author_association": "OWNER", "body": "Idea: allow a `conversions=` parameter, as seen on `.insert_all()` and friends, which lets you apply a SQL transformation function as part of the operation. E.g.:\r\n\r\n```python\r\ntable.transform({\"age\": int}, conversions={\"name\": \"upper(?)\"})\r\n```\r\n\r\nhttps://sqlite-utils.readthedocs.io/en/stable/python-api.html#converting-column-values-using-sql-functions", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696449345", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/162", "id": 696449345, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ0OTM0NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T00:22:46Z", "updated_at": "2020-09-22T00:22:46Z", "author_association": "OWNER", "body": "Inspired by the idea of adding `conversions=` to #114 - since this would make it easy to register custom Python functions that can be used to convert the values in a table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705995722, "label": "A decorator for registering custom SQL functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696454084", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/162", "id": 696454084, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ1NDA4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T00:40:44Z", "updated_at": "2020-09-22T00:40:44Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#registering-custom-sql-functions", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705995722, "label": "A decorator for registering custom SQL functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696454485", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114", "id": 696454485, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ1NDQ4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T00:42:35Z", "updated_at": "2020-09-22T00:42:35Z", "author_association": "OWNER", "body": "The reason I'm working on this now is that I'd like to support many more options for data cleanup in the Datasette ecosystem - so being able to do things like convert the type of existing columns becomes increasingly important.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621989740, "label": "table.transform() method for advanced alter table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/163#issuecomment-696465788", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/163", "id": 696465788, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ2NTc4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T01:33:04Z", "updated_at": "2020-09-22T01:33:04Z", "author_association": "OWNER", "body": "This would apply to `.transform()` in #114 too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 706001517, "label": "Idea: conversions= could take Python functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696473559", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/164", "id": 696473559, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ3MzU1OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T02:10:37Z", "updated_at": "2020-09-22T02:10:37Z", "author_association": "OWNER", "body": "Maybe something like this:\r\n\r\n    sqlite-utils transform mydb.db mytable -c age integer --rename age dog_age\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 706017416, "label": "sqlite-utils transform sub-command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696480925", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696480925, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ4MDkyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T02:45:47Z", "updated_at": "2020-09-22T02:45:47Z", "author_association": "OWNER", "body": "I'm not going to do `conversions=` because it would be inconsistent with how they work elsewhere. The SQL generated by this function looks like this:\r\n\r\n    INSERT INTO dogs_new_tmp VALUES (a, b) SELECT a, b from dogs;\r\n\r\nSo passing `conversions={\"name\": \"upper(?)\"})` wouldn't make sense, since we're not using arguments hence there is no-where for that `?` to go.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696485791", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696485791, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ4NTc5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T03:10:15Z", "updated_at": "2020-09-22T03:10:15Z", "author_association": "OWNER", "body": "Design decision needed on foreign keys: what does the syntax look like for removing an existing foreign key?\r\n\r\nSince I already have a good implementation of `add_foreign_key()` I'm tempted to only support dropping them. Maybe like this:\r\n\r\n```python\r\ntable.transform(drop_foreign_keys=[(\"author_id\", \"author\", \"id\")])\r\n```\r\nIt's a bit crufty but it's such a rare use-case that I think this will be good enough.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696488201", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696488201, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ4ODIwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T03:21:16Z", "updated_at": "2020-09-22T03:21:16Z", "author_association": "OWNER", "body": "Just needs documentation now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696490851", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696490851, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ5MDg1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T03:33:54Z", "updated_at": "2020-09-22T03:33:54Z", "author_association": "OWNER", "body": "It would be neat if `.transform(pk=None)` converted a primary key table to a rowid table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696494070", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161", "id": 696494070, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjQ5NDA3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T03:48:58Z", "updated_at": "2020-09-22T03:48:58Z", "author_association": "OWNER", "body": "One last thing. https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_change says that the first step should be:\r\n\r\n>  If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. \r\n\r\nAnd the last steps should be:\r\n\r\n> If foreign key constraints were originally enabled then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints.\r\n>\r\n> Commit the transaction started in step 2.\r\n>\r\n> If foreign keys constraints were originally enabled, reenable them now. \r\n\r\nI need to implement that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705975133, "label": "table.transform() method"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696500767", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114", "id": 696500767, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjUwMDc2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T04:21:45Z", "updated_at": "2020-09-22T04:21:45Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621989740, "label": "table.transform() method for advanced alter table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696500922", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/164", "id": 696500922, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjUwMDkyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T04:22:40Z", "updated_at": "2020-09-22T04:22:40Z", "author_association": "OWNER", "body": "Documentation for the `.transform()` method #114 (now landed) is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 706017416, "label": "sqlite-utils transform sub-command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696520928", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/164", "id": 696520928, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjUyMDkyOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T05:50:17Z", "updated_at": "2020-09-22T05:50:17Z", "author_association": "OWNER", "body": "Idea for CLI options:\r\n```\r\n--type age integer\r\n--drop colname\r\n--rename oldname newname\r\n--not-null col\r\n--not-null-false col\r\n--pk new_id\r\n--pk-none\r\n--default col value\r\n--default-none column\r\n--drop-foreign-key col other_table other_column\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 706017416, "label": "sqlite-utils transform sub-command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/167#issuecomment-696565981", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/167", "id": 696565981, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjU2NTk4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T07:53:13Z", "updated_at": "2020-09-22T07:53:13Z", "author_association": "OWNER", "body": "Confirmed this is a bug, https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes explicitly says you should do the `PRAGMA foreign_keys` bits before and after the transaction, not during.\r\n\r\nRight now my code does this INSIDE the transaction: https://github.com/simonw/sqlite-utils/blob/f29f6821f2d08e91c5c6d65d885a1bbc0c743bdd/sqlite_utils/db.py#L790-L793\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 706098005, "label": "Review the foreign key pragma stuff"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/26#issuecomment-696566750", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/26", "id": 696566750, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjU2Njc1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T07:55:00Z", "updated_at": "2020-09-22T07:55:00Z", "author_association": "OWNER", "body": "Problem: `extract` means something else now, see #47 and the upcoming work in #42.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 455486286, "label": "Mechanism for turning nested JSON into foreign keys / many-to-many"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696567460", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696567460, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjU2NzQ2MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T07:56:42Z", "updated_at": "2020-09-22T07:56:42Z", "author_association": "OWNER", "body": "`.transform()` has landed now which should make this a lot easier to solve.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696567988", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/164", "id": 696567988, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjU2Nzk4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T07:57:50Z", "updated_at": "2020-09-22T07:57:50Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#transforming-tables", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 706017416, "label": "sqlite-utils transform sub-command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/168#issuecomment-696573944", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/168", "id": 696573944, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjU3Mzk0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T08:11:30Z", "updated_at": "2020-09-22T08:11:30Z", "author_association": "OWNER", "body": "Huh... maybe I don't need to do anything here? It looks like it's been kept up to date: https://github.com/Homebrew/homebrew-core/commits/master/Formula/sqlite-utils.rb", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 706167456, "label": "Automate (as much as possible) updates published to Homebrew"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-696769501", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 696769501, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njc2OTUwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T14:45:49Z", "updated_at": "2020-09-22T14:45:49Z", "author_association": "OWNER", "body": "I put together a minimal prototype of this and it feels pretty good:\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 20aae7d..fb3bdad 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -4,6 +4,7 @@ import collections\r\n import datetime\r\n import glob\r\n import hashlib\r\n+import httpx\r\n import inspect\r\n import itertools\r\n from itsdangerous import BadSignature\r\n@@ -312,6 +313,7 @@ class Datasette:\r\n         self._register_renderers()\r\n         self._permission_checks = collections.deque(maxlen=200)\r\n         self._root_token = secrets.token_hex(32)\r\n+        self.client = DatasetteClient(self)\r\n \r\n     async def invoke_startup(self):\r\n         for hook in pm.hook.startup(datasette=self):\r\n@@ -1209,3 +1211,25 @@ def route_pattern_from_filepath(filepath):\r\n \r\n class NotFoundExplicit(NotFound):\r\n     pass\r\n+\r\n+\r\n+class DatasetteClient:\r\n+    def __init__(self, ds):\r\n+        self.app = ds.app()\r\n+\r\n+    def _fix(self, path):\r\n+        if path.startswith(\"/\"):\r\n+            path = \"http://localhost{}\".format(path)\r\n+        return path\r\n+\r\n+    async def get(self, path, **kwargs):\r\n+        async with httpx.AsyncClient(app=self.app) as client:\r\n+            return await client.get(self._fix(path), **kwargs)\r\n+\r\n+    async def post(self, path, **kwargs):\r\n+        async with httpx.AsyncClient(app=self.app) as client:\r\n+            return await client.post(self._fix(path), **kwargs)\r\n+\r\n+    async def options(self, path, **kwargs):\r\n+        async with httpx.AsyncClient(app=self.app) as client:\r\n+            return await client.options(self._fix(path), **kwargs)\r\n```\r\nUsed like this in `ipython`:\r\n```\r\nIn [1]: from datasette.app import Datasette\r\n\r\nIn [2]: ds = Datasette([\"fixtures.db\"])\r\n\r\nIn [3]: (await ds.client.get(\"/-/config.json\")).json()\r\nOut[3]: \r\n{'default_page_size': 100,\r\n 'max_returned_rows': 1000,\r\n 'num_sql_threads': 3,\r\n 'sql_time_limit_ms': 1000,\r\n 'default_facet_size': 30,\r\n 'facet_time_limit_ms': 200,\r\n 'facet_suggest_time_limit_ms': 50,\r\n 'hash_urls': False,\r\n 'allow_facet': True,\r\n 'allow_download': True,\r\n 'suggest_facets': True,\r\n 'default_cache_ttl': 5,\r\n 'default_cache_ttl_hashed': 31536000,\r\n 'cache_size_kb': 0,\r\n 'allow_csv_stream': True,\r\n 'max_csv_mb': 100,\r\n 'truncate_cells_html': 2048,\r\n 'force_https_urls': False,\r\n 'template_debug': False,\r\n 'base_url': '/'}\r\n\r\nIn [4]: (await ds.client.get(\"/fixtures/facetable.json?_shape=array\")).json()\r\nOut[4]: \r\n[{'pk': 1,\r\n  'created': '2019-01-14 08:00:00',\r\n  'planet_int': 1,\r\n  'on_earth': 1,\r\n  'state': 'CA',\r\n  'city_id': 1,\r\n  'neighborhood': 'Mission',\r\n  'tags': '[\"tag1\", \"tag2\"]',\r\n  'complex_array': '[{\"foo\": \"bar\"}]',\r\n  'distinct_some_null': 'one'},\r\n {'pk': 2,\r\n  'created': '2019-01-14 08:00:00',\r\n  'planet_int': 1,\r\n  'on_earth': 1,\r\n  'state': 'CA',\r\n  'city_id': 1,\r\n  'neighborhood': 'Dogpatch',\r\n  'tags': '[\"tag1\", \"tag3\"]',\r\n  'complex_array': '[]',\r\n  'distinct_some_null': 'two'},\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-696769853", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 696769853, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njc2OTg1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T14:46:21Z", "updated_at": "2020-09-22T14:46:21Z", "author_association": "OWNER", "body": "This adds `httpx` as a dependency - I think I'm OK with that. I use it for testing in all of my plugins anyway.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-696774711", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 696774711, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njc3NDcxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T14:53:56Z", "updated_at": "2020-09-22T14:53:56Z", "author_association": "OWNER", "body": "How important is it to use `httpx.AsyncClient` with a context manager?\r\n\r\nhttps://www.python-httpx.org/async/#opening-and-closing-clients says:\r\n\r\n> Alternatively, use `await client.aclose()` if you want to close a client explicitly:\r\n> \r\n> ```\r\n> client = httpx.AsyncClient()\r\n> ...\r\n> await client.aclose()\r\n> ```\r\nThe `.aclose()` method has a comment saying \"Close transport and proxies\" - I'm not using proxies, so the relevant implementation seems to be a call to `await self._transport.aclose()` in https://github.com/encode/httpx/blob/f932af9172d15a803ad40061a4c2c0cd891645cf/httpx/_client.py#L1741-L1751\r\n\r\nThe transport I am using is a class called `ASGITransport` in https://github.com/encode/httpx/blob/master/httpx/_transports/asgi.py\r\n\r\nThe `aclose()` method on that class does nothing. So it looks like I can instantiate a client without bothering with the `async with httpx.AsyncClient` bit.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-696775516", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 696775516, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njc3NTUxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T14:55:10Z", "updated_at": "2020-09-22T14:55:10Z", "author_association": "OWNER", "body": "Even smaller `DatasetteClient` implementation:\r\n```python\r\nclass DatasetteClient:\r\n    def __init__(self, ds):\r\n        self._client = httpx.AsyncClient(app=ds.app())\r\n\r\n    def _fix(self, path):\r\n        if path.startswith(\"/\"):\r\n            path = \"http://localhost{}\".format(path)\r\n        return path\r\n\r\n    async def get(self, path, **kwargs):\r\n        return await self._client.get(self._fix(path), **kwargs)\r\n\r\n    async def post(self, path, **kwargs):\r\n        return await self._client.post(self._fix(path), **kwargs)\r\n\r\n    async def options(self, path, **kwargs):\r\n        return await self._client.options(self._fix(path), **kwargs)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-696776828", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 696776828, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njc3NjgyOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T14:57:13Z", "updated_at": "2020-09-22T14:57:13Z", "author_association": "OWNER", "body": "I may as well implement all of the HTTP methods supported by the `httpx` client:\r\n\r\n- get\r\n- options\r\n- head\r\n- post\r\n- put\r\n- patch\r\n- delete", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-696777886", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 696777886, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njc3Nzg4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T14:58:54Z", "updated_at": "2020-09-22T14:58:54Z", "author_association": "OWNER", "body": "```python\r\nclass DatasetteClient:\r\n    def __init__(self, ds):\r\n        self._client = httpx.AsyncClient(app=ds.app())\r\n\r\n    def _fix(self, path):\r\n        if path.startswith(\"/\"):\r\n            path = \"http://localhost{}\".format(path)\r\n        return path\r\n\r\n    async def get(self, path, **kwargs):\r\n        return await self._client.get(self._fix(path), **kwargs)\r\n\r\n    async def options(self, path, **kwargs):\r\n        return await self._client.options(self._fix(path), **kwargs)\r\n\r\n    async def head(self, path, **kwargs):\r\n        return await self._client.head(self._fix(path), **kwargs)\r\n\r\n    async def post(self, path, **kwargs):\r\n        return await self._client.post(self._fix(path), **kwargs)\r\n\r\n    async def put(self, path, **kwargs):\r\n        return await self._client.put(self._fix(path), **kwargs)\r\n\r\n    async def patch(self, path, **kwargs):\r\n        return await self._client.patch(self._fix(path), **kwargs)\r\n\r\n    async def delete(self, path, **kwargs):\r\n        return await self._client.delete(self._fix(path), **kwargs)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-696778735", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 696778735, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njc3ODczNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T15:00:13Z", "updated_at": "2020-09-22T15:00:39Z", "author_association": "OWNER", "body": "Am I going to rewrite ALL of my tests to use this instead? It would clean up a lot of test code, at the cost of quite a bit of work.\r\n\r\nIt would make for much neater plugin tests too, and neater testing documentation: https://docs.datasette.io/en/stable/testing_plugins.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/969#issuecomment-696788109", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/969", "id": 696788109, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njc4ODEwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T15:15:14Z", "updated_at": "2020-09-22T15:15:14Z", "author_association": "OWNER", "body": "I don't think a standard \"pass these extra arguments to the publish tool\" mechanism will work because there's no guarantee that a publisher uses a CLI tool - or if it does, it might make several calls to different CLI tools. The Cloud Run one runs a couple of commands, as illustrated by this test:\r\n\r\nhttps://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/tests/test_publish_cloudrun.py#L63-L73\r\n\r\nAdding a `--tar` option for `datasette publish heroku` is a good fix for this though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 705057955, "label": "Add --tar option to \"datasette publish heroku\""}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/973#issuecomment-696798114", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/973", "id": 696798114, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njc5ODExNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T15:31:25Z", "updated_at": "2020-09-22T15:31:25Z", "author_association": "OWNER", "body": "D'oh because I have a new variable called `open`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 706486323, "label": "'bool' object is not callable error"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/973#issuecomment-696800410", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/973", "id": 696800410, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjgwMDQxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T15:35:28Z", "updated_at": "2020-09-22T15:35:28Z", "author_association": "OWNER", "body": "Confirmed in local dev:\r\n```\r\n% datasette fixtures.db --inspect-file inspect.json\r\nTraceback (most recent call last):\r\n  File \"/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette\", line 11, in <module>\r\n    load_entry_point('datasette', 'console_scripts', 'datasette')()\r\n  File \"/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n    return self.main(*args, **kwargs)\r\n  File \"/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n    rv = self.invoke(ctx)\r\n  File \"/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n  File \"/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n    return ctx.invoke(self.callback, **ctx.params)\r\n  File \"/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n    return callback(*args, **kwargs)\r\n  File \"/Users/simon/Dropbox/Development/datasette/datasette/cli.py\", line 406, in serve\r\n    inspect_data = json.load(open(inspect_file))\r\nTypeError: 'bool' object is not callable\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 706486323, "label": "'bool' object is not callable error"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893244", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696893244, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njg5MzI0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T18:14:33Z", "updated_at": "2020-09-22T18:14:45Z", "author_association": "OWNER", "body": "Thinking more about this one:\r\n```\r\n$ sqlite-utils extract my.db \\\r\n    dea_sales company_name company_address \\\r\n    --table companies\r\n```\r\nThe goal here is to pull the company name and address pair out into a separate table.\r\n\r\nSome questions:\r\n- should this first verify that every company_name has just one company_address? I like the idea of a unique constraint on the created table for this.\r\n- what should the foreign key column that gets added to the `companies` table be called?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893774", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696893774, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njg5Mzc3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T18:15:33Z", "updated_at": "2020-09-22T18:15:33Z", "author_association": "OWNER", "body": "I think the new foreign key column is called `company_name_id` by default in this example but can be customized by passing `--fk-column=xxx`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696976678", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696976678, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk3NjY3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T20:57:57Z", "updated_at": "2020-09-22T20:57:57Z", "author_association": "OWNER", "body": "I think I understand the shape of this feature now. It lets you specify one or more columns on the source table which will be extracted into another table. It uses the `.lookup()` mechanism to populate that other table, which means each unique column value / pair / triple will be assigned an integer ID.\r\n\r\nThat integer ID gets written back into the first of the columns that are being transformed. A `.transform()` call then converts that column to an integer (and drops the additional columns). Finally we set up the new foreign key relationship.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979168", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696979168, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk3OTE2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T21:02:24Z", "updated_at": "2020-09-22T21:02:24Z", "author_association": "OWNER", "body": "In Python it looks like this:\r\n```python\r\n# Simple case - species column species_id pointing to species table\r\ndb[\"trees\"].extract(\"species\")\r\n\r\n# Setting a custom table\r\ndb[\"trees\"].extract(\"species\", table=\"Species\")\r\n\r\n# Custom foreign key column on trees\r\ndb[\"trees\"].extract(\"species\", fk_column=\"species\")\r\n\r\n# Extracting multiple columns\r\ndb[\"trees\"].extract([\"common_name\", \"latin_name\"])\r\n# (this creates a lookup table called common_name_latin_name ref'd by common_name_latin_name_id)\r\n\r\n# Or with explicit table (fk_column here defaults to species_id because of the table name)\r\ndb[\"trees\"].extract([\"common_name\", \"latin_name\"], table=\"species\")\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979626", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696979626, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk3OTYyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T21:03:11Z", "updated_at": "2020-09-22T21:03:11Z", "author_association": "OWNER", "body": "And if you want to rename some of the columns in the new table:\r\n```python\r\ndb[\"trees\"].extract([\"common_name\", \"latin_name\"], table=\"species\", rename={\"common_name\": \"name\"})\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980503", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696980503, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk4MDUwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T21:04:45Z", "updated_at": "2020-09-22T21:04:45Z", "author_association": "OWNER", "body": "`table.extract()` can take an optional `progress=` argument which is a callback which will be used to report progress - called after each batch with `(num_done, total)`. It will get called with `(0, total)` once at the start to allow progress bars to be initialized. The command-line progress bar will use this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980709", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696980709, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk4MDcwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T21:05:07Z", "updated_at": "2020-09-22T21:05:07Z", "author_association": "OWNER", "body": "So `.extract()` probably takes a `batch_size=` argument too, which defaults to maybe 1000.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987257", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696987257, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk4NzI1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T21:17:34Z", "updated_at": "2020-09-22T21:17:34Z", "author_association": "OWNER", "body": "What to do if the table already exists? The `.lookup()` function already knows how to modify an existing table to create the correct constraints etc, so I'll rely on that mechanism.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987925", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696987925, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk4NzkyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T21:19:04Z", "updated_at": "2020-09-22T21:19:04Z", "author_association": "OWNER", "body": "Need to make sure this works correctly for `rowid` tables.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697012111", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 697012111, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzAxMjExMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T22:18:13Z", "updated_at": "2020-09-22T22:18:13Z", "author_association": "OWNER", "body": "Here's how I'm generating the examples for the documentation:\r\n```\r\nIn [2]: import sqlite_utils\r\n\r\nIn [3]: db = sqlite_utils.Database(memory=True)\r\n\r\nIn [4]: db[\"Trees\"].insert({\"id\": 1, \"TreeAddress\": \"52 Vine St\", \"CommonName\":\r\n   ...: \"Palm\", \"LatinName\": \"foo\"}, pk=\"id\")\r\nOut[4]: <Table Trees (id, TreeAddress, CommonName, LatinName)>\r\n\r\nIn [5]: db[\"Trees\"].extract([\"CommonName\", \"LatinName\"], table=\"Species\", fk_col\r\n   ...: umn=\"species_id\")\r\n\r\nIn [6]: print(db[\"Trees\"].schema)\r\nCREATE TABLE \"Trees\" (\r\n   [id] INTEGER PRIMARY KEY,\r\n   [TreeAddress] TEXT,\r\n   [species_id] INTEGER,\r\n   FOREIGN KEY(species_id) REFERENCES Species(id)\r\n)\r\n\r\nIn [7]: print(db[\"Species\"].schema)\r\nCREATE TABLE [Species] (\r\n   [id] INTEGER PRIMARY KEY,\r\n   [CommonName] TEXT,\r\n   [LatinName] TEXT\r\n)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697013681", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 697013681, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzAxMzY4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T22:22:49Z", "updated_at": "2020-09-22T22:22:49Z", "author_association": "OWNER", "body": "The command-line version of this needs to accept a table and one or more columns, then a `--table` and `--fk-column` option.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697019944", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 697019944, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzAxOTk0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T22:40:00Z", "updated_at": "2020-09-22T22:40:00Z", "author_association": "OWNER", "body": "I tried out the prototype of the CLI on the Global Power Plants data:\r\n```\r\nwget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'\r\nsqlite-utils insert global.db power_plants global_power_plant_database.csv --csv\r\nsqlite-utils extract global.db power_plants country country_long\r\n```\r\nThis threw an error because `rowid` columns are not yet supported. I fixed that like so:\r\n```\r\nsqlite-utils transform global.db power_plants --rename rowid id\r\nsqlite-utils extract global.db power_plants country country_long\r\n```\r\nThat worked! But it didn't play great with Datasette, because the resulting extracted table had columns `country` and `country_long` and neither of those are called `name` or `value` or `title`.\r\n\r\nBased on this I need to add `rowid` table support AND I need to implement the proposed `rename=` argument for renaming columns on their way into the new table.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697025403", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 697025403, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzAyNTQwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T22:57:53Z", "updated_at": "2020-09-22T22:57:53Z", "author_association": "OWNER", "body": "The documentation for the `.extract()` method is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#extracting-columns-into-a-separate-table", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697031174", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 697031174, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzAzMTE3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T23:16:00Z", "updated_at": "2020-09-22T23:16:00Z", "author_association": "OWNER", "body": "Trying this demo again:\r\n```\r\nwget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'\r\nsqlite-utils insert global.db power_plants global_power_plant_database.csv --csv\r\nsqlite-utils extract global.db power_plants country country_long --table countries --rename country_long name\r\n```\r\nIt worked!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697037974", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 697037974, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzAzNzk3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T23:39:31Z", "updated_at": "2020-09-22T23:39:31Z", "author_association": "OWNER", "body": "Documentation for `sqlite-utils extract`: https://sqlite-utils.readthedocs.io/en/latest/cli.html#extracting-columns-into-a-separate-table", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null}