{"html_url": "https://github.com/simonw/sqlite-utils/issues/26#issuecomment-696566750", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/26", "id": 696566750, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjU2Njc1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T07:55:00Z", "updated_at": "2020-09-22T07:55:00Z", "author_association": "OWNER", "body": "Problem: `extract` means something else now, see #47 and the upcoming work in #42.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 455486286, "label": "Mechanism for turning nested JSON into foreign keys / many-to-many"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696567460", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696567460, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NjU2NzQ2MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T07:56:42Z", "updated_at": "2020-09-22T07:56:42Z", "author_association": "OWNER", "body": "`.transform()` has landed now which should make this a lot easier to solve.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893244", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696893244, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njg5MzI0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T18:14:33Z", "updated_at": "2020-09-22T18:14:45Z", "author_association": "OWNER", "body": "Thinking more about this one:\r\n```\r\n$ sqlite-utils extract my.db \\\r\n dea_sales company_name company_address \\\r\n --table companies\r\n```\r\nThe goal here is to pull the company name and address pair out into a separate table.\r\n\r\nSome questions:\r\n- should this first verify that every company_name has just one company_address? I like the idea of a unique constraint on the created table for this.\r\n- what should the foreign key column that gets added to the `companies` table be called?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893774", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696893774, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njg5Mzc3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T18:15:33Z", "updated_at": "2020-09-22T18:15:33Z", "author_association": "OWNER", "body": "I think the new foreign key column is called `company_name_id` by default in this example but can be customized by passing `--fk-column=xxx`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696976678", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696976678, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk3NjY3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T20:57:57Z", "updated_at": "2020-09-22T20:57:57Z", "author_association": "OWNER", "body": "I think I understand the shape of this feature now. It lets you specify one or more columns on the source table which will be extracted into another table. It uses the `.lookup()` mechanism to populate that other table, which means each unique column value / pair / triple will be assigned an integer ID.\r\n\r\nThat integer ID gets written back into the first of the columns that are being transformed. A `.transform()` call then converts that column to an integer (and drops the additional columns). Finally we set up the new foreign key relationship.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979168", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696979168, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk3OTE2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T21:02:24Z", "updated_at": "2020-09-22T21:02:24Z", "author_association": "OWNER", "body": "In Python it looks like this:\r\n```python\r\n# Simple case - species column species_id pointing to species table\r\ndb[\"trees\"].extract(\"species\")\r\n\r\n# Setting a custom table\r\ndb[\"trees\"].extract(\"species\", table=\"Species\")\r\n\r\n# Custom foreign key column on trees\r\ndb[\"trees\"].extract(\"species\", fk_column=\"species\")\r\n\r\n# Extracting multiple columns\r\ndb[\"trees\"].extract([\"common_name\", \"latin_name\"])\r\n# (this creates a lookup table called common_name_latin_name ref'd by common_name_latin_name_id)\r\n\r\n# Or with explicit table (fk_column here defaults to species_id because of the table name)\r\ndb[\"trees\"].extract([\"common_name\", \"latin_name\"], table=\"species\")\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979626", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696979626, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk3OTYyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T21:03:11Z", "updated_at": "2020-09-22T21:03:11Z", "author_association": "OWNER", "body": "And if you want to rename some of the columns in the new table:\r\n```python\r\ndb[\"trees\"].extract([\"common_name\", \"latin_name\"], table=\"species\", rename={\"common_name\": \"name\"})\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980503", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696980503, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk4MDUwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T21:04:45Z", "updated_at": "2020-09-22T21:04:45Z", "author_association": "OWNER", "body": "`table.extract()` can take an optional `progress=` argument which is a callback which will be used to report progress - called after each batch with `(num_done, total)`. It will get called with `(0, total)` once at the start to allow progress bars to be initialized. The command-line progress bar will use this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980709", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696980709, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk4MDcwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T21:05:07Z", "updated_at": "2020-09-22T21:05:07Z", "author_association": "OWNER", "body": "So `.extract()` probably takes a `batch_size=` argument too, which defaults to maybe 1000.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987257", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696987257, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk4NzI1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T21:17:34Z", "updated_at": "2020-09-22T21:17:34Z", "author_association": "OWNER", "body": "What to do if the table already exists? The `.lookup()` function already knows how to modify an existing table to create the correct constraints etc, so I'll rely on that mechanism.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987925", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 696987925, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Njk4NzkyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T21:19:04Z", "updated_at": "2020-09-22T21:19:04Z", "author_association": "OWNER", "body": "Need to make sure this works correctly for `rowid` tables.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 470345929, "label": "table.extract(...) method and \"sqlite-utils extract\" command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697012111", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42", "id": 697012111, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzAxMjExMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-22T22:18:13Z", "updated_at": "2020-09-22T22:18:13Z", "author_association": "OWNER", "body": "Here's how I'm generating the examples for the documentation:\r\n```\r\nIn [2]: import sqlite_utils\r\n\r\nIn [3]: db = sqlite_utils.Database(memory=True)\r\n\r\nIn [4]: db[\"Trees\"].insert({\"id\": 1, \"TreeAddress\": \"52 Vine St\", \"CommonName\":\r\n ...: \"Palm\", \"LatinName\": \"foo\"}, pk=\"id\")\r\nOut[4]: