{"html_url": "https://github.com/simonw/datasette/issues/417#issuecomment-474280581", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/417", "id": 474280581, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3NDI4MDU4MQ==", "user": {"value": 82988, "label": "psychemedia"}, "created_at": "2019-03-19T10:06:42Z", "updated_at": "2019-03-19T10:06:42Z", "author_association": "CONTRIBUTOR", "body": "This would be really interesting but several possibilities in use arise, I think?\r\n\r\nFor example:\r\n\r\n- I put a new CSV file into the import dir and a new table is created therefrom\r\n- I put a CSV file into the import dir that replaces a previous file / table of the same name as a pre-existing table (eg files that contain monthly data in year to date). The data may also patch previous months, so a full replace / DROP on the original table may well be in order.\r\n- I put a CSV file into the import dir that updates a table of the same name as a pre-existing table (eg files that contain last month's data)\r\n\r\nCSV files may also have messy names compared to the table you want. Or for an update CSV, may have the form `MYTABLENAME-February2019.csv` etc", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421546944, "label": "Datasette Library"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/412#issuecomment-474282321", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/412", "id": 474282321, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3NDI4MjMyMQ==", "user": {"value": 82988, "label": "psychemedia"}, "created_at": "2019-03-19T10:09:46Z", "updated_at": "2019-03-19T10:09:46Z", "author_association": "CONTRIBUTOR", "body": "Does this also relate to https://github.com/simonw/datasette/issues/283 and the ability to `ATTACH DATABASE`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 411257981, "label": "Linked Data(sette)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/420#issuecomment-474398127", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/420", "id": 474398127, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3NDM5ODEyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-19T14:34:55Z", "updated_at": "2019-03-19T14:34:55Z", "author_association": "OWNER", "body": "I systematically reviewed the codebase for things that `.inspect()` is used for:\r\n\r\nIn `app.py`:\r\n\r\n* `table_exists()` uses `table in self.inspect().get(database, {}).get(\"tables\")`\r\n* `.execute()` looks up the database name to get the `info[\"file\"]` (the correct filename with the `.db` extension)\r\n\r\nIn `cli.py`:\r\n\r\n* The `datasette inspect` command dumps it to JSON\r\n* `datasette skeleton` iterates over it\r\n* `datasette serve` calls it on startup (to populate static cache of inspect data)\r\n\r\nIn `base.py`:\r\n\r\n* `.database_url(database)` calls it to lookup the hash (if `hash_urls` config turned on)\r\n* `.resolve_db_name()` uses it to lookup the hash\r\n\r\nIn `database.py`:\r\n\r\n* `DatabaseView` uses it to find up the list of tables and views to display, plus the size of the DB file in bytes\r\n* `DatabaseDownload` uses it to get the filepath for download\r\n\r\nIn `index.py`:\r\n\r\n* `IndexView` uses it _extensively_ - to loop through every database and every table. This would make a good starting point for the refactor.\r\n\r\nIn `table.py`:\r\n\r\n* `sortable_columns_for_table()` uses it to find the columns in a table\r\n* `expandable_columns()` uses it to find foreign keys\r\n* `expand_foreign_keys()` uses it to find foreign keys\r\n* `display_columns_and_rows()` uses it to find primary keys and foreign keys... but also has access to a `cursor.description` which it uses to list the columns\r\n* `TableView.data` uses it to lookup columns and primary keys and the `table_rows_count` (used if the thing isn't a view) and probably a few more things, this method is huge!\r\n* `RowView.data` uses it for primary keys\r\n* `foreign_key_tables()` uses it for foreign keys\r\n\r\nIn the tests it's used by `test_api.test_inspect_json()` and by a couple of tests in `test_inspect`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421971339, "label": "Fix all the places that currently use .inspect() data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/420#issuecomment-474399630", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/420", "id": 474399630, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3NDM5OTYzMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-19T14:38:14Z", "updated_at": "2019-03-19T14:38:14Z", "author_association": "OWNER", "body": "Most of these can be replaced with relatively straight-forward direct introspection of the SQLite table.\r\n\r\nThe one exception is the incoming foreign keys: these can only be found by inspecting ALL of the other tables. \r\n\r\nThis requires running `PRAGMA foreign_key_list([table_name])` against every other table in the database. How expensive is doing this on a database with hundreds of tables?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421971339, "label": "Fix all the places that currently use .inspect() data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/420#issuecomment-474407617", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/420", "id": 474407617, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3NDQwNzYxNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-19T14:55:51Z", "updated_at": "2019-03-19T14:55:51Z", "author_association": "OWNER", "body": "A microbenchmark against `fivethirtyeight.db` (415 tables):\r\n\r\n    In [1]: import sqlite3                                                                                              \r\n    In [2]: c = sqlite3.connect(\"fivethirtyeight.db\")                                                                   \r\n    In [3]: %timeit c.execute(\"select name from sqlite_master where type = 'table'\").fetchall()                         \r\n    283 \u00b5s \u00b1 12.3 \u00b5s per loop (mean \u00b1 std. dev. of 7 runs, 1000 loops each)\r\n    In [4]: tables = [r[0] for r in c.execute(\"select name from sqlite_master where type = 'table'\").fetchall()]        \r\n    In [5]: len(tables)                                                                                                 \r\n    Out[5]: 415\r\n    In [6]: %timeit [c.execute(\"pragma foreign_keys([{}])\".format(t)).fetchall() for t in tables]                       \r\n    1.81 ms \u00b1 161 \u00b5s per loop (mean \u00b1 std. dev. of 7 runs, 1000 loops each)\r\n\r\nSo running `pragma foreign_keys()` against 415 tables only takes 1.81ms. This is going to be fine.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 421971339, "label": "Fix all the places that currently use .inspect() data"}, "performed_via_github_app": null}