{"html_url": "https://github.com/simonw/datasette/issues/422#issuecomment-474888132", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/422", "id": 474888132, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3NDg4ODEzMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-20T15:34:37Z", "updated_at": "2019-03-20T15:34:37Z", "author_association": "OWNER", "body": "Here's a trick for lower bound counts which looks like it might actually work. Consider the following queries:\r\n\r\n```\r\nselect count(*) from (\r\n  select rowid from [most-common-name/surnames] limit 1000\r\n)\r\n```\r\nhttps://fivethirtyeight.datasettes.com/fivethirtyeight-b76415d?sql=select+count%28*%29+from+%28%0D%0A++select+rowid+from+%5Bmost-common-name%2Fsurnames%5D+limit+1000%0D%0A%29\r\n\r\nTakes 0.827ms (it took longer with `select * from` in the subquery).\r\n\r\nSame query but with limit 10,000:\r\n\r\nhttps://fivethirtyeight.datasettes.com/fivethirtyeight-b76415d?sql=select+count%28*%29+from+%28%0D%0A++select+rowid++from+%5Bmost-common-name%2Fsurnames%5D+limit+10000%0D%0A%29\r\n\r\nTook 2.335ms\r\n\r\nWith 100,000 limit:\r\n\r\nhttps://fivethirtyeight.datasettes.com/fivethirtyeight-b76415d?sql=select+count%28*%29+from+%28%0D%0A++select+rowid++from+%5Bmost-common-name%2Fsurnames%5D+limit+100000%0D%0A%29\r\n\r\nTook 27.558ms\r\n\r\nSo one solution here would be to pick an upper bound (maybe 100,001) and use this query, which should give an accurate count below that upper bound but allow us to show \"100,000+\" as a count if the table exceeds that boundary.\r\n\r\nMaybe the boundary is a config setting? Also, if a tighter timeout (maybe 20ms) is exceeded for that boundary we could halve it and try again.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 423316403, "label": "Figure out what to do about table counts in a mutable world"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/422#issuecomment-474890920", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/422", "id": 474890920, "node_id": "MDEyOklzc3VlQ29tbWVudDQ3NDg5MDkyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-03-20T15:39:58Z", "updated_at": "2019-03-20T15:39:58Z", "author_association": "OWNER", "body": "The page with the most table counts on it is the index page: https://fivethirtyeight.datasettes.com/fivethirtyeight-b76415d\r\n\r\n<img width=\"1032\" alt=\"Screen Shot 2019-03-20 at 8 36 00 AM\" src=\"https://user-images.githubusercontent.com/9599/54697753-5710ce80-4aeb-11e9-89aa-13f843a78d2c.png\">\r\n\r\nIf I paginate this (which needs to happen anyway for Datasette Library #417) the impact here won't be as bad. I could even load in the table row counts asynchronously via JavaScript?\r\n\r\nBigger problem is this total summed count representation on the homepage:\r\n\r\n<img width=\"1022\" alt=\"Screen Shot 2019-03-20 at 8 38 53 AM\" src=\"https://user-images.githubusercontent.com/9599/54697913-a3f4a500-4aeb-11e9-97a4-be501d737642.png\">\r\n\r\nI think that feature just won't be feasibly against large databases in a mutable world. Maybe we consider to show that total but only for immutable databases? May be easier just to drop it entirely (we will still show the table count).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 423316403, "label": "Figure out what to do about table counts in a mutable world"}, "performed_via_github_app": null}