{"html_url": "https://github.com/simonw/datasette/issues/856#issuecomment-843065142", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/856", "id": 843065142, "node_id": "MDEyOklzc3VlQ29tbWVudDg0MzA2NTE0Mg==", "user": {"value": 5268174, "label": "bram2000"}, "created_at": "2021-05-18T10:49:11Z", "updated_at": "2021-05-18T10:49:29Z", "author_association": "NONE", "body": "Hi Simon, I'm using a canned query to do some geospatial stuff, but it maxes out at 1000 rows returned. I can't see any `Link` headers to follow to get the next page of data. Is there any way currently to work around this 1000 row limit for canned queries?\r\n\r\nThanks,\r\nJon", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642296989, "label": "Consider pagination of canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/856#issuecomment-843291675", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/856", "id": 843291675, "node_id": "MDEyOklzc3VlQ29tbWVudDg0MzI5MTY3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-05-18T15:56:45Z", "updated_at": "2021-05-18T15:56:45Z", "author_association": "OWNER", "body": "Tables and views get \"stream all rows\" at the moment, so one workaround is to define a SQL view for your query - this only works for queries that don't take any parameters though (although you may be able to define a view and then pass it extra fields using the Datasette table interface, like on https://latest.datasette.io/fixtures/paginated_view?content_extra__contains=9)\r\n\r\nI've explored this problem in a bit more detail in https://githu.com/simonw/django-sql-dashboard and I think I have a pattern that could work.\r\n\r\nFor your canned query, you could implement the pattern yourself by setting up two canned queries that look something like this:\r\n\r\nhttps://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+sha%2C+author_date+from+commits+order+by+rowid+limit+1000\r\n\r\n```sql\r\nselect rowid, sha, author_date from commits order by rowid limit 1000\r\n```\r\nThat gets you the first set of 1,000 results. The important thing here is to order by a unique column, in this case `rowid` - because then subsequent pages can be loaded by a separate canned query that looks like this:\r\n```sql\r\nselect rowid, sha, author_date from commits where rowid > :after order by rowid limit 1000\r\n```\r\nhttps://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+sha%2C+author_date+from+commits+where+rowid+%3E+%3Aafter+order+by+rowid+limit+1000&after=1000\r\n\r\nYou then need to write code which knows how to generate these queries - start with the first query with no `where` clause (or if you are using `rowid` you can just use the second query and pass it `?after=0` for the first call) - then keep calling the query passing in the last rowid you recieved as the `after` parameter.\r\n\r\nBasically this is an implementation of keyset pagination with a smart client. When Datasette grows the ability to do this itself it will work by executing this mechanism inside the Python code, which is how the \"stream all rows\" option for tables works at the moment.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642296989, "label": "Consider pagination of canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/856#issuecomment-845985439", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/856", "id": 845985439, "node_id": "MDEyOklzc3VlQ29tbWVudDg0NTk4NTQzOQ==", "user": {"value": 5268174, "label": "bram2000"}, "created_at": "2021-05-21T14:22:41Z", "updated_at": "2021-05-21T14:22:41Z", "author_association": "NONE", "body": "Thanks Simon this is working very well.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642296989, "label": "Consider pagination of canned queries"}, "performed_via_github_app": null}