issue_comments
9,947 rows sorted by node_id
This data as json, CSV (advanced)
Suggested facets: reactions, created_at (date)
issue >1000 ✖
- Redesign default .json format 54
- Show column metadata plus links for foreign keys on arbitrary query results 51
- ?_extra= support (draft) 49
- Rethink how .ext formats (v.s. ?_format=) works before 1.0 48
- Upgrade to CodeMirror 6, add SQL autocomplete 48
- JavaScript plugin hooks mechanism similar to pluggy 47
- Updated Dockerfile with SpatiaLite version 5.0 45
- Complete refactor of TableView and table.html template 45
- Port Datasette to ASGI 42
- Authentication (and permissions) as a core concept 40
- invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 36
- Deploy a live instance of demos/apache-proxy 34
- await datasette.client.get(path) mechanism for executing internal requests 33
- Maintain an in-memory SQLite table of connected databases and their tables 32
- Research: demonstrate if parallel SQL queries are worthwhile 32
- Ability to sort (and paginate) by column 31
- Default API token authentication mechanism 30
- Port as many tests as possible to async def tests against ds_client 29
- link_or_copy_directory() error - Invalid cross-device link 28
- Add ?_extra= mechanism for requesting extra properties in JSON 27
- Export to CSV 27
- base_url configuration setting 27
- Documentation with recommendations on running Datasette in production without using Docker 27
- Optimize all those calls to index_list and foreign_key_list 27
- Support cross-database joins 26
- Ability for a canned query to write to the database 26
- table.transform() method for advanced alter table 26
- New pattern for views that return either JSON or HTML, available for plugins 26
- Proof of concept for Datasette on AWS Lambda with EFS 25
- WIP: Add Gmail takeout mbox import 25
- Make it easier to insert geometries, with documentation and maybe code 25
- Redesign register_output_renderer callback 24
- API explorer tool 24
- Stream all results for arbitrary SQL and canned queries 23
- "datasette insert" command and plugin hook 23
- Datasette Plugins 22
- .json and .csv exports fail to apply base_url 22
- Use YAML examples in documentation by default, not JSON 22
- Idea: import CSV to memory, run SQL, export in a single command 22
- Plugin hook for dynamic metadata 22
- Option for importing CSV data using the SQLite .import mechanism 22
- base_url is omitted in JSON and CSV views 22
- API to insert a single record into an existing table 22
- UI to create reduced scope tokens from the `/-/create-token` page 22
- Handle spatialite geometry columns better 21
- table.extract(...) method and "sqlite-utils extract" command 21
- Database page loads too slowly with many large tables (due to table counts) 21
- ?sort=colname~numeric to sort by by column cast to real 21
- Switch documentation theme to Furo 21
- "flash messages" mechanism 20
- Move CI to GitHub Issues 20
- load_template hook doesn't work for include/extends 20
- Mechanism for storing metadata in _metadata tables 20
- Introduce concept of a database `route`, separate from its name 20
- CSV files with too many values in a row cause errors 20
- register_permissions(datasette) plugin hook 20
- Better way of representing binary data in .csv output 19
- Introspect if table is FTS4 or FTS5 19
- A proper favicon 19
- `datasette create-token` ability to create tokens with a reduced set of permissions 19
- Package as standalone binary 18
- Ability to ship alpha and beta releases 18
- Magic parameters for canned queries 18
- Support column descriptions in metadata.json 18
- datasette.client internal requests mechanism 18
- Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified 18
- Figure out why SpatiaLite 5.0 hangs the database page on Linux 18
- Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 18
- Update screenshots in documentation to match latest designs 18
- Consider using CSP to protect against future XSS 17
- Facets 16
- ?_col= and ?_nocol= support for toggling columns on table view 16
- Support "allow" block on root, databases and tables, not just queries 16
- Action menu for table columns 16
- Make it possible to download BLOB data from the Datasette UI 16
- sqlite-utils extract could handle nested objects 16
- `--batch-size 1` doesn't seem to commit for every item 16
- create-index should run analyze after creating index 16
- Add new spatialite helper methods 16
- Intermittent "Too many open files" error running tests 16
- Update a single record in an existing table 16
- Autocomplete text entry for filter values that correspond to facets 16
- feat: Javascript Plugin API (Custom panels, column menu items with JS actions) 16
- Resolve the difference between `wrap_view()` and `BaseView` 16
- Bug: Sort by column with NULL in next_page URL 15
- Mechanism for customizing the SQL used to select specific columns in the table view 15
- The ".upsert()" method is misnamed 15
- --dirs option for scanning directories for SQLite databases 15
- Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread() 15
- latest.datasette.io is no longer updating 15
- "sqlite-utils convert" command to replace the separate "sqlite-transform" tool 15
- --lines and --text and --convert and --import 15
- Documentation should clarify /stable/ vs /latest/ 15
- Tests reliably failing on Python 3.7 15
- Transformation type `--type DATETIME` 15
- Ability to customize presentation of specific columns in HTML view 14
- Allow plugins to define additional URL routes and views 14
- Mechanism for turning nested JSON into foreign keys / many-to-many 14
- "Invalid SQL" page should let you edit the SQL 14
- Prototoype for Datasette on PostgreSQL 14
- .execute_write() and .execute_write_fn() methods on Database 14
- Upload all my photos to a secure S3 bucket 14
- Canned query permissions mechanism 14
- Incorrect URLs when served behind a proxy with base_url set 14
- "datasette -p 0 --root" gives the wrong URL 14
- Plugin hook for loading templates 14
- Advanced class-based `conversions=` mechanism 14
- "permissions" propery in metadata for configuring arbitrary permissions 14
- Design plugin hook for extras 14
- `handle_exception` plugin hook for custom error handling 14
- Refactor out the keyset pagination code 14
- Potential feature: special support for `?a=1&a=2` on the query page 14
- Dockerfile should build more recent SQLite with FTS5 and spatialite support 13
- Fix all the places that currently use .inspect() data 13
- Plugin hook: filters_from_request 13
- Get Datasette tests passing on Windows in GitHub Actions 13
- If you apply ?_facet_array=tags then &_facet=tags does nothing 13
- Mechanism for adding arbitrary pages like /about 13
- Import machine-learning detected labels (dog, llama etc) from Apple Photos 13
- Mechanism for skipping CSRF checks on API posts 13
- table.transform() method 13
- register_output_renderer() should support streaming data 13
- Policy on documenting "public" datasette.utils functions 13
- Async support 13
- Serve using UNIX domain socket 13
- `register_commands()` plugin hook to register extra CLI commands 13
- Fix compatibility with Python 3.10 13
- Support STRICT tables 13
- Optional Pandas integration 13
- Refactor TableView to use asyncinject 13
- Writable canned queries fail with useless non-error against immutable databases 13
- google cloudrun updated their limits on maxscale based on memory and cpu count 13
- Mechanism for ensuring a table has all the columns 13
- docker image is duplicating db files somehow 13
- Write API in Datasette core 13
- Errors when using table filters behind a proxy 13
- Call for birthday presents: if you're using Datasette, let us know how you're using it here 13
- Make sure CORS works for write APIs 13
- Make detailed notes on how table, query and row views work right now 13
- Add “updated” to metadata 12
- Metadata should be a nested arbitrary KV store 12
- Mechanism for ranking results from SQLite full-text search 12
- Sanely handle Infinity/-Infinity values in JSON using ?_json_infinity=1 12
- Package datasette for installation using homebrew 12
- Datasette Library 12
- _facet_array should work against views 12
- Full text search of all tables at once? 12
- Port Datasette from Sanic to ASGI + Uvicorn 12
- Populate "endpoint" key in ASGI scope 12
- --cp option for datasette publish and datasette package for shipping additional files and directories 12
- base_url doesn't entirely work for running Datasette inside Binder 12
- Having view-table permission but NOT view-database should still grant access to /db/table 12
- Efficiently calculate list of databases/tables a user can view 12
- Support creating descending order indexes 12
- Rethink approach to [ and ] in column names (currently throws error) 12
- Research: CTEs and union all to calculate facets AND query at the same time 12
- Traces should include SQL executed by subtasks created with `asyncio.gather` 12
- Ensure "pip install datasette" still works with Python 3.6 12
- Tilde encoding: use ~ instead of - for dash-encoding 12
- Document how to use a `--convert` function that runs initialization code first 12
- Code examples in the documentation should be formatted with Black 12
- Implement ?_extra and new API design for TableView 12
- Misleading progress bar against utf-16-le CSV input 12
- sqlite-utils query --functions mechanism for registering extra functions 12
- SQL query field can't begin by a comment 12
- API for bulk inserting records into a table 12
- `/db/-/create` API for creating tables 12
- Implement command-line tool interface 11
- Option to expose expanded foreign keys in JSON/CSV 11
- datasette publish lambda plugin 11
- Mechanism for checking if a SQLite database file is safe to open 11
- Expand plugins documentation to multiple pages 11
- Mechanism for plugins to add action menu items for various things 11
- --since feature can be confused by retweets 11
- bpylist.archiver.CircularReference: archive has a cycle with uid(13) 11
- Datasette secret mechanism - initially for signed cookies 11
- Writable canned queries live demo on Glitch 11
- base_url doesn't seem to work when adding criteria and clicking "apply" 11
- POST to /db/canned-query that returns JSON should be supported (for API clients) 11
- datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates 11
- Writable canned queries with magic parameters fail if POST body is empty 11
- Database class mechanism for cross-connection in-memory databases 11
- Ability for plugins to collaborate when adding extra HTML to blocks in default templates 11
- Race condition errors in new refresh_schemas() mechanism 11
- "Query parameters" form shows wrong input fields if query contains "03:31" style times 11
- render_cell() hook should support returning an awaitable 11
- sqlite-utils index-foreign-keys fails due to pre-existing index 11
- `sqlite-utils insert --convert` option 11
- Research how much of a difference analyze / sqlite_stat1 makes 11
- Table+query JSON and CSV links broken when using `base_url` setting 11
- Options for how `r.parsedate()` should handle invalid dates 11
- Research: how much overhead does the n=1 time limit have? 11
- Expose convert recipes to `sqlite-utils --functions` 11
- `prepare_jinja2_environment()` hook should take `datasette` argument 11
- Ensure insert API has good tests for rowid and compound primark key tables 11
- datasette package --spatialite throws error during build 11
- datasette --root running in Docker doesn't reliably show the magic URL 11
- Set up some example datasets on a Cloudflare-backed domain 10
- Filter UI on table page 10
- Support for units 10
- Build Dockerfile with recent Sqlite + Spatialite 10
- Table view should support filtering via many-to-many relationships 10
- Default to opening files in mutable mode, special option for immutable files 10
- New design for facet abstraction, including querystring and metadata.json 10
- Improvements to table label detection 10
- Syntactic sugar for creating m2m records 10
- Option to display binary data 10
- extracts= should support multiple-column extracts 10
- Documented internals API for use in plugins 10
- Mechanism for writing to database via a queue 10
- See if I can get Datasette working on Zeit Now v2 10
- Ability to serve thumbnailed Apple Photo from its place on disk 10
- Release Datasette 0.44 10
- Rename master branch to main 10
- Plugin hook for instance/database/table metadata 10
- Refactor default views to use register_routes 10
- CLI utility for inserting binary files into SQLite 10
- FTS table with 7 rows has _fts_docsize table with 9,141 rows 10
- Switch to .blob render extension for BLOB downloads 10
- Navigation menu plus plugin hook 10
- Adopt Prettier for JavaScript code formatting 10
- --no-headers option for CSV and TSV 10
- Add support for Jinja2 version 3.0 10
- Test Datasette Docker images built for different architectures 10
- Research: syntactic sugar for using --get with SQL queries, maybe "datasette query" 10
- `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 10
- Add reference page to documentation using Sphinx autodoc 10
- [Enhancement] Please allow 'insert-files' to insert content as text. 10
- Docker configuration for exercising Datasette behind Apache mod_proxy 10
- Python library methods for calling ANALYZE 10
- Scripted exports 10
- Remove Hashed URL mode 10
- If user can see table but NOT database/instance nav links should not display 10
- test_recreate failing on Windows Python 3.11 10
- `.json` errors should be returned as JSON 10
- Failing test: httpx.InvalidURL: URL too long 10
- Use sqlean if available in environment 10
- Config file with support for defining canned queries 9
- Datasette serve should accept paths/URLs to CSVs and other file formats 9
- Figure out some interesting example SQL queries 9
- bump uvicorn to 0.9.0 to be Python-3.8 friendly 9
- Handle really wide tables better 9
- Refactor TableView.data() method 9
- Set up a live demo Datasette instance 9
- Move hashed URL mode out to a plugin 9
- ?_searchmode=raw option for running FTS searches without escaping characters 9
- Option to automatically configure based on directory layout 9
- Replace "datasette publish --extra-options" with "--setting" 9
- New WIP writable canned queries 9
- Example permissions plugin 9
- Research feasibility of 100% test coverage 9
- canned_queries() plugin hook 9
- Consider dropping explicit CSRF protection entirely? 9
- Add insert --truncate option 9
- .delete_where() does not auto-commit (unlike .insert() or .upsert()) 9
- Improve performance of extract operations 9
- Figure out how to run an environment that exercises the base_url proxy setting 9
- sqlite-utils search command 9
- GENERATED column support 9
- Datasette on Amazon Linux on ARM returns 404 for static assets 9
- "Stream all rows" is not at all obvious 9
- Better internal database_name for _internal database 9
- Mechanism for minifying JavaScript that ships with Datasette 9
- Mechanism for executing JavaScript unit tests 9
- Use _counts to speed up counts 9
- Use force_https_urls on when deploying with Cloud Run 9
- Ability to increase size of the SQL editor window 9
- Custom pages don't work with base_url setting 9
- CSV ?_stream=on redundantly calculates facets for every page 9
- "invalid reference format" publishing Docker image 9
- Show count of facet values if ?_facet_size=max 9
- Manage /robots.txt in Datasette core, block robots by default 9
- Exceeding Cloud Run memory limits when deploying a 4.8G database 9
- Test against pysqlite3 running SQLite 3.37 9
- Allow to set `facets_array` in metadata (like current `facets`) 9
- Add SpatiaLite helpers to CLI 9
- Get Datasette compatible with Pyodide 9
- Add --ignore option to more commands 9
- Ability to set a custom favicon 9
- i18n support 9
- Add new entrypoint option to `--load-extension` 9
- Ability to load JSON records held in a file with a single top level key that is a list of objects 9
- Tool for simulating permission checks against actors 9
- Serve schema JSON to the SQL editor to enable autocomplete 9
- Release Datasette 1.0a0 9
- Refactor test suite to use mostly `async def` tests 9
- 500 "attempt to write a readonly database" error caused by "PRAGMA schema_version" 9
- `table.upsert_all` fails to write rows when `not_null` is present 9
- Make URLs immutable 8
- datasette publish heroku 8
- Ability to bundle and serve additional static files 8
- Add GraphQL endpoint 8
- prepare_context() plugin hook 8
- Wildcard support in query parameters 8
- URL hashing now optional: turn on with --config hash_urls:1 (#418) 8
- "datasette publish cloudrun" command to publish to Google Cloud Run 8
- Add register_output_renderer hook 8
- sqlite-utils create-table command 8
- Add a universal navigation bar which can be modified by plugins 8
- Command to fetch stargazers for one or more repos 8
- allow leading comments in SQL input field 8
- Helper methods for working with SpatiaLite 8
- datasette publish cloudrun --memory option 8
- Commits in GitHub API can have null author 8
- extra_template_vars() sending wrong view_name for index 8
- Import photo metadata from Apple Photos into SQLite 8
- Visually distinguish integer and text columns 8
- sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns 8
- Allow-list pragma_table_info(tablename) and similar 8
- Rename project to dogsheep-photos 8
- Consolidate request.raw_args and request.args 8
- Group permission checks by request on /-/permissions debug page 8
- Upgrade CodeMirror 8
- Mechanism for defining custom display of results 8
- the JSON object must be str, bytes or bytearray, not 'Undefined' 8
- Wide tables should scroll horizontally within the page 8
- OPTIONS requests return a 500 error 8
- Bring date parsing into Datasette core 8
- Establish pattern for release branches to support bug fixes 8
- Improve the display of facets information 8
- GitHub Actions workflow to build and sign macOS binary executables 8
- Make original path available to render hooks 8
- --sniff option for sniffing delimiters 8
- --crossdb option for joining across databases 8
- sqlite-utils memory command for directly querying CSV/JSON data 8
- absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs 8
- Tests failing with FileNotFoundError in runner.isolated_filesystem 8
- Rename Datasette.__init__(config=) parameter to settings= 8
- Allow passing a file of code to "sqlite-utils convert" 8
- Documented JavaScript variables on different templates made available for plugins 8
- Support for generated columns 8
- Get rid of the no-longer necessary ?_format=json hack for tables called x.json 8
- Refactor and simplify Datasette routing and views 8
- Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 8
- "Error: near "(": syntax error" when using sqlite-utils indexes CLI 8
- Table/database that is private due to inherited permissions does not show padlock 8
- /db/table/-/upsert API 8
- Some plugins show "home" breadcrumbs twice in the top left 8
- /db/table/-/upsert 8
- array facet: don't materialize unnecessary columns 8
- Datasette should serve Access-Control-Max-Age 8
- ?_group_count=country - return counts by specific column(s) 7
- Ship a Docker image of the whole thing 7
- add "format sql" button to query page, uses sql-formatter 7
- Windows installation error 7
- Keyset pagination doesn't work correctly for compound primary keys 7
- Validate metadata.json on startup 7
- inspect should record column types 7
- Travis should push tagged images to Docker Hub for each release 7
- Improve and document foreign_keys=... argument to insert/create/etc 7
- ?_where=sql-fragment parameter for table views 7
- Define mechanism for plugins to return structured data 7
- Utility mechanism for plugins to render templates 7
- Syntax for ?_through= that works as a form field 7
- Problem with square bracket in CSV column name 7
- Update SQLite bundled with Docker container 7
- index.html is not reliably loaded from a plugin 7
- .columns_dict doesn't work for all possible column types 7
- Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 7
- Expose scores from ZCOMPUTEDASSETATTRIBUTES 7
- publish heroku does not work on Windows 10 7
- Demo is failing to deploy 7
- Support reverse pagination (previous page, has-previous-items) 7
- Docker container is no longer being pushed (it's stuck on 0.45) 7
- insert_all(..., alter=True) should work for new columns introduced after the first 100 records 7
- Push to Docker Hub failed - but it shouldn't run for alpha releases anyway 7
- Simplify imports of common classes 7
- SQLITE_MAX_VARS maybe hard-coded too low 7
- Commands for making authenticated API calls 7
- Pagination 7
- Support the dbstat table 7
- Much, much faster extract() implementation 7
- Documented HTML hooks for JavaScript plugin authors 7
- Redesign application homepage 7
- "Edit SQL" button on canned queries 7
- Fix last remaining links to "/" that do not respect base_url 7
- .extract() shouldn't extract null values 7
- export.xml file name varies with different language settings 7
- Documentation and unit tests for urls.row() urls.row_blob() methods 7
- "View all" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them 7
- changes to allow for compound foreign keys 7
- table.pks_and_rows_where() method returning primary keys along with the rows 7
- Invalid SQL: "no such table: pragma_database_list" on database page 7
- Latest Datasette tags missing from Docker Hub 7
- "More" link for facets that shows _facet_size=max results 7
- ?_nocol= does not interact well with default facets 7
- sqlite-utils memory should handle TSV and JSON in addition to CSV 7
- Introspection property for telling if a table is a rowid table 7
- Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions 7
- New pattern for async view classes 7
- Extra options to `lookup()` which get passed to `insert()` 7
- Columns starting with an underscore behave poorly in filters 7
- Test failure in test_rebuild_fts 7
- Support for CHECK constraints 7
- `.execute_write(... block=True)` should be the default behaviour 7
- Maybe let plugins define custom serve options? 7
- Link to stable docs from older versions 7
- Add SpatiaLite helpers to CLI 7
- Use dash encoding for table names and row primary keys in URLs 7
- I forgot to include the changelog in the 3.25.1 release 7
- Remove hashed URL mode 7
- Extract out `check_permissions()` from `BaseView 7
- `--nolock` feature for opening locked databases 7
- CLI eats my cursor 7
- truncate_cells_html does not work for links? 7
- progressbar for inserts/upserts of all fileformats, closes #485 7
- Ability to merge databases and tables 7
- Expose `sql` and `params` arguments to various plugin hooks 7
- Upgrade Datasette Docker to Python 3.11 7
- Figure out design for JSON errors (consider RFC 7807) 7
- Hacker News Datasette write demo 7
- First working version 7
- WIP new JSON for queries 7
- Proposal: Combine settings, metadata, static, etc. into a single `datasette.toml` File 7
- API tokens with view-table but not view-database/view-instance cannot access the table 7
- Addressable pages for every row in a table 6
- Default HTML/CSS needs to look reasonable and be responsive 6
- Support Django-style filters in querystring arguments 6
- Detect foreign keys and use them to link HTML pages together 6
- [WIP] Add publish to heroku support 6
- Nasty bug: last column not being correctly displayed 6
- Figure out how to bundle a more up-to-date SQLite 6
- Don't duplicate simple primary keys in the link column 6
- Load plugins from a `--plugins-dir=plugins/` directory 6
- Ability for plugins to define extra JavaScript and CSS 6
- inspect() should detect many-to-many relationships 6
- Deploy demo of Datasette on every commit that passes tests 6
- Plugin hook for loading metadata.json 6
- Faceted browse against a JSON list of tags 6
- CSV export in "Advanced export" pane doesn't respect query 6
- Additional Column Constraints? 6
- Rename metadata.json to config.json 6
- Easier way of creating custom row templates 6
- Experiment with type hints 6
- Command for running a search and saving tweets for that search 6
- Ways to improve fuzzy search speed on larger data sets? 6
- Improve UI of "datasette publish cloudrun" to reduce chances of accidentally over-writing a service 6
- Mechanism for indicating foreign key relationships in the table and query page URLs 6
- updating metadata.json without recreating the app 6
- Provide a cookiecutter template for creating new plugins 6
- upsert_all() throws issue when upserting to empty table 6
- "Templates considered" comment broken in >=0.35 6
- Documentation for the "request" object 6
- Support YAML in metadata - metadata.yaml 6
- Command for retrieving dependents for a repo 6
- Question: Access to immutable database-path 6
- Support decimal.Decimal type 6
- allow_by_query setting for configuring permissions with a SQL statement 6
- python tests/fixtures.py command has a bug 6
- Mechanism for specifying allow_sql permission in metadata.json 6
- Way to enable a default=False permission for anonymous users 6
- Ability to set ds_actor cookie such that it expires 6
- startup() plugin hook 6
- "Too many open files" error running tests 6
- datasette.add_message() doesn't work inside plugins 6
- Feature: pull request reviews and comments 6
- Datasette sdist is missing templates (hence broken when installing from Homebrew) 6
- End-user documentation 6
- extra_ plugin hooks should take the same arguments 6
- Mechanism for differentiating between "by me" and "liked by me" 6
- Progress bar for sqlite-utils insert 6
- Rendering glitch with column headings on mobile 6
- Change "--config foo:bar" to "--setting foo bar" 6
- Add Link: pagination HTTP headers 6
- Figure out how to display images from <en-media> tags inline in Datasette 6
- Method for datasette.client() to forward on authentication 6
- Fallback to databases in inspect-data.json when no -i options are passed 6
- Better display of binary data on arbitrary query results page 6
- Table actions menu on view pages, not on query pages 6
- load_template() plugin hook 6
- PrefixedUrlString mechanism broke everything 6
- Support order by relevance against FTS4 6
- Support linking to compound foreign keys 6
- Support for generated columns 6
- sqlite-utils analyze-tables command and table.analyze_column() method 6
- More flexible CORS support in core, to encourage good security practices 6
- Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions 6
- `sqlite-utils indexes` command 6
- Error: Use either --since or --since_id, not both 6
- `db.query()` method (renamed `db.execute_returning_dicts()`) 6
- "searchmode": "raw" in table metadata 6
- `table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option 6
- sqlite-utils insert errors should show SQL and parameters, if possible 6
- Mechanism to cause specific branches to deploy their own demos 6
- clean checkout & clean environment has test failures 6
- Win32 "used by another process" error with datasette publish 6
- ReadTheDocs build failed for 0.59.2 release 6
- Command for creating an empty database 6
- Idea: hover to reveal details of linked row 6
- Writable canned queries fail to load custom templates 6
- filters_from_request plugin hook, now used in TableView 6
- Release Datasette 0.60 6
- introduce new option for datasette package to use a slim base image 6
- Drop support for Python 3.6 6
- Support mutating row in `--convert` without returning it 6
- Reconsider policy on blocking queries containing the string "pragma" 6
- datasette one.db one.db opens database twice, as one and one_2 6
- `deterministic=True` fails on versions of SQLite prior to 3.8.3 6
- Ship Datasette 0.61 6
- Proposal: datasette query 6
- .db downloads should be served with an ETag 6
- db[table].create(..., transform=True) and create-table --transform 6
- Upgrade `--load-extension` to accept entrypoints like Datasette 6
- Ability to set a custom facet_size per table 6
- don't use immutable=1, only mode=ro 6
- Exclude virtual tables from datasette inspect 6
- Interactive demo of Datasette 1.0 write APIs 6
- register_permissions() plugin hook 6
- `datasette.create_token(...)` method for creating signed API tokens 6
- `publish cloudrun` reuses image tags, which can lead to very surprising deploy problems 6
- Folder support 6
- GitHub Action to lint Python code with ruff 6
- Try out Trogon for a tui interface 6
- Make as many examples in the CLI docs as possible copy-and-pastable 6
- Experiment with patterns for concurrent long running queries 5
- Create neat example database 5
- Redesign JSON output, ditch jsono, offer variants controlled by parameter instead 5
- Option to open readonly but not immutable 5
- datasette publish can fail if /tmp is on a different device 5
- Refactor views 5
- Add links to example Datasette instances to appropiate places in docs 5
- Ability to enable/disable specific features via --config 5
- Custom URL routing with independent tests 5
- datasette inspect takes a very long time on large dbs 5
- Get Datasette working with Zeit Now v2's 100MB image size limit 5
- Hashed URLs should be optional 5
- Plugin for allowing CORS from specified hosts 5
- Design changes to homepage to support mutable files 5
- Option to facet by date using month or year 5
- extra_template_vars plugin hook 5
- Ability to list views, and to access db["view_name"].rows / rows_where / etc 5
- Rethink progress bars for various commands 5
- [enhancement] Method to delete a row in python 5
- Testing utilities should be available to plugins 5
- If you have databases called foo.db and foo-bar.db you cannot visit /foo-bar 5
- Don't auto-format SQL on page load 5
- stargazers command, refs #4 5
- Add this view for seeing new releases 5
- Escape_fts5_query-hookimplementation does not work with queries to standard tables 5
- on_create mechanism for after table creation 5
- Datasette.render_template() method 5
- Rethink how sanity checks work 5
- Release automation: automate the bit that posts the GitHub release 5
- table.disable_fts() method and "sqlite-utils disable-fts ..." command 5
- twitter-to-sqlite user-timeline [screen_names] --sql / --attach 5
- Option in metadata.json to set default sort order for a table 5
- Feature: record history of follower counts 5
- Custom CSS class on body for styling canned queries 5
- Repos have a big blob of JSON in the organization column 5
- Annotate photos using the Google Cloud Vision API 5
- Create a public demo 5
- Unit test that checks that all plugin hooks have corresponding unit tests 5
- Ability to sign in to Datasette as a root account 5
- CSRF protection 5
- Consider using enable_callback_tracebacks(True) 5
- Fix the demo - it breaks because of the tags table change 5
- Mechanism for passing additional options to `datasette my.db` that affect plugins 5
- sqlite-utils insert: options for column types 5
- Features for enabling and disabling WAL mode 5
- Add homebrew installation to documentation 5
- 'datasette --get' option, refs #926 5
- Path parameters for custom pages 5
- Private/secret databases: database files that are only visible to plugins 5
- Handle case where subsequent records (after first batch) include extra columns 5
- Better handling of encodings other than utf-8 for "sqlite-utils insert" 5
- For 1.0 update trove classifier in setup.py 5
- How should datasette.client interact with base_url 5
- Add documentation on serving Datasette behind a proxy using base_url 5
- Add search highlighting snippets 5
- datasette.urls.static_plugins(...) method 5
- Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows 5
- Default menu links should check a real permission 5
- Rethink how table.search() method works 5
- Foreign key links break for compound foreign keys 5
- Rename datasette.config() method to datasette.setting() 5
- Show pysqlite3 version on /-/versions 5
- Update for Big Sur 5
- UNIQUE constraint failed: workouts.id 5
- Feature Request: Gmail 5
- Archive import appears to be broken on recent exports 5
- Release notes for Datasette 0.54 5
- 500 error caused by faceting if a column called `n` exists 5
- Share button for copying current URL 5
- Allow canned query params to specify default values 5
- Research using CTEs for faster facet counts 5
- Better default display of arrays of items 5
- Upgrade to Python 3.9.4 5
- Make row available to `render_cell` plugin hook 5
- Add Docker multi-arch support with Buildx 5
- ?_facet_size=X to increase number of facets results on the page 5
- `table.xindexes` using `PRAGMA index_xinfo(table)` 5
- DRAFT: A new plugin hook for dynamic metadata 5
- feature: support "events" 5
- Serve all db files in a folder 5
- .transform(types=) turns rowid into a concrete column 5
- Stop using generated columns in fixtures.db 5
- `datasette publish cloudrun --cpu X` option 5
- Ability to search for text across all columns in a table 5
- Ability to insert file contents as text, in addition to blob 5
- Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects') 5
- Allow routes to have extra options 5
- Way to test SQLite 3.37 (and potentially other versions) in CI 5
- Redesign CSV export to improve usability 5
- Add KNN and data_licenses to hidden tables list 5
- Move canned queries closer to the SQL input area 5
- Improvements to help make Datasette a better tool for learning SQL 5
- JSON link on row page is 404 if base_url setting is used 5
- Creating tables with custom datatypes 5
- Test failures with SQLite 3.37.0+ due to column affinity case 5
- Implement redirects from old % encoding to new dash encoding 5
- Adopt a code of conduct 5
- Display autodoc type information more legibly 5
- `sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher 5
- Datasette feature for publishing snapshots of query results 5
- Research running SQL in table view in parallel using `asyncio.gather()` 5
- Support `rows_where()`, `delete_where()` etc for attached alias databases 5
- CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool 5
- Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto' 5
- Upgrade to 3.10.6-slim-bullseye Docker base image 5
- minor a11y: <select> has no visual indicator when tabbed to 5
- 500 error in github-to-sqlite demo 5
- feature request: pivot command 5
- Link from documentation to source code 5
- Move "datasette --get" from Getting Started to CLI Reference 5
- Database() constructor currently defaults is_mutable to False 5
- `sqlite-utils transform` should set empty strings to null when converting text columns to integer/float 5
- NoneType' object has no attribute 'actor' 5
- Create a new table from one or more records, `sqlite-utils` style 5
- Design URLs for the write API 5
- Make it easier to fix URL proxy problems 5
- upsert of new row with check constraints fails 5
- ignore:true/replace:true options for /db/-/create API 5
- More useful error message if enable_load_extension is not available 5
- `Table.convert()` skips falsey values 5
- Expand foreign key references in row view as well 5
- codespell test failure 5
- Protect against malicious SQL that causes damage even though our DB is immutable 4
- Homepage UI for editing metadata file 4
- Switch to ujson 4
- Pick a name 4
- datasette publish hyper 4
- Support for title/source/license metadata 4
- Enforce pagination (or at least limits) for arbitrary custom SQL 4
- Add NHS England Hospitals example to wiki 4
- Consider data-package as a format for metadata 4
- add support for ?field__isnull=1 4
- Plugin that adds an authentication layer of some sort 4
- ?_json=foo&_json=bar query string argument 4
- A primary key column that has foreign key restriction associated won't rendering label column 4
- Support WITH query 4
- 500 from missing table name 4
- Ability to bundle metadata and templates inside the SQLite file 4
- Ability to apply sort on mobile in portrait mode 4
- metadata.json support for plugin configuration options 4
- Explore "distinct values for column" in inspect() 4
- Escaping named parameters in canned queries 4
- Mechanism for automatically picking up changes when on-disk .db file changes 4
- Add version number support with Versioneer 4
- Support table names ending with .json or .csv 4
- Explore if SquashFS can be used to shrink size of packaged Docker containers 4
- Limit text display in cells containing large amounts of text 4
- Datasette on Zeit Now returns http URLs for facet and next links 4
- Expose SANIC_RESPONSE_TIMEOUT config option in a sensible way 4
- Requesting support for query description 4
- render_cell(value) plugin hook 4
- Ability to display facet counts for many-to-many relationships 4
- Integration with JupyterLab 4
- add_column() should support REFERENCES {other_table}({other_column}) 4
- Figure out what to do about table counts in a mutable world 4
- Refactor facets to a class and new plugin, refs #427 4
- Tracing support for seeing what SQL queries were executed 4
- Paginate + search for databases/tables on the homepage 4
- Replace most of `.inspect()` (and `datasette inspect`) with table counting 4
- Decide what to do about /-/inspect 4
- Allow .insert(..., foreign_keys=()) to auto-detect table and primary key 4
- Facets not correctly persisted in hidden form fields 4
- Every datasette plugin on the ecosystem page should have a screenshot 4
- Support opening multiple databases with the same stem 4
- Enforce import sort order with isort 4
- Decide what goes into Datasette 1.0 4
- Fix static mounts using relative paths and prevent traversal exploits 4
- Get tests running on Windows using Travis CI 4
- Support unicode in url 4
- Too many SQL variables 4
- "Too many SQL variables" on large inserts 4
- More advanced connection pooling 4
- Option to fetch only checkins more recent than the current max checkin 4
- Add triggers while enabling FTS 4
- --sql and --attach options for feeding commands from SQL queries 4
- Use better pagination (and implement progress bar) 4
- Command to import home-timeline 4
- retweets-of-me command 4
- `import` command fails on empty files 4
- Failed to import workout points 4
- Datasette should work with Python 3.8 (and drop compatibility with Python 3.5) 4
- Mechanism for register_output_renderer to suggest extension or not 4
- Assets table with downloads 4
- Exception running first command: IndexError: list index out of range 4
- Allow creation of virtual tables at startup 4
- order_by mechanism 4
- Remove .detect_column_types() from table, make it a documented API 4
- Cashe-header missing in http-response 4
- Add documentation on Database introspection methods to internals.rst 4
- Adding a "recreate" flag to the `Database` constructor 4
- Custom pages mechanism, refs #648 4
- escape_fts() does not correctly escape * wildcards 4
- Fall back to authentication via ENV 4
- Directory configuration mode should support metadata.yaml 4
- Cloud Run fails to serve database files larger than 32MB 4
- [Feature Request] Support Repo Name in Search 🥺 4
- Ability to set custom default _size on a per-table basis 4
- Try out ExifReader 4
- add_foreign_key(...., ignore=True) 4
- register_output_renderer can_render mechanism 4
- Error pages not correctly loading CSS 4
- Publish secrets 4
- Example authentication plugin 4
- /-/metadata and so on should respect view-instance permission 4
- Log out mechanism for clearing ds_actor cookie 4
- Take advantage of .coverage being a SQLite database 4
- Skip counting hidden tables 4
- Use white-space: pre-wrap on ALL table cell contents 4
- github-to-sqlite tags command for fetching tags 4
- Output binary columns in "sqlite-utils query" JSON 4
- Security issue: read-only canned queries leak CSRF token in URL 4
- Test failures caused by failed attempts to mock pip 4
- --load-extension option for sqlite-utils query 4
- github-to-sqlite should handle rate limits better 4
- request an "-o" option on "datasette server" to open the default browser at the running url 4
- Idea: conversions= could take Python functions 4
- sqlite-utils transform sub-command 4
- sqlite-utils transform/insert --detect-types 4
- from_json jinja2 filter 4
- column name links broken in 0.50.1 4
- extra_js_urls and extra_css_urls should respect base_url setting 4
- Some workout columns should be float, not text 4
- Include LICENSE in sdist 4
- Add template block prior to extra URL loaders 4
- .blob output renderer 4
- Table/database action menu cut off if too short 4
- Rebrand and redirect config.rst as settings.rst 4
- --load-extension=spatialite not working with datasetteproject/datasette docker image 4
- Fix footer not sticking to bottom in short pages 4
- "_searchmode=raw" throws an index out of range error when combined with "_search_COLUMN" 4
- sqlite-utils should suggest --csv if JSON parsing fails 4
- sqlite-utils analyze-tables command 4
- Searching for "github-to-sqlite" throws an error 4
- Modernize code to Python 3.6+ 4
- Prettier package not actually being cached 4
- reset_counts() method and command 4
- Use structlog for logging 4
- Certain database names results in 404: "Database not found: None" 4
- view_name = "query" for the query page 4
- Tests are very slow. 4
- Possible to deploy as a python app (for Rstudio connect server)? 4
- photo-to-sqlite: command not found 4
- Installing datasette via docker: Path 'fixtures.db' does not exist 4
- Support SSL/TLS directly 4
- Error reading csv files with large column data 4
- --port option should validate port is between 0 and 65535 4
- Escaping FTS search strings 4
- Refresh SpatiaLite documentation 4
- Feature or Documentation Request: Individual table as home page template 4
- Dockerfile: use Ubuntu 20.10 as base 4
- improve table horizontal scroll experience 4
- Document how to send multiple values for "Named parameters" 4
- Avoid error sorting by relationships if related tables are not allowed 4
- Can't use apt-get in Dockerfile when using datasetteproj/datasette as base 4
- Figure out how to publish alpha/beta releases to Docker Hub 4
- Intermittent CI failure: restore_working_directory FileNotFoundError 4
- row.update() or row.pk 4
- db.schema property and sqlite-utils schema command 4
- Cannot set type JSON 4
- Automatic type detection for CSV data 4
- Big performance boost on faceting: skip the inner order by 4
- Command for fetching Hacker News threads from the search API 4
- Add Gmail takeout mbox import (v2) 4
- Ability to default to hiding the SQL for a canned query 4
- Document exceptions that can be raised by db.execute() and friends 4
- Add reference documentation generated from docstrings 4
- xml.etree.ElementTree.ParseError: not well-formed (invalid token) 4
- sqlite-utils memory can't deal with multiple files with the same name 4
- ?_sort=rowid with _next= returns error 4
- `table.lookup()` option to populate additional columns when creating a record 4
- Improve Apache proxy documentation, link to demo 4
- Provide function to generate hash_id from specified columns 4
- Use datasette-table Web Component to guide the design of the JSON API for 1.0 4
- Add `Link: rel="alternate"` header pointing to JSON for a table/query 4
- Maybe return JSON from HTML pages if `Accept: application/json` is sent 4
- `sqlite-utils insert --extract colname` 4
- Allow users to pass a full convert() function definition 4
- Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1 4
- Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key 4
- Better error message if `--convert` code fails to return a dict 4
- `--fmt` should imply `-t` 4
- Add documentation page with the output of `--help` 4
- Release notes for 0.60 4
- `sqlite-utils bulk --batch-size` option 4
- Document how to add a primary key to a rowid table using `sqlite-utils transform --pk` 4
- Update Dockerfile generated by `datasette publish` 4
- Sensible `cache-control` headers for static assets, including those served by plugins 4
- [feature] immutable mode for a directory, not just individual sqlite file 4
- Automated test for Pyodide compatibility 4
- ?_trace=1 fails with datasette-geojson for some reason 4
- Combining `rows_where()` and `search()` to limit which rows are searched 4
- Utilities for duplicating tables and creating a table with the results of a query 4
- 500 error if sorted by a column not in the ?_col= list 4
- Cross-link CLI to Python docs 4
- Adjust height of textarea for no JS case 4
- Research an upgrade to CodeMirror 6 4
- search_sql add include_rank option 4
- Parts of YAML file do not work when db name is "off" 4
- fails before generating views. ERR: table sqlite_master may not be modified 4
- Featured table(s) on the homepage 4
- Ability to insert multi-line files 4
- Setting to turn off table row counts entirely 4
- devrel/python api: Pylance type hinting 4
- Turn --flatten into a documented utility function 4
- Tests failing due to updated tabulate library 4
- `max_signed_tokens_ttl` setting for a maximum duration on API tokens 4
- Delete a single record from an existing table 4
- API to drop a table 4
- Datasette with many and large databases > Memory use 4
- 1.0a0 release notes 4
- Extract logic for resolving a URL to a database / table / row 4
- Clicking within the CodeMirror area below the SQL (i.e. when there's only a single line) doesn't cause the editor to get focused 4
- `publish heroku` failing due to old Python version 4
- Docs for replace:true and ignore:true options for insert API 4
- Incorrect link from the API explorer to the JSON API documentation 4
- Feature request: output number of ignored/replaced rows for insert command 4
- render_cell plugin hook's row object is not a sqlite.Row 4
- installpython3.com is now a spam website 4
- Reconsider pattern where plugins could break existing template context 4
- Datasette is not compatible with SQLite's strict quoting compilation option 4
- Repeated calls to `Table.convert()` fail 4
- How to redirect from "/" to a specific db/table 4
- Add paths for homebrew on Apple silicon 4
- Custom SQL queries should use new JSON ?_extra= format 4
- Datasette cannot be installed with Rye 4
- `--raw-lines` option, like `--raw` for multiple lines 4
- sphinx.builders.linkcheck build error 4
- Implement sensible query pagination 3
- Command line tool for uploading one or more DBs to Now 3
- Ability to plot a simple graph 3
- date, year, month and day querystring lookups 3
- Implement a better database index page 3
- Add more detailed API documentation to the README 3
- UI for editing named parameters 3
- Link to JSON for the list of tables 3
- UI support for running FTS searches 3
- If view is filtered, search should apply within those filtered rows 3
- ?_search=x should work if used directly against a FTS virtual table 3
- Show extra instructions with the interrupted 3
- apsw as alternative sqlite3 binding (for full text search) 3
- _group_count= feature improvements 3
- Datasette CSS should include content hash in the URL 3
- datasette skeleton command for kick-starting database and table metadata 3
- Custom template for named canned query 3
- proposal new option to disable user agents cache 3
- Cleaner mechanism for handling custom errors 3
- Run pks_for_table in inspect, executing once at build time rather than constantly 3
- Hide Spatialite system tables 3
- Support filtering with units and more 3
- Allow plugins to add new cli sub commands 3
- datasette publish --install=name-of-plugin 3
- label_column option in metadata.json 3
- External metadata.json 3
- Add new metadata key persistent_urls which removes the hash from all database urls 3
- Facets should not execute for ?shape=array|object 3
- Documentation for URL hashing, redirects and cache policy 3
- Build smallest possible Docker image with Datasette plus recent SQLite (with json1) plus Spatialite 4.4.0 3
- Support multiple filters of the same type 3
- ?_ttl= parameter to control caching 3
- Avoid plugins accidentally loading dependencies twice 3
- Per-database and per-table /-/ URL namespace 3
- Ability to configure SQLite cache_size 3
- Installation instructions, including how to use the docker image 3
- Ensure --help examples in docs are always up to date 3
- Use pysqlite3 if available 3
- datasette publish digitalocean plugin 3
- Update official datasetteproject/datasette Docker container to SQLite 3.26.0 3
- Ensure downloading a 100+MB SQLite database file works 3
- How to pass configuration to plugins? 3
- Use SQLITE_DBCONFIG_DEFENSIVE plus other recommendations from SQLite security docs 3
- Experiment: run Jinja in async mode 3
- .insert_all() should accept a generator and process it efficiently 3
- Problems handling column names containing spaces or - 3
- Zeit API v1 does not work for new users - need to migrate to v2 3
- Utilities for adding indexes 3
- Add query parameter to hide SQL textarea 3
- Datasette doesn't reload when database file changes 3
- Installing installs the tests package 3
- Fix the "datasette now publish ... --alias=x" option 3
- Make it so Docker build doesn't delay PyPI release 3
- Option to ignore inserts if primary key exists already 3
- Accessibility for non-techie newsies? 3
- Test against Python 3.8-dev using Travis 3
- Exporting sqlite database(s)? 3
- "about" parameter in metadata does not appear when alone 3
- asgi_wrapper plugin hook 3
- Unable to use rank when fts-table generated with csvs-to-sqlite 3
- Mechanism for secrets in plugin configuration 3
- datasette publish option for setting plugin configuration secrets 3
- Potential improvements to facet-by-date 3
- CodeMirror fails to load on database page 3
- .add_column() doesn't match indentation of initial creation 3
- extracts= option for insert/update/etc 3
- Script uses a lot of RAM 3
- Datasette Edit 3
- "twitter-to-sqlite user-timeline" command for pulling tweets by a specific user 3
- Exposing Datasette via Jupyter-server-proxy 3
- Added support for multi arch builds 3
- Extract "source" into a separate lookup table 3
- Track and use the 'since' value 3
- Queries per DB table in metadata.json 3
- Handle spaces in DB names 3
- since_id support for home-timeline 3
- make uvicorn optional dependancy (because not ok on windows python yet) 3
- --since support for various commands for refresh-by-cron 3
- upgrade to uvicorn-0.9 to be Python-3.8 friendly 3
- Offer to format readonly SQL 3
- _where= parameter is not persisted in hidden form fields 3
- /-/plugins shows incorrect name for plugins 3
- Static assets no longer loading for installed plugins 3
- Add this repos_starred view 3
- Publish to Heroku is broken: "WARNING: You must pass the application as an import string to enable 'reload' or 'workers" 3
- rowid is not included in dropdown filter menus 3
- Custom queries with 0 results should say "0 results" 3
- Don't suggest column for faceting if all values are 1 3
- Command for importing events 3
- Make database level information from metadata.json available in the index.html template 3
- Feature request: enable extensions loading 3
- Add a glossary to the documentation 3
- fts5 syntax error when using punctuation 3
- Template debug mode that outputs template context 3
- Copy and paste doesn't work reliably on iPhone for SQL editor 3
- Tests are failing due to missing FTS5 3
- Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column 3
- --port option to expose a port other than 8001 in "datasette package" 3
- Tutorial command no longer works 3
- Use inspect-file, if possible, for total row count 3
- prepare_connection() plugin hook should accept optional datasette argument 3
- Ability to customize columns used by extracts= feature 3
- Variables from extra_template_vars() not exposed in _context=1 3
- Search box CSS doesn't look great on OS X Safari 3
- Handle "User not found" error 3
- WIP implementation of writable canned queries 3
- --plugin-secret over-rides existing metadata.json plugin config 3
- Update aiofiles requirement from ~=0.4.0 to >=0.4,<0.6 3
- Pull repository contributors 3
- Mechanism for forcing column-type, over-riding auto-detection 3
- Issue and milestone should have foreign key to repo 3
- Issue comments don't appear to populate issues foreign key 3
- strange behavior using accented characters 3
- Configuration directory mode 3
- Create index on issue_comments(user) and other foreign keys 3
- Mechanism for creating views if they don't yet exist 3
- Add notlike table filter 3
- Question: Any fixed date for the release with the uft8-encoding fix? 3
- fts search on a column doesn't work anymore due to escape_fts 3
- Only install osxphotos if running on macOS 3
- Way of seeing full schema for a database 3
- Add PyPI project urls to setup.py 3
- request.url and request.scheme should obey force_https_urls config setting 3
- CSRF protection for /-/messages tool and writable canned queries 3
- Documentation for new "params" setting for canned queries 3
- Ability to customize what happens when a view permission fails 3
- Documentation is inconsistent about "id" as required field on actor 3
- Document the ds_actor signed cookie 3
- Horizontal scrollbar on changelog page on mobile 3
- Redesign register_facet_classes plugin hook 3
- Fall back to FTS4 if FTS5 is not available 3
- Consider pagination of canned queries 3
- Script to generate larger SQLite test files 3
- Support for compound (composite) foreign keys 3
- initial windows ci setup 3
- "Logged in as: XXX - logout" navigation item 3
- Canned query page should show the name of the canned query 3
- asgi_wrapper plugin hook is crashing at startup 3
- Ability to remove a foreign key 3
- Improved (and better documented) support for transactions 3
- Some links don't honor base_url 3
- Add a table of contents to the README 3
- "allow": true for anyone, "allow": false for nobody 3
- Interactive debugging tool for "allow" blocks 3
- Ability to insert files piped to insert-files stdin 3
- Support tokenize option for FTS 3
- Refactor TableView class so things like datasette-graphql can reuse the logic 3
- Travis should not build the master branch, only the main branch 3
- "datasette install" and "datasette uninstall" commands 3
- db.execute_write_fn(create_tables, block=True) hangs a thread if connection fails 3
- Pass columns to extra CSS/JS/etc plugin hooks 3
- Code for finding SpatiaLite in the usual locations 3
- --load-extension=spatialite shortcut option 3
- Try out CodeMirror SQL hints 3
- insert_all(..., alter=True) should work for new columns introduced after the first 100 records 3
- Datasette plugin to provide custom page for running faceted, ranked searches 3
- Timeline view 3
- table.optimize() should delete junk rows from *_fts_docsize 3
- Documentation for 404.html, 500.html templates 3
- …
id | html_url | issue_url | node_id ▼ | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
1053973425 | https://github.com/simonw/datasette/issues/1439#issuecomment-1053973425 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-0lux | simonw 9599 | 2022-02-28T07:40:12Z | 2022-02-28T07:40:12Z | OWNER | If I make this change it will break existing links to one of the oldest Datasette demos: http://fivethirtyeight.datasettes.com/fivethirtyeight/avengers%2Favengers A plugin that fixes those by redirecting them on 404 would be neat. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1043609198 | https://github.com/simonw/datasette/issues/327#issuecomment-1043609198 | https://api.github.com/repos/simonw/datasette/issues/327 | IC_kwDOBm6k_c4-NDZu | dholth 208018 | 2022-02-17T23:21:36Z | 2022-02-17T23:33:01Z | NONE | On fly.io. This particular database goes from 1.4GB to 200M. Slower, part of that might be having no `--inspect-file`? ``` $ datasette publish fly ... --generate-dir /tmp/deploy-this ... $ mksquashfs large.db large.squashfs $ rm large.db # don't accidentally put it in the image $ cat Dockerfile FROM python:3.8 COPY . /app WORKDIR /app ENV DATASETTE_SECRET 'xyzzy' RUN pip install -U datasette # RUN datasette inspect large.db --inspect-file inspect-data.json ENV PORT 8080 EXPOSE 8080 CMD mount -o loop -t squashfs large.squashfs /mnt; datasette serve --host 0.0.0.0 -i /mnt/large.db --cors --port $PORT ``` It would also be possible to copy the file onto the ~6GB available on the ephemeral container filesystem on startup. A little against the spirit of the thing? On this example the whole docker image is 2.42 GB and the squashfs version is 1.14 GB. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Explore if SquashFS can be used to shrink size of packaged Docker containers 335200136 | |
1043626870 | https://github.com/simonw/datasette/issues/327#issuecomment-1043626870 | https://api.github.com/repos/simonw/datasette/issues/327 | IC_kwDOBm6k_c4-NHt2 | dholth 208018 | 2022-02-17T23:37:24Z | 2022-02-17T23:37:24Z | NONE | On second thought any kind of quick-to-decompress-on-startup could be helpful if we're paying for the container registry and deployment bandwidth but not ephemeral storage. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Explore if SquashFS can be used to shrink size of packaged Docker containers 335200136 | |
1045131086 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045131086 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-S29O | simonw 9599 | 2022-02-18T20:22:13Z | 2022-02-18T20:22:47Z | OWNER | Should it encode `%` symbols too, since they have a special meaning in URLs and we can't guarantee that every single web server / proxy out there will round-trip them safely using percentage encoding? If so, would need to pick a different encoding character for them. Maybe `%` becomes `-p` - and in that case `/` could become `-s` too. Is it worth expanding dash-encoding outside of just `/` and `-` and `.` though? Not sure. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045134050 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045134050 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-S3ri | simonw 9599 | 2022-02-18T20:25:04Z | 2022-02-18T20:25:04Z | OWNER | Here's a useful modern spec for how existing URL percentage encoding is supposed to work: https://url.spec.whatwg.org/#percent-encoded-bytes | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045024276 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045024276 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-Sc4U | simonw 9599 | 2022-02-18T19:01:42Z | 2022-02-18T19:55:24Z | OWNER | > Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly. ```python def dash_encode(s): return s.replace("-", "--").replace(".", "-.").replace("/", "-/") def dash_decode(s): return s.replace("-/", "/").replace("-.", ".").replace("--", "-") ``` ```pycon >>> dash_encode("foo/bar/baz.csv") 'foo-/bar-/baz-.csv' >>> dash_decode('foo-/bar-/baz-.csv') 'foo/bar/baz.csv' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045027067 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045027067 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-Sdj7 | simonw 9599 | 2022-02-18T19:03:26Z | 2022-02-18T19:03:26Z | OWNER | (If I make this change it may break some existing Datasette installations when they upgrade - I could try and build a plugin for them which triggers on 404s and checks to see if the old format would return a 200 response, then returns that.) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045032377 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045032377 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-Se25 | simonw 9599 | 2022-02-18T19:06:50Z | 2022-02-18T19:06:50Z | OWNER | How does URL routing for https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv work? Right now it's https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/app.py#L1098-L1101 That's not going to capture the dot-dash encoding version of that table name: ```pycon >>> dot_dash_encode("table/with/slashes.csv") 'table-/with-/slashes-.csv' ``` Probably needs a fancy regex trick like a negative lookbehind assertion or similar. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045055772 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045055772 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-Skkc | simonw 9599 | 2022-02-18T19:23:33Z | 2022-02-18T19:25:42Z | OWNER | I want a match for this URL: /db/table-/with-/slashes-.csv Maybe this: ^/(?P<db_name>[^/]+)/(?P<table_and_format>([^/]*|(\-/)*|(\-\.)*|(\.\.)*)*$) Here we are matching a sequence of: ([^/]*|(\-/)*|(\-\.)*|(\-\-)*)* So a combination of not-slashes OR -/ or -. Or -- sequences <img width="224" alt="image" src="https://user-images.githubusercontent.com/9599/154748362-84909d4e-dccf-454b-a9cd-a036f9f66f09.png"> ^/(?P<db_name>[^/]+)/(?P<table_and_format>([^/]*|(\-/)*|(\-\.)*|(\-\-)*)*$) Try that with non-capturing bits: ^/(?P<db_name>[^/]+)/(?P<table_and_format>(?:[^/]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*$) `(?:[^/]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*` visualized is: <img width="193" alt="image" src="https://user-images.githubusercontent.com/9599/154748441-decea502-0d04-44f4-9ca9-fb6883767833.png"> Here's the explanation on regex101.com https://regex101.com/r/CPnsIO/1 <img width="1074" alt="image" src="https://user-images.githubusercontent.com/9599/154748720-cdda61db-5498-49a8-91c2-e726b394fa49.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045059427 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045059427 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-Sldj | simonw 9599 | 2022-02-18T19:26:25Z | 2022-02-18T19:26:25Z | OWNER | With this new pattern I could probably extract out the optional `.json` format string as part of the initial route capturing regex too, rather than the current `table_and_format` hack. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045069481 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-Sn6p | simonw 9599 | 2022-02-18T19:34:41Z | 2022-03-05T21:32:22Z | OWNER | I think I got format extraction working! https://regex101.com/r/A0bW1D/1 ^/(?P<database>[^/]+)/(?P<table>(?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*?)(?:(?<!\-)\.(?P<format>\w+))?$ I had to make that crazy inner one even more complicated to stop it from capturing `.` that was not part of `-.`. (?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)* Visualized: <img width="222" alt="image" src="https://user-images.githubusercontent.com/9599/154749714-44579899-5dc7-4e5f-ad4f-dc59dac48979.png"> So now I have a regex which can extract out the dot-encoded table name AND spot if there is an optional `.format` at the end: <img width="1090" alt="image" src="https://user-images.githubusercontent.com/9599/156900484-7912073f-28aa-4301-86e2-e5cbe625e1d5.png"> If I end up using this in Datasette it's going to need VERY comprehensive unit tests and inline documentation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045077590 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045077590 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-Sp5W | simonw 9599 | 2022-02-18T19:41:37Z | 2022-02-18T19:42:41Z | OWNER | Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where "system" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence. And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too. Maybe change this system to use `.` as the escaping character instead of `-`? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045075207 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045075207 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-SpUH | simonw 9599 | 2022-02-18T19:39:35Z | 2022-02-18T19:40:13Z | OWNER | > And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this: > > * `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version > * `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version > * `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version Here's what those look like with the updated version of `dot_dash_encode()` that also encodes `/` as `-/`: - `/db/-/db-/table---.csv-.csv` - HTML - `/db/-/db-/table---.csv-.csv.csv` - CSV - `/db/-/db-/table---.csv-.csv.json` - JSON <img width="1050" alt="image" src="https://user-images.githubusercontent.com/9599/154750631-a8a23c62-3dfc-43e4-8026-4d117dc4bf8d.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045081042 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045081042 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-SqvS | simonw 9599 | 2022-02-18T19:44:12Z | 2022-02-18T19:51:34Z | OWNER | ```python def dot_encode(s): return s.replace(".", "..").replace("/", "./") def dot_decode(s): return s.replace("./", "/").replace("..", ".") ``` No need for hyphen encoding in this variant at all, which simplifies things a bit. (Update: this is flawed, see https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045086033 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-Sr9R | simonw 9599 | 2022-02-18T19:47:43Z | 2022-02-18T19:51:11Z | OWNER | - https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv - https://til.simonwillison.net/-/asgi-scope/db/./db./table-..csv..csv Do both of those survive the round-trip to populate `raw_path` correctly? No! In both cases the `/./` bit goes missing. It looks like this might even be a client issue - `curl` shows me this: ``` ~ % curl -vv -i 'https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv' * Trying 216.239.32.21:443... * Connected to datasette.io (216.239.32.21) port 443 (#0) * ALPN, offering http/1.1 * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: datasette.io * Server certificate: R3 * Server certificate: ISRG Root X1 > GET /-/asgi-scope/db/db./table-..csv..csv HTTP/1.1 ``` So `curl` decided to turn `/-/asgi-scope/db/./db./table` into `/-/asgi-scope/db/db./table` before even sending the request. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045082891 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045082891 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-SrML | simonw 9599 | 2022-02-18T19:45:32Z | 2022-02-18T19:45:32Z | OWNER | ```pycon >>> dot_encode("/db/table-.csv.csv") './db./table-..csv..csv' >>> dot_decode('./db./table-..csv..csv') '/db/table-.csv.csv' ``` I worry that web servers might treat `./` in a special way though. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045095348 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045095348 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-SuO0 | simonw 9599 | 2022-02-18T19:53:48Z | 2022-02-18T19:53:48Z | OWNER | > Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where "system" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence. > > And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too. I don't think this matters. The new regex does indeed capture that kind of page: <img width="1052" alt="image" src="https://user-images.githubusercontent.com/9599/154752309-e1787755-3bdb-47c2-867c-7ac5fe65664d.png"> But Datasette goes through configured route regular expressions in order - so I can have the regex that captures `/db/-/special` routes listed before the one that captures tables and formats. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045099290 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045099290 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-SvMa | simonw 9599 | 2022-02-18T19:56:18Z | 2022-02-18T19:56:30Z | OWNER | > ```python > def dash_encode(s): > return s.replace("-", "--").replace(".", "-.").replace("/", "-/") > > def dash_decode(s): > return s.replace("-/", "/").replace("-.", ".").replace("--", "-") > ``` I think **dash-encoding** (new name for this) is the right way forward here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045108611 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045108611 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-SxeD | simonw 9599 | 2022-02-18T20:02:19Z | 2022-02-18T20:08:34Z | OWNER | One other potential variant: ```python def dash_encode(s): return s.replace("-", "-dash-").replace(".", "-dot-").replace("/", "-slash-") def dash_decode(s): return s.replace("-slash-", "/").replace("-dot-", ".").replace("-dash-", "-") ``` Except this has bugs - it doesn't round-trip safely, because it can get confused about things like `-dash-slash-` in terms of is that a `-dash-` or a `-slash-`? ```pycon >>> dash_encode("/db/table-.csv.csv") '-slash-db-slash-table-dash--dot-csv-dot-csv' >>> dash_decode('-slash-db-slash-table-dash--dot-csv-dot-csv') '/db/table-.csv.csv' >>> dash_encode('-slash-db-slash-table-dash--dot-csv-dot-csv') '-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv' >>> dash_decode('-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv') '-dash/dash-db-dash/dash-table-dash--dash.dash-csv-dash.dash-csv' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045111309 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045111309 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-SyIN | simonw 9599 | 2022-02-18T20:04:24Z | 2022-02-18T20:05:40Z | OWNER | This made me worry that my current `dash_decode()` implementation had unknown round-trip bugs, but thankfully this works OK: ```pycon >>> dash_encode("/db/table-.csv.csv") '-/db-/table---.csv-.csv' >>> dash_encode('-/db-/table---.csv-.csv') '---/db---/table-------.csv---.csv' >>> dash_decode('---/db---/table-------.csv---.csv') '-/db-/table---.csv-.csv' >>> dash_decode('-/db-/table---.csv-.csv') '/db/table-.csv.csv' ``` The regex still works against that double-encoded example too: <img width="1032" alt="image" src="https://user-images.githubusercontent.com/9599/154753916-b7d2159e-4284-4c92-ae61-110671fa320e.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045117304 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045117304 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-Szl4 | simonw 9599 | 2022-02-18T20:09:22Z | 2022-02-18T20:09:22Z | OWNER | Adopting this could result in supporting database files with surprising characters in their filename too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1045269544 | https://github.com/simonw/datasette/issues/1439#issuecomment-1045269544 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-TYwo | simonw 9599 | 2022-02-18T22:19:29Z | 2022-02-18T22:19:29Z | OWNER | Note that I've ruled out using `Accept: application/json` to return JSON because it turns out Cloudflare and potentially other CDNs ignore the `Vary: Accept` header entirely: - https://github.com/simonw/datasette/issues/1534 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1049114724 | https://github.com/simonw/datasette/issues/1439#issuecomment-1049114724 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-iDhk | simonw 9599 | 2022-02-23T19:04:40Z | 2022-02-23T19:04:40Z | OWNER | I'm going to try dash encoding for table names (and row IDs) in a branch and see how I like it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1049124390 | https://github.com/simonw/datasette/issues/1439#issuecomment-1049124390 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-iF4m | simonw 9599 | 2022-02-23T19:15:00Z | 2022-02-23T19:15:00Z | OWNER | I'll start by modifying this function: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/utils/__init__.py#L732-L749 Later I want to move this to the routing layer to split out `format` automatically, as seen in the regexes here: https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1049126151 | https://github.com/simonw/datasette/issues/1439#issuecomment-1049126151 | https://api.github.com/repos/simonw/datasette/issues/1439 | IC_kwDOBm6k_c4-iGUH | simonw 9599 | 2022-02-23T19:17:01Z | 2022-02-23T19:17:01Z | OWNER | Actually the relevant code looks to be: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/views/base.py#L481-L498 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047 | |
1049879118 | https://github.com/simonw/datasette/issues/1641#issuecomment-1049879118 | https://api.github.com/repos/simonw/datasette/issues/1641 | IC_kwDOBm6k_c4-k-JO | fgregg 536941 | 2022-02-24T13:49:26Z | 2022-02-24T13:49:26Z | CONTRIBUTOR | maybe worth considering adding buttons for paren, asterisk, etc. under the input text box on mobile? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tweak mobile keyboard settings 1149310456 | |
1051473892 | https://github.com/simonw/datasette/issues/260#issuecomment-1051473892 | https://api.github.com/repos/simonw/datasette/issues/260 | IC_kwDOBm6k_c4-rDfk | zaneselvans 596279 | 2022-02-26T02:24:15Z | 2022-02-26T02:24:15Z | NONE | Is there already functionality that can be used to validate the `metadata.json` file? Is there a JSON Schema that defines it? Or a validation that's available via datasette with Python? We're working on [automatically building the metadata](https://github.com/catalyst-cooperative/pudl/pull/1479) in CI and when we deploy to cloud run, and it would be nice to be able to check whether the the metadata we're outputting is valid in our tests. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Validate metadata.json on startup 323223872 | |
888694144 | https://github.com/simonw/datasette/issues/1405#issuecomment-888694144 | https://api.github.com/repos/simonw/datasette/issues/1405 | IC_kwDOBm6k_c40-GWA | simonw 9599 | 2021-07-28T23:51:59Z | 2021-07-28T23:51:59Z | OWNER | https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/datasette/utils/__init__.py#L918-L926 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | utils.parse_metadata() should be a documented internal function 955316250 | |
888694261 | https://github.com/simonw/datasette/issues/1405#issuecomment-888694261 | https://api.github.com/repos/simonw/datasette/issues/1405 | IC_kwDOBm6k_c40-GX1 | simonw 9599 | 2021-07-28T23:52:21Z | 2021-07-28T23:52:21Z | OWNER | Document that it can raise a `BadMetadataError` exception. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | utils.parse_metadata() should be a documented internal function 955316250 | |
886968648 | https://github.com/simonw/datasette/issues/1402#issuecomment-886968648 | https://api.github.com/repos/simonw/datasette/issues/1402 | IC_kwDOBm6k_c403hFI | simonw 9599 | 2021-07-26T19:30:14Z | 2021-07-26T19:30:14Z | OWNER | I really like this idea. I was thinking it might make a good plugin, but there's not a great mechanism for plugins to inject extra `<head>` content at the moment - plus this actually feels like a reasonable feature for Datasette core itself. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | feature request: social meta tags 951185411 | |
886969541 | https://github.com/simonw/datasette/issues/1402#issuecomment-886969541 | https://api.github.com/repos/simonw/datasette/issues/1402 | IC_kwDOBm6k_c403hTF | simonw 9599 | 2021-07-26T19:31:40Z | 2021-07-26T19:31:40Z | OWNER | Datasette could do a pretty good job of this by default, using `twitter:card` and `og:url` tags - like on https://til.simonwillison.net/jq/extracting-objects-recursively I could also provide a mechanism to customize these - in particular to add images of some sort. It feels like something that should tie in to the metadata mechanism. | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0} | feature request: social meta tags 951185411 | |
887095569 | https://github.com/simonw/datasette/issues/1404#issuecomment-887095569 | https://api.github.com/repos/simonw/datasette/issues/1404 | IC_kwDOBm6k_c404AER | simonw 9599 | 2021-07-26T23:27:07Z | 2021-07-26T23:27:07Z | OWNER | Updated documentation: https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/docs/plugin_hooks.rst#register-routes-datasette | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `register_routes()` hook should take `datasette` argument 953352015 | |
881125124 | https://github.com/simonw/datasette/issues/759#issuecomment-881125124 | https://api.github.com/repos/simonw/datasette/issues/759 | IC_kwDOBm6k_c40hOcE | simonw 9599 | 2021-07-16T02:11:48Z | 2021-07-16T02:11:54Z | OWNER | I added `"searchmode": "raw"` as a supported option for table metadata in #1389 and released that in Datasette 0.58. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | fts search on a column doesn't work anymore due to escape_fts 612673948 | |
881129149 | https://github.com/simonw/datasette/issues/1394#issuecomment-881129149 | https://api.github.com/repos/simonw/datasette/issues/1394 | IC_kwDOBm6k_c40hPa9 | simonw 9599 | 2021-07-16T02:23:32Z | 2021-07-16T02:23:32Z | OWNER | Wrote about this in the annotated release notes for 0.58: https://simonwillison.net/2021/Jul/16/datasette-058/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Big performance boost on faceting: skip the inner order by 944870799 | |
881204782 | https://github.com/simonw/datasette/issues/1231#issuecomment-881204782 | https://api.github.com/repos/simonw/datasette/issues/1231 | IC_kwDOBm6k_c40hh4u | simonw 9599 | 2021-07-16T06:14:12Z | 2021-07-16T06:14:12Z | OWNER | Here's the traceback I got from `datasette-graphql` (annoyingly only running the tests in GitHub Actions CI - I've not been able to replicate on my laptop yet): ``` tests/test_utils.py . [100%] =================================== FAILURES =================================== _________________________ test_graphql_examples[path0] _________________________ ds = <datasette.app.Datasette object at 0x7f6b8b6f8fd0> path = PosixPath('/home/runner/work/datasette-graphql/datasette-graphql/examples/filters.md') @pytest.mark.asyncio @pytest.mark.parametrize( "path", (pathlib.Path(__file__).parent.parent / "examples").glob("*.md") ) async def test_graphql_examples(ds, path): content = path.read_text() query = graphql_re.search(content)[1] try: variables = variables_re.search(content)[1] except TypeError: variables = "{}" expected = json.loads(json_re.search(content)[1]) response = await ds.client.post( "/graphql", json={ "query": query, "variables": json.loads(variables), }, ) > assert response.status_code == 200, response.json() E AssertionError: {'data': {'repos_arraycontains': None, 'users_contains': None, 'users_date': None, 'users_endswith': None, ...}, 'erro...", 'path': ['users_gt']}, {'locations': [{'column': 5, 'line': 34}], 'message': "'rows'", 'path': ['users_gte']}, ...]} E assert 500 == 200 E + where 500 = <Response [500 Internal Server Error]>.status_code tests/test_graphql.py:142: AssertionError ----------------------------- Captured stderr call ----------------------------- table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists … | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Race condition errors in new refresh_schemas() mechanism 811367257 | |
881204343 | https://github.com/simonw/datasette/issues/1231#issuecomment-881204343 | https://api.github.com/repos/simonw/datasette/issues/1231 | IC_kwDOBm6k_c40hhx3 | simonw 9599 | 2021-07-16T06:13:11Z | 2021-07-16T06:13:11Z | OWNER | This just broke the `datasette-graphql` test suite: https://github.com/simonw/datasette-graphql/issues/77 - I need to figure out a solution here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Race condition errors in new refresh_schemas() mechanism 811367257 | |
881663968 | https://github.com/simonw/datasette/issues/1231#issuecomment-881663968 | https://api.github.com/repos/simonw/datasette/issues/1231 | IC_kwDOBm6k_c40jR_g | simonw 9599 | 2021-07-16T19:18:42Z | 2021-07-16T19:18:42Z | OWNER | The race condition happens inside this method - initially with the call to `await init_internal_db()`: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/app.py#L334-L359 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Race condition errors in new refresh_schemas() mechanism 811367257 | |
881664408 | https://github.com/simonw/datasette/issues/1231#issuecomment-881664408 | https://api.github.com/repos/simonw/datasette/issues/1231 | IC_kwDOBm6k_c40jSGY | simonw 9599 | 2021-07-16T19:19:35Z | 2021-07-16T19:19:35Z | OWNER | The only place that calls `refresh_schemas()` is here: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/views/base.py#L120-L124 Ideally only one call to `refresh_schemas()` would be running at any one time. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Race condition errors in new refresh_schemas() mechanism 811367257 | |
881665383 | https://github.com/simonw/datasette/issues/1231#issuecomment-881665383 | https://api.github.com/repos/simonw/datasette/issues/1231 | IC_kwDOBm6k_c40jSVn | simonw 9599 | 2021-07-16T19:21:35Z | 2021-07-16T19:21:35Z | OWNER | https://stackoverflow.com/a/25799871/6083 has a good example of using `asyncio.Lock()`: ```python stuff_lock = asyncio.Lock() async def get_stuff(url): async with stuff_lock: if url in cache: return cache[url] stuff = await aiohttp.request('GET', url) cache[url] = stuff return stuff ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Race condition errors in new refresh_schemas() mechanism 811367257 | |
881671706 | https://github.com/simonw/datasette/issues/1231#issuecomment-881671706 | https://api.github.com/repos/simonw/datasette/issues/1231 | IC_kwDOBm6k_c40jT4a | simonw 9599 | 2021-07-16T19:32:05Z | 2021-07-16T19:32:05Z | OWNER | The test suite passes with that change. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Race condition errors in new refresh_schemas() mechanism 811367257 | |
881668759 | https://github.com/simonw/datasette/issues/1231#issuecomment-881668759 | https://api.github.com/repos/simonw/datasette/issues/1231 | IC_kwDOBm6k_c40jTKX | simonw 9599 | 2021-07-16T19:27:46Z | 2021-07-16T19:27:46Z | OWNER | Second attempt at this: ```diff diff --git a/datasette/app.py b/datasette/app.py index 5976d8b..5f348cb 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -224,6 +224,7 @@ class Datasette: self.inspect_data = inspect_data self.immutables = set(immutables or []) self.databases = collections.OrderedDict() + self._refresh_schemas_lock = asyncio.Lock() self.crossdb = crossdb if memory or crossdb or not self.files: self.add_database(Database(self, is_memory=True), name="_memory") @@ -332,6 +333,12 @@ class Datasette: self.client = DatasetteClient(self) async def refresh_schemas(self): + if self._refresh_schemas_lock.locked(): + return + async with self._refresh_schemas_lock: + await self._refresh_schemas() + + async def _refresh_schemas(self): internal_db = self.databases["_internal"] if not self.internal_db_created: await init_internal_db(internal_db) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Race condition errors in new refresh_schemas() mechanism 811367257 | |
881674857 | https://github.com/simonw/datasette/issues/1231#issuecomment-881674857 | https://api.github.com/repos/simonw/datasette/issues/1231 | IC_kwDOBm6k_c40jUpp | simonw 9599 | 2021-07-16T19:38:39Z | 2021-07-16T19:38:39Z | OWNER | I can't replicate the race condition locally with or without this patch. I'm going to push the commit and then test the CI run from `datasette-graphql` that was failing against it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Race condition errors in new refresh_schemas() mechanism 811367257 | |
881677620 | https://github.com/simonw/datasette/issues/1231#issuecomment-881677620 | https://api.github.com/repos/simonw/datasette/issues/1231 | IC_kwDOBm6k_c40jVU0 | simonw 9599 | 2021-07-16T19:44:12Z | 2021-07-16T19:44:12Z | OWNER | That fixed the race condition in the `datasette-graphql` tests, which is the only place that I've been able to successfully replicate this. I'm going to land this change. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Race condition errors in new refresh_schemas() mechanism 811367257 | |
881686662 | https://github.com/simonw/datasette/issues/1396#issuecomment-881686662 | https://api.github.com/repos/simonw/datasette/issues/1396 | IC_kwDOBm6k_c40jXiG | simonw 9599 | 2021-07-16T20:02:44Z | 2021-07-16T20:02:44Z | OWNER | Confirmed fixed: 0.58.1 was successfully published to Docker Hub in https://github.com/simonw/datasette/runs/3089447346 and the `latest` tag on https://hub.docker.com/r/datasetteproject/datasette/tags was updated. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | "invalid reference format" publishing Docker image 944903881 | |
882096402 | https://github.com/simonw/datasette/issues/123#issuecomment-882096402 | https://api.github.com/repos/simonw/datasette/issues/123 | IC_kwDOBm6k_c40k7kS | RayBB 921217 | 2021-07-18T18:07:29Z | 2021-07-18T18:07:29Z | NONE | I also love the idea for this feature and wonder if it could work without having to download the whole database into memory at once if it's a rather large db. Obviously this could be slower but could have many use cases. My comment is partially inspired by this post about streaming sqlite dbs from github pages or such https://news.ycombinator.com/item?id=27016630 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Datasette serve should accept paths/URLs to CSVs and other file formats 275125561 | |
881932880 | https://github.com/simonw/datasette/issues/1199#issuecomment-881932880 | https://api.github.com/repos/simonw/datasette/issues/1199 | IC_kwDOBm6k_c40kTpQ | simonw 9599 | 2021-07-17T17:39:17Z | 2021-07-17T17:39:17Z | OWNER | I asked about optimizing performance on the SQLite forum and this came up as a suggestion: https://sqlite.org/forum/forumpost/9a6b9ae8e2048c8b?t=c I can start by trying this: PRAGMA mmap_size=268435456; | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Experiment with PRAGMA mmap_size=N 792652391 | |
882138084 | https://github.com/simonw/datasette/issues/123#issuecomment-882138084 | https://api.github.com/repos/simonw/datasette/issues/123 | IC_kwDOBm6k_c40lFvk | simonw 9599 | 2021-07-19T00:04:31Z | 2021-07-19T00:04:31Z | OWNER | I've been thinking more about this one today too. An extension of this (touched on in #417, Datasette Library) would be to support pointing Datasette at a directory and having it automatically load any CSV files it finds anywhere in that folder or its descendants - either loading them fully, or providing a UI that allows users to select a file to open it in Datasette. For larger files I think the right thing to do is import them into an on-disk SQLite database, which is limited only by available disk space. For smaller files loading them into an in-memory database should work fine. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Datasette serve should accept paths/URLs to CSVs and other file formats 275125561 | |
882542519 | https://github.com/simonw/datasette/pull/1400#issuecomment-882542519 | https://api.github.com/repos/simonw/datasette/issues/1400 | IC_kwDOBm6k_c40moe3 | codecov[bot] 22429695 | 2021-07-19T13:20:52Z | 2021-07-19T13:20:52Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1400](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e95c685) into [main](https://codecov.io/gh/simonw/datasette/commit/c73af5dd72305f6a01ea94a2c76d52e5e26de38b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c73af5d) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1400/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1400 +/- ## ======================================= Coverage 91.62% 91.62% ======================================= Files 34 34 Lines 4371 4371 ======================================= Hits 4005 4005 Misses 366 366 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Bump black from 21.6b0 to 21.7b0 947640902 | |
884910320 | https://github.com/simonw/datasette/issues/1401#issuecomment-884910320 | https://api.github.com/repos/simonw/datasette/issues/1401 | IC_kwDOBm6k_c40vqjw | fgregg 536941 | 2021-07-22T13:26:01Z | 2021-07-22T13:26:01Z | CONTRIBUTOR | ordered lists didn't work either, btw | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | unordered list is not rendering bullet points in description_html on database page 950664971 | |
904024939 | https://github.com/simonw/datasette/issues/1445#issuecomment-904024939 | https://api.github.com/repos/simonw/datasette/issues/1445 | IC_kwDOBm6k_c414lNr | simonw 9599 | 2021-08-23T18:52:35Z | 2021-08-23T18:52:35Z | OWNER | The downside of the current implementation of this trick is that it only works for exact LIKE partial matches in a specific table - if you search for `dog cat` and `dog` appears in `title` but `cat` appears in `description` you won't get back that result. I think that's fine though. If you want more advanced search there are other mechanisms you can use. This is meant to be a very quick and dirty starting point for exploring a brand new table. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to search for text across all columns in a table 977323133 | |
904026253 | https://github.com/simonw/datasette/issues/1445#issuecomment-904026253 | https://api.github.com/repos/simonw/datasette/issues/1445 | IC_kwDOBm6k_c414liN | simonw 9599 | 2021-08-23T18:54:49Z | 2021-08-23T18:54:49Z | OWNER | The bigger problem here is UI design. This feels like a pretty niche requirement to me, so adding a prominent search box to the table page (which already has the filters interface, plus the full-text search box for tables that have FTS configured) feels untidy. I could tuck it away in the table cog menu, but that's a weird place for something like this to live. Maybe add it as a new type of filter? Filters apply to specific columns though, so this would be the first filter that applied to _all_ columns - which doesn't really fit the existing filter interface very well. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to search for text across all columns in a table 977323133 | |
904027166 | https://github.com/simonw/datasette/issues/1445#issuecomment-904027166 | https://api.github.com/repos/simonw/datasette/issues/1445 | IC_kwDOBm6k_c414lwe | simonw 9599 | 2021-08-23T18:56:20Z | 2021-08-23T18:56:20Z | OWNER | A related but potentially even more useful ability would be running a search across every column of every table in a whole database. For anything less than a few 100MB this could be incredibly useful. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to search for text across all columns in a table 977323133 | |
904036200 | https://github.com/simonw/datasette/issues/1445#issuecomment-904036200 | https://api.github.com/repos/simonw/datasette/issues/1445 | IC_kwDOBm6k_c414n9o | simonw 9599 | 2021-08-23T19:08:54Z | 2021-08-23T19:08:54Z | OWNER | Figured out a query for searching across every column in every table! https://til.simonwillison.net/datasette/search-all-columns-trick#user-content-same-trick-for-the-entire-database ```sql with tables as ( select name as table_name from sqlite_master where type = 'table' ), queries as ( select 'select ''' || tables.table_name || ''' as _table, rowid from "' || tables.table_name || '" where ' || group_concat( '"' || name || '" like ''%'' || :search || ''%''', ' or ' ) as query from pragma_table_info(tables.table_name), tables group by tables.table_name ) select group_concat(query, ' union all ') from queries ``` The SQL query this generates for larger databases is _extremely_ long - but it does seem to work for smaller databases. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to search for text across all columns in a table 977323133 | |
904037087 | https://github.com/simonw/datasette/issues/1445#issuecomment-904037087 | https://api.github.com/repos/simonw/datasette/issues/1445 | IC_kwDOBm6k_c414oLf | simonw 9599 | 2021-08-23T19:10:17Z | 2021-08-23T19:10:17Z | OWNER | Rather than trying to run that monstrosity in a single `union all` query, a better approach may be to use `fetch()` requests as seen in https://datasette.io/plugins/datasette-search-all | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to search for text across all columns in a table 977323133 | |
904866495 | https://github.com/simonw/datasette/issues/1446#issuecomment-904866495 | https://api.github.com/repos/simonw/datasette/issues/1446 | IC_kwDOBm6k_c417yq_ | simonw 9599 | 2021-08-24T18:13:49Z | 2021-08-24T18:13:49Z | OWNER | OK, now the following optional CSS gives us a sticky footer: ```css html, body { height: 100%; } body { display: flex; flex-direction: column; } .not-footer { flex: 1 0 auto; } footer { flex-shrink: 0; } ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Modify base.html template to support optional sticky footer 978357984 | |
904954530 | https://github.com/simonw/datasette/issues/1446#issuecomment-904954530 | https://api.github.com/repos/simonw/datasette/issues/1446 | IC_kwDOBm6k_c418IKi | simonw 9599 | 2021-08-24T20:32:47Z | 2021-08-24T20:32:47Z | OWNER | Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Modify base.html template to support optional sticky footer 978357984 | |
904982056 | https://github.com/simonw/datasette/issues/859#issuecomment-904982056 | https://api.github.com/repos/simonw/datasette/issues/859 | IC_kwDOBm6k_c418O4o | brandonrobertz 2670795 | 2021-08-24T21:15:04Z | 2021-08-24T21:15:30Z | CONTRIBUTOR | I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Database page loads too slowly with many large tables (due to table counts) 642572841 | |
905097468 | https://github.com/simonw/datasette/pull/1447#issuecomment-905097468 | https://api.github.com/repos/simonw/datasette/issues/1447 | IC_kwDOBm6k_c418rD8 | simonw 9599 | 2021-08-25T01:28:53Z | 2021-08-25T01:28:53Z | OWNER | Good catch, thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove underscore from search mode parameter name 978614898 | |
889525741 | https://github.com/simonw/datasette/issues/1405#issuecomment-889525741 | https://api.github.com/repos/simonw/datasette/issues/1405 | IC_kwDOBm6k_c41BRXt | simonw 9599 | 2021-07-29T23:33:30Z | 2021-07-29T23:33:30Z | OWNER | New documentation section for `datasette.utils` is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | utils.parse_metadata() should be a documented internal function 955316250 | |
889539227 | https://github.com/simonw/datasette/issues/1241#issuecomment-889539227 | https://api.github.com/repos/simonw/datasette/issues/1241 | IC_kwDOBm6k_c41BUqb | simonw 9599 | 2021-07-30T00:15:26Z | 2021-07-30T00:15:26Z | OWNER | One possible treatment: <img width="442" alt="fixtures__sortable__201_rows" src="https://user-images.githubusercontent.com/9599/127581105-dca1193c-1bdd-491e-89d1-ac3136c05c3b.png"> ```html <style> a.action-button { display: inline-block; border: 1.5px solid #666; border-radius: 0.5em; background-color: #ffffff; color: #666; padding: 0.1em 0.8em; text-decoration: none; margin-right: 0.3em; font-size: 0.85em; } </style> <p> {% if query.sql and allow_execute_sql %} <a class="action-button" title="{{ query.sql }}" href="{{ urls.database(database) }}?{{ {'sql': query.sql}|urlencode|safe }}{% if query.params %}&{{ query.params|urlencode|safe }}{% endif %}"> View and edit SQL </a> {% endif %} <a href="#" class="action-button">Copy and share link</a> </p> ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Share button for copying current URL 814595021 | |
889548536 | https://github.com/simonw/datasette/issues/1406#issuecomment-889548536 | https://api.github.com/repos/simonw/datasette/issues/1406 | IC_kwDOBm6k_c41BW74 | simonw 9599 | 2021-07-30T00:43:47Z | 2021-07-30T00:43:47Z | OWNER | Still couldn't replicate on my laptop. On a hunch, I'm going to add `@pytest.mark.serial` to every test that uses `runner.isolated_filesystem()`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470 | |
889547142 | https://github.com/simonw/datasette/issues/1406#issuecomment-889547142 | https://api.github.com/repos/simonw/datasette/issues/1406 | IC_kwDOBm6k_c41BWmG | simonw 9599 | 2021-07-30T00:39:49Z | 2021-07-30T00:39:49Z | OWNER | It happens in CI but not on my laptop. I think I need to run the tests on my laptop like this: https://github.com/simonw/datasette/blob/121e10c29c5b412fddf0326939f1fe46c3ad9d4a/.github/workflows/test.yml#L27-L30 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470 | |
889550391 | https://github.com/simonw/datasette/issues/1406#issuecomment-889550391 | https://api.github.com/repos/simonw/datasette/issues/1406 | IC_kwDOBm6k_c41BXY3 | simonw 9599 | 2021-07-30T00:49:31Z | 2021-07-30T00:49:31Z | OWNER | That fixed it. My hunch is that Click's `runner.isolated_filesystem()` mechanism doesn't play well with `pytest-xdist`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470 | |
889553052 | https://github.com/simonw/datasette/issues/1406#issuecomment-889553052 | https://api.github.com/repos/simonw/datasette/issues/1406 | IC_kwDOBm6k_c41BYCc | simonw 9599 | 2021-07-30T00:58:43Z | 2021-07-30T00:58:43Z | OWNER | Tests are still failing in the job that calculates coverage. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470 | |
889555977 | https://github.com/simonw/datasette/issues/1406#issuecomment-889555977 | https://api.github.com/repos/simonw/datasette/issues/1406 | IC_kwDOBm6k_c41BYwJ | simonw 9599 | 2021-07-30T01:06:57Z | 2021-07-30T01:06:57Z | OWNER | Looking at the source code in Click for `isolated_filesystem()`: https://github.com/pallets/click/blob/9da166957f5848b641231d485467f6140bca2bc0/src/click/testing.py#L450-L468 ```python @contextlib.contextmanager def isolated_filesystem( self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None ) -> t.Iterator[str]: """A context manager that creates a temporary directory and changes the current working directory to it. This isolates tests that affect the contents of the CWD to prevent them from interfering with each other. :param temp_dir: Create the temporary directory under this directory. If given, the created directory is not removed when exiting. .. versionchanged:: 8.0 Added the ``temp_dir`` parameter. """ cwd = os.getcwd() t = tempfile.mkdtemp(dir=temp_dir) os.chdir(t) ``` How about if I pass in that optional `temp_dir` as a temp directory created using the `pytest-xdist` aware pytest mechanisms: https://docs.pytest.org/en/6.2.x/tmpdir.html | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470 | |
889599513 | https://github.com/simonw/datasette/issues/1398#issuecomment-889599513 | https://api.github.com/repos/simonw/datasette/issues/1398 | IC_kwDOBm6k_c41BjYZ | aitoehigie 192984 | 2021-07-30T03:21:49Z | 2021-07-30T03:21:49Z | NONE | Does the library support this now? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Documentation on using Datasette as a library 947044667 | |
890259755 | https://github.com/simonw/datasette/issues/1406#issuecomment-890259755 | https://api.github.com/repos/simonw/datasette/issues/1406 | IC_kwDOBm6k_c41EEkr | simonw 9599 | 2021-07-31T00:04:54Z | 2021-07-31T00:04:54Z | OWNER | STILL failing. I'm going to try removing all instances of `isolated_filesystem()` in favour of a different pattern using pytest temporary files, then see if I can get that to work without the serial hack. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470 | |
890388200 | https://github.com/simonw/datasette/issues/1407#issuecomment-890388200 | https://api.github.com/repos/simonw/datasette/issues/1407 | IC_kwDOBm6k_c41Ej7o | simonw 9599 | 2021-07-31T18:38:41Z | 2021-07-31T18:38:41Z | OWNER | The `path` variable there looked like this: `/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-696/popen-gw0/uds0/datasette.sock` I think what's happening here is that `pytest-xdist` causes `tmp_path_factory.mktemp("uds")` to create significantly longer paths, which in this case is breaking some limit. So for this code to work with `pytest-xdist` I need to make sure the random path to `datasette.sock` is shorter. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | OSError: AF_UNIX path too long in ds_unix_domain_socket_server 957298475 | |
890388656 | https://github.com/simonw/datasette/issues/1407#issuecomment-890388656 | https://api.github.com/repos/simonw/datasette/issues/1407 | IC_kwDOBm6k_c41EkCw | simonw 9599 | 2021-07-31T18:42:41Z | 2021-07-31T18:42:41Z | OWNER | I'll try `tempfile.gettempdir()` - on macOS it returns something like `'/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T'` which is still long but hopefully not too long. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | OSError: AF_UNIX path too long in ds_unix_domain_socket_server 957298475 | |
890390198 | https://github.com/simonw/datasette/issues/1406#issuecomment-890390198 | https://api.github.com/repos/simonw/datasette/issues/1406 | IC_kwDOBm6k_c41Eka2 | simonw 9599 | 2021-07-31T18:55:33Z | 2021-07-31T18:55:33Z | OWNER | To clarify: the core problem here is that an error is thrown any time you call `os.getcwd()` but the directory you are currently in has been deleted. `runner.isolated_filesystem()` assumes that the current directory in has not been deleted. But the various temporary directory utilities in `pytest` work by creating directories and then deleting them. Maybe there's a larger problem here that I play a bit fast and loose with `os.chdir()` in both the test suite and in various lines of code in Datasette itself (in particular in the publish commands)? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470 | |
890390342 | https://github.com/simonw/datasette/issues/1406#issuecomment-890390342 | https://api.github.com/repos/simonw/datasette/issues/1406 | IC_kwDOBm6k_c41EkdG | simonw 9599 | 2021-07-31T18:56:35Z | 2021-07-31T18:56:35Z | OWNER | But... I've lost enough time to this already, and removing `runner.isolated_filesystem()` has the tests passing again. So I'm not going to work on this any more. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470 | |
890390495 | https://github.com/simonw/datasette/issues/1408#issuecomment-890390495 | https://api.github.com/repos/simonw/datasette/issues/1408 | IC_kwDOBm6k_c41Ekff | simonw 9599 | 2021-07-31T18:57:39Z | 2021-07-31T18:57:39Z | OWNER | Opening this issue as an optional follow-up to the work I did in #1406. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Review places in codebase that use os.chdir(), in particularly relating to tests 957302085 | |
890390845 | https://github.com/simonw/datasette/issues/1408#issuecomment-890390845 | https://api.github.com/repos/simonw/datasette/issues/1408 | IC_kwDOBm6k_c41Ekk9 | simonw 9599 | 2021-07-31T19:00:32Z | 2021-07-31T19:00:32Z | OWNER | When I revisit this I can also look at dropping the `@pytest.mark.serial` hack, and maybe the `restore_working_directory()` fixture hack too: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/pytest.ini#L9-L10 https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/tests/conftest.py#L62-L75 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Review places in codebase that use os.chdir(), in particularly relating to tests 957302085 | |
890400059 | https://github.com/simonw/datasette/issues/1409#issuecomment-890400059 | https://api.github.com/repos/simonw/datasette/issues/1409 | IC_kwDOBm6k_c41Em07 | simonw 9599 | 2021-07-31T20:21:51Z | 2021-07-31T20:21:51Z | OWNER | One of these two options: - `--setting default_allow_sql off` - `--setting allow_sql_default off` Existing settings from https://docs.datasette.io/en/0.58.1/settings.html with similar names that I need to be consistent with: - `default_page_size` - `allow_facet` - `default_facet_size` - `allow_download` - `default_cache_ttl` - `default_cache_ttl_hashed` - `allow_csv_stream` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278 | |
890400121 | https://github.com/simonw/datasette/issues/1409#issuecomment-890400121 | https://api.github.com/repos/simonw/datasette/issues/1409 | IC_kwDOBm6k_c41Em15 | simonw 9599 | 2021-07-31T20:22:21Z | 2021-07-31T20:23:34Z | OWNER | I think `default_allow_sql` is more consistent with the current naming conventions, because both `allow` and `default` are used as prefixes at the moment but neither of them are ever used as a suffix. Plus `default_allow_sql off` makes sense to me but `allow_default_sql off` does not - what is "default SQL"? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278 | |
890400425 | https://github.com/simonw/datasette/issues/1409#issuecomment-890400425 | https://api.github.com/repos/simonw/datasette/issues/1409 | IC_kwDOBm6k_c41Em6p | simonw 9599 | 2021-07-31T20:25:16Z | 2021-07-31T20:26:25Z | OWNER | If I was prone to over-thinking (which I am) I'd note that `allow_facet` and `allow_download` and `allow_csv_stream` are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent. So maybe there's a future task where I introduce those as both permissions and metadata `"allow_x"` blocks, then rename the settings themselves to be called `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`. If I was going to do that I should get it in before Datasette 1.0. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278 | |
890397124 | https://github.com/simonw/datasette/issues/1409#issuecomment-890397124 | https://api.github.com/repos/simonw/datasette/issues/1409 | IC_kwDOBm6k_c41EmHE | simonw 9599 | 2021-07-31T19:51:10Z | 2021-07-31T19:51:10Z | OWNER | I think I may like `disable_sql` better. Some options: - `--setting allow_sql off` (consistent with `allow_facet` and `allow_download` and `allow_csv_stream` - all which default to `on` already) - `--setting disable_sql on` - `--setting disable_custom_sql on` The existence of three `allow_*` settings does make a strong argument for staying consistent with that. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278 | |
890397169 | https://github.com/simonw/datasette/issues/1409#issuecomment-890397169 | https://api.github.com/repos/simonw/datasette/issues/1409 | IC_kwDOBm6k_c41EmHx | simonw 9599 | 2021-07-31T19:51:35Z | 2021-07-31T19:51:35Z | OWNER | I'm going to stick with `--setting allow_sql off`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278 | |
890397261 | https://github.com/simonw/datasette/issues/1409#issuecomment-890397261 | https://api.github.com/repos/simonw/datasette/issues/1409 | IC_kwDOBm6k_c41EmJN | simonw 9599 | 2021-07-31T19:52:25Z | 2021-07-31T19:52:25Z | OWNER | I think I can make this modification by teaching the default permissions code here to take the `allow_sql` setting into account: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/datasette/default_permissions.py#L38-L45 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278 | |
890397652 | https://github.com/simonw/datasette/issues/1409#issuecomment-890397652 | https://api.github.com/repos/simonw/datasette/issues/1409 | IC_kwDOBm6k_c41EmPU | simonw 9599 | 2021-07-31T19:56:48Z | 2021-07-31T19:56:48Z | OWNER | The other option would be to use the setting to pick the `default=` argument when calling `self.ds.permission_allowed( request.actor, "execute-sql", resource=database, default=True)`. The problem with that is that there are actually a few different places which perform that check, so changing all of them raises the risk of missing one in the future: https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L436-L444 https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L964-L966 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L220-L221 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L343-L345 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L134-L136 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278 | |
890397753 | https://github.com/simonw/datasette/issues/1409#issuecomment-890397753 | https://api.github.com/repos/simonw/datasette/issues/1409 | IC_kwDOBm6k_c41EmQ5 | simonw 9599 | 2021-07-31T19:57:56Z | 2021-07-31T19:57:56Z | OWNER | I think the correct solution is for the default permissions logic to take the `allow_sql` setting into account, and to return `False` if that setting is set to `off` AND the current actor fails the `actor_matches_allow` checks. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278 | |
890399806 | https://github.com/simonw/datasette/issues/1409#issuecomment-890399806 | https://api.github.com/repos/simonw/datasette/issues/1409 | IC_kwDOBm6k_c41Emw- | simonw 9599 | 2021-07-31T20:18:46Z | 2021-07-31T20:18:46Z | OWNER | My rationale for removing it: https://github.com/simonw/datasette/issues/813#issuecomment-640916290 > Naming problem: Datasette already has a config option with this name: > > $ datasette serve data.db --config allow_sql:1 > > https://datasette.readthedocs.io/en/stable/config.html#allow-sql > > It's confusing to have two things called `allow_sql` that do slightly different things. > > I could retire the `--config allow_sql:0` option entirely, since the new `metadata.json` mechanism can be used to achieve the exact same thing. > > I'm going to do that. This is true. The `"allow_sql"` permissions block in `metadata.json` does indeed have a name that is easily confused with `--setting allow_sql off`. So I definitely need to pick a different name from the setting. `--setting default_allow_sql off` is a good option here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278 | |
890441844 | https://github.com/simonw/datasette/issues/1411#issuecomment-890441844 | https://api.github.com/repos/simonw/datasette/issues/1411 | IC_kwDOBm6k_c41ExB0 | simonw 9599 | 2021-08-01T03:27:30Z | 2021-08-01T03:27:30Z | OWNER | Confirmed: https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1 no longer exhibits the bug. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Canned query ?sql= is pointlessly echoed in query string starting from hidden mode 957345476 | |
891352132 | https://github.com/simonw/datasette/issues/1227#issuecomment-891352132 | https://api.github.com/repos/simonw/datasette/issues/1227 | IC_kwDOBm6k_c41IPRE | simonw 9599 | 2021-08-02T21:38:39Z | 2021-08-02T21:38:39Z | OWNER | Relevant TIL: https://til.simonwillison.net/vscode/vs-code-regular-expressions | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Configure sphinx.ext.extlinks for issues 810394616 | |
891979858 | https://github.com/simonw/datasette/issues/1417#issuecomment-891979858 | https://api.github.com/repos/simonw/datasette/issues/1417 | IC_kwDOBm6k_c41KohS | simonw 9599 | 2021-08-03T16:15:30Z | 2021-08-03T16:15:30Z | OWNER | Docs: https://pypi.org/project/codespell/ There's a `codespell --ignore-words=FILE` option for ignoring words. I don't have any that need ignoring yet but I'm going to add that file anyway, that way I can have codespell be a failing test but still provide a way to work around it if it incorrectly flags a correct word. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Use codespell in CI to spot spelling errors 959278472 | |
891984359 | https://github.com/simonw/datasette/pull/1418#issuecomment-891984359 | https://api.github.com/repos/simonw/datasette/issues/1418 | IC_kwDOBm6k_c41Kpnn | simonw 9599 | 2021-08-03T16:21:56Z | 2021-08-03T16:21:56Z | OWNER | Failed with: ``` docs/authentication.rst:63: perfom ==> perform docs/authentication.rst:76: perfom ==> perform docs/changelog.rst:429: repsonse ==> response docs/changelog.rst:503: permissons ==> permissions docs/changelog.rst:717: compatibilty ==> compatibility docs/changelog.rst:1172: browseable ==> browsable docs/deploying.rst:191: similiar ==> similar docs/internals.rst:434: Respons ==> Response, respond docs/internals.rst:440: Respons ==> Response, respond docs/internals.rst:717: tha ==> than, that, the docs/performance.rst:42: databse ==> database docs/plugin_hooks.rst:667: utilites ==> utilities docs/publish.rst:168: countainer ==> container docs/settings.rst:352: inalid ==> invalid docs/sql_queries.rst:406: preceeded ==> preceded, proceeded ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Spelling corrections plus CI job for codespell 959284434 | |
891987129 | https://github.com/simonw/datasette/pull/1418#issuecomment-891987129 | https://api.github.com/repos/simonw/datasette/issues/1418 | IC_kwDOBm6k_c41KqS5 | codecov[bot] 22429695 | 2021-08-03T16:26:01Z | 2021-08-03T16:32:20Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1418](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (532170f) into [main](https://codecov.io/gh/simonw/datasette/commit/54b6e96ee8aa553b6671e341a1944f93f3fb89c3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (54b6e96) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1418/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1418 +/- ## ======================================= Coverage 91.64% 91.64% ======================================= Files 34 34 Lines 4382 4382 ======================================= Hits 4016 4016 Misses 366 366 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Spelling corrections plus CI job for codespell 959284434 | |
892276385 | https://github.com/simonw/datasette/issues/1419#issuecomment-892276385 | https://api.github.com/repos/simonw/datasette/issues/1419 | IC_kwDOBm6k_c41Lw6h | fgregg 536941 | 2021-08-04T00:58:49Z | 2021-08-04T00:58:49Z | CONTRIBUTOR | yes, [filter clause on aggregate queries were added to sqlite3 in 3.30](https://www.sqlite.org/releaselog/3_30_1.html) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `publish cloudrun` should deploy a more recent SQLite version 959710008 | |
892365639 | https://github.com/simonw/datasette/issues/1420#issuecomment-892365639 | https://api.github.com/repos/simonw/datasette/issues/1420 | IC_kwDOBm6k_c41MGtH | simonw 9599 | 2021-08-04T05:05:07Z | 2021-08-04T05:05:07Z | OWNER | https://github.com/simonw/datasette/blob/cd8b7bee8fb5c1cdce7c8dbfeb0166011abc72c6/datasette/publish/cloudrun.py#L153-L158 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `datasette publish cloudrun --cpu X` option 959898166 | |
892372509 | https://github.com/simonw/datasette/issues/1420#issuecomment-892372509 | https://api.github.com/repos/simonw/datasette/issues/1420 | IC_kwDOBm6k_c41MIYd | simonw 9599 | 2021-08-04T05:22:29Z | 2021-08-04T05:22:29Z | OWNER | Testing this manually with: datasette publish cloudrun fixtures.db --memory 8G --cpu 4 \ --service fixtures-over-provisioned-issue-1420 --install datasette-psutil And for comparison: datasette publish cloudrun fixtures.db --service fixtures-default-issue-1420 \ --install datasette-psutil | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `datasette publish cloudrun --cpu X` option 959898166 | |
892374253 | https://github.com/simonw/datasette/issues/1420#issuecomment-892374253 | https://api.github.com/repos/simonw/datasette/issues/1420 | IC_kwDOBm6k_c41MIzt | simonw 9599 | 2021-08-04T05:27:21Z | 2021-08-04T05:29:59Z | OWNER | I'll delete these services shortly afterwards. Right now: https://fixtures-over-provisioned-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil (deployed with `--memory 8G --cpu 4`) returns: ``` process.memory_info() pmem(rss=60456960, vms=518930432, shared=0, text=0, lib=0, data=0, dirty=0) ... psutil.cpu_times(True) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) psutil.virtual_memory() svmem(total=2147483648, available=2092531712, percent=2.6, used=33103872, free=2092531712, active=44130304, inactive=10792960, buffers=0, cached=21848064, shared=262144, slab=0) ``` https://fixtures-default-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil returns: ``` process.memory_info() pmem(rss=49324032, vms=140595200, shared=0, text=0, lib=0, data=0, dirty=0) ... psutil.cpu_times(True) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) psutil.virtual_memory() svmem(total=2147483648, available=2091188224, percent=2.6, used=40071168, free=2091188224, active=41586688, inactive=7983104, buffers=0, cached=16224256, shared=262144, slab=0) ``` These numbers are different enough that I assume this works as advertised. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `datasette publish cloudrun --cpu X` option 959898166 | |
892376353 | https://github.com/simonw/datasette/issues/1420#issuecomment-892376353 | https://api.github.com/repos/simonw/datasette/issues/1420 | IC_kwDOBm6k_c41MJUh | simonw 9599 | 2021-08-04T05:33:12Z | 2021-08-04T05:33:12Z | OWNER | In the Cloud Run console (before I deleted these services) when I click "Edit and deploy new revision" I see this for the default one: <img width="551" alt="Deploy_new_revision_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/128127309-7ec2064f-b107-4d99-b630-ee58734c4a72.png"> And this for the big one: <img width="546" alt="Deploy_new_revision_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/128127366-7cdcb245-2e83-4b88-9d4f-0c595a486139.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `datasette publish cloudrun --cpu X` option 959898166 | |
893079520 | https://github.com/simonw/datasette/issues/1420#issuecomment-893079520 | https://api.github.com/repos/simonw/datasette/issues/1420 | IC_kwDOBm6k_c41O0_g | simonw 9599 | 2021-08-05T00:54:59Z | 2021-08-05T00:54:59Z | OWNER | Just saw this error: `ERROR: (gcloud.run.deploy) The `--cpu` flag is not supported on the fully managed version of Cloud Run. Specify `--platform gke` or run `gcloud config set run/platform gke` to work with Cloud Run for Anthos deployed on Google Cloud.` Which is weird because I managed to run this successfully the other day. Maybe a region difference thing? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `datasette publish cloudrun --cpu X` option 959898166 | |
893114612 | https://github.com/simonw/datasette/issues/1419#issuecomment-893114612 | https://api.github.com/repos/simonw/datasette/issues/1419 | IC_kwDOBm6k_c41O9j0 | fgregg 536941 | 2021-08-05T02:29:06Z | 2021-08-05T02:29:06Z | CONTRIBUTOR | there's a lot of complexity here, that's probably not worth addressing. i got what i needed by patching the dockerfile that cloudrun uses to install a newer version of sqlite. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `publish cloudrun` should deploy a more recent SQLite version 959710008 | |
893122356 | https://github.com/simonw/datasette/issues/1422#issuecomment-893122356 | https://api.github.com/repos/simonw/datasette/issues/1422 | IC_kwDOBm6k_c41O_c0 | simonw 9599 | 2021-08-05T02:52:31Z | 2021-08-05T02:52:44Z | OWNER | If you do this it should still be possible to view the SQL - which means we need a new parameter. I propose `?_show_sql=1` to over-ride the hidden default. I think the configuration should use `hide_sql: true` - looking like this: ```yaml databases: fixtures: queries: neighborhood_search: hide_sql: true sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood title: Search neighborhoods ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to default to hiding the SQL for a canned query 961367843 | |
893131703 | https://github.com/simonw/datasette/issues/1422#issuecomment-893131703 | https://api.github.com/repos/simonw/datasette/issues/1422 | IC_kwDOBm6k_c41PBu3 | simonw 9599 | 2021-08-05T03:16:46Z | 2021-08-05T03:16:46Z | OWNER | The logic for this is a little bit fiddly, due to the need to switch to using `?_show_sql=1` on the link depending on the context. - If metadata says hide and there's no query string, hide and link to `?_show_sql=1` - If metadata says hide but query string says `?_show_sql=1`, show and have hide link linking to URL without `?_show_sql=1` - Otherwise, show and link to `?_hide_sql=1` - ... or if that query string is there then hide and link to URL without `?_hide_sql=1` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to default to hiding the SQL for a canned query 961367843 | |
893133496 | https://github.com/simonw/datasette/issues/1419#issuecomment-893133496 | https://api.github.com/repos/simonw/datasette/issues/1419 | IC_kwDOBm6k_c41PCK4 | simonw 9599 | 2021-08-05T03:22:44Z | 2021-08-05T03:22:44Z | OWNER | I ran into this exact same problem today! I only just learned how to use filter on aggregates: https://til.simonwillison.net/sqlite/sqlite-aggregate-filter-clauses A workaround I used is to add this to the deploy command: datasette publish cloudrun ... --install=pysqlite3-binary This will install the https://pypi.org/project/pysqlite3-binary for package which bundles a more recent SQLite version. | {"total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `publish cloudrun` should deploy a more recent SQLite version 959710008 | |
893996604 | https://github.com/simonw/datasette/issues/1423#issuecomment-893996604 | https://api.github.com/repos/simonw/datasette/issues/1423 | IC_kwDOBm6k_c41SU48 | simonw 9599 | 2021-08-06T04:43:07Z | 2021-08-06T04:43:37Z | OWNER | Problem: on a page which doesn't have quite enough facet values to trigger the display of the "..." link that links to `?_facet_size=max` the user would still have to manually count the values - up to 30 by default. So maybe the count should always be shown, perhaps as a non-bold light colored number? I could even hide it in a non-discoverable tooltip. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Show count of facet values if ?_facet_size=max 962391325 | |
894452990 | https://github.com/simonw/datasette/issues/1423#issuecomment-894452990 | https://api.github.com/repos/simonw/datasette/issues/1423 | IC_kwDOBm6k_c41UET- | simonw 9599 | 2021-08-06T18:49:37Z | 2021-08-06T18:49:37Z | OWNER | Could display them always, like this: <img width="1299" alt="fixtures__compound_three_primary_keys__1_001_rows_and_School_enrollment_analysis_Colorado_-_Google_Docs" src="https://user-images.githubusercontent.com/9599/128557997-6bedb52b-45f1-4ab1-9a79-ddf6c01f6036.png"> That's with `<span style="font-size: 0.8em;color: #666;">23</span>` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Show count of facet values if ?_facet_size=max 962391325 | |
894453520 | https://github.com/simonw/datasette/issues/1423#issuecomment-894453520 | https://api.github.com/repos/simonw/datasette/issues/1423 | IC_kwDOBm6k_c41UEcQ | simonw 9599 | 2021-08-06T18:50:40Z | 2021-08-06T18:50:40Z | OWNER | Point of confusion: if only 30 options are shown, but there's a `...` at the end, what would the number be? It can't be the total number of facets because we haven't counted them all - but if it's just the number of displayed facets that's like to be confusing. So the original idea of showing the counts only if `_facet_size=max` is a good one. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Show count of facet values if ?_facet_size=max 962391325 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
updated_at (date) >1000 ✖