html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/datasette/issues/1143#issuecomment-744757558,https://api.github.com/repos/simonw/datasette/issues/1143,744757558,MDEyOklzc3VlQ29tbWVudDc0NDc1NzU1OA==,9599,simonw,2020-12-14T22:42:10Z,2020-12-14T22:42:10Z,OWNER,"This may involve a breaking change to the CLI settings interface, so I'm adding this to the 1.0 milestone.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices", https://github.com/simonw/datasette/issues/1171#issuecomment-754911290,https://api.github.com/repos/simonw/datasette/issues/1171,754911290,MDEyOklzc3VlQ29tbWVudDc1NDkxMTI5MA==,59874,rcoup,2021-01-05T21:31:15Z,2021-01-05T21:31:15Z,NONE,"We did this for [Sno](https://sno.earth) under macOS — it's a PyInstaller binary/setup which uses [Packages](http://s.sudre.free.fr/Software/Packages/about.html) for packaging. * [Building & Signing](https://github.com/koordinates/sno/blob/master/platforms/Makefile#L67-L95) * [Packaging & Notarizing](https://github.com/koordinates/sno/blob/master/platforms/Makefile#L121-L215) * [Github Workflow](https://github.com/koordinates/sno/blob/master/.github/workflows/build.yml#L228-L269) has the CI side of it FYI (if you ever get to it) for Windows you need to get a code signing certificate. And if you want automated CI, you'll want to get an ""EV CodeSigning for HSM"" certificate from GlobalSign, which then lets you put the certificate into Azure Key Vault. Which you can use with [azuresigntool](https://github.com/vcsjones/AzureSignTool) to sign your code & installer. (Non-EV certificates are a waste of time, the user still gets big warnings at install time). ","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables, https://github.com/simonw/datasette/issues/1240#issuecomment-786812716,https://api.github.com/repos/simonw/datasette/issues/1240,786812716,MDEyOklzc3VlQ29tbWVudDc4NjgxMjcxNg==,9599,simonw,2021-02-26T18:18:18Z,2021-02-26T18:18:18Z,OWNER,"Agreed, this would be extremely useful. I'd love to be able to facet against custom queries. It's a fair bit of work to implement but it's not impossible. Closing this as a duplicate of #972.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",814591962,Allow facetting on custom queries, https://github.com/simonw/datasette/issues/1387#issuecomment-873156408,https://api.github.com/repos/simonw/datasette/issues/1387,873156408,MDEyOklzc3VlQ29tbWVudDg3MzE1NjQwOA==,9599,simonw,2021-07-02T17:37:30Z,2021-07-02T17:37:30Z,OWNER,Updated documentation is here: https://docs.datasette.io/en/latest/deploying.html#apache-proxy-configuration,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs, https://github.com/simonw/datasette/issues/1402#issuecomment-886969541,https://api.github.com/repos/simonw/datasette/issues/1402,886969541,IC_kwDOBm6k_c403hTF,9599,simonw,2021-07-26T19:31:40Z,2021-07-26T19:31:40Z,OWNER,"Datasette could do a pretty good job of this by default, using `twitter:card` and `og:url` tags - like on https://til.simonwillison.net/jq/extracting-objects-recursively I could also provide a mechanism to customize these - in particular to add images of some sort. It feels like something that should tie in to the metadata mechanism.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",951185411,feature request: social meta tags, https://github.com/simonw/datasette/issues/1528#issuecomment-988468238,https://api.github.com/repos/simonw/datasette/issues/1528,988468238,IC_kwDOBm6k_c466tQO,30934,20after4,2021-12-08T03:35:45Z,2021-12-08T03:35:45Z,NONE,"FWIW I implemented something similar with a bit of plugin code: ```python @hookimpl def canned_queries(datasette: Datasette, database: str) -> Mapping[str, str]: # load ""canned queries"" from the filesystem under # www/sql/db/query_name.sql queries = {} sqldir = Path(__file__).parent.parent / ""sql"" if database: sqldir = sqldir / database if not sqldir.is_dir(): return queries for f in sqldir.glob('*.sql'): try: sql = f.read_text('utf8').strip() if not len(sql): log(f""Skipping empty canned query file: {f}"") continue queries[f.stem] = { ""sql"": sql } except OSError as err: log(err) return queries ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1060631257,"Add new `""sql_file""` key to Canned Queries in metadata?", https://github.com/simonw/datasette/issues/1546#issuecomment-997124280,https://api.github.com/repos/simonw/datasette/issues/1546,997124280,IC_kwDOBm6k_c47bui4,9599,simonw,2021-12-18T02:05:16Z,2021-12-18T02:05:16Z,OWNER,"Sure - there are actually several levels to this. The code that creates connections to the database is this: https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/database.py#L72-L95 For files on disk, it does this: ```python # For read-only connections conn = sqlite3.connect( ""file:my.db?mode=ro"", uri=True, check_same_thread=False) # For connections that should be treated as immutable: conn = sqlite3.connect( ""file:my.db?immutable=1"", uri=True, check_same_thread=False) ``` For in-memory databases it runs this after the connection has been created: ```python conn.execute(""PRAGMA query_only=1"") ``` SQLite `PRAGMA` queries are treated as dangerous: someone could run `PRAGMA query_only=0` to turn that previous option off for example. So this function runs against any incoming SQL to verify that it looks like a `SELECT ...` and doesn't have anything like that in it. https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/utils/__init__.py#L195-L204 You can see the tests for that here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/tests/test_utils.py#L136-L170","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1076057610,validating the sql, https://github.com/simonw/datasette/issues/1553#issuecomment-996103956,https://api.github.com/repos/simonw/datasette/issues/1553,996103956,IC_kwDOBm6k_c47X1cU,9599,simonw,2021-12-16T19:14:38Z,2021-12-16T19:14:38Z,OWNER,This is a really interesting idea - kind of similar to how many APIs include custom HTTP headers informing of rate-limits.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1079111498,if csv export is truncated in non streaming mode set informative response header, https://github.com/simonw/datasette/issues/1886#issuecomment-1313052863,https://api.github.com/repos/simonw/datasette/issues/1886,1313052863,IC_kwDOBm6k_c5OQ5i_,9599,simonw,2022-11-14T03:40:50Z,2022-11-14T03:40:50Z,OWNER,"Tim Sherratt on Twitter: https://twitter.com/wragge/status/1591930345469153282 > Where do I start? The [#GLAMWorkbench](https://twitter.com/hashtag/GLAMWorkbench?src=hashtag_click) now includes a number of examples where GLAM data is harvested, processed, and then made available for exploration via Datasette. > > https://glam-workbench.net/ > > For example the GLAM Name Index Search brings together 10+ million entries from 240 indexes and provides an aggregated search using the Datasette search-all plugin: > > https://glam-workbench.net/name-search/ > > Most recently I converted PDFs of the Tasmanian Postal Directories to a big Datasette instance: https://updates.timsherratt.org/2022/09/15/from-pdfs-to.html the process is documented and reusable.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/283#issuecomment-855369819,https://api.github.com/repos/simonw/datasette/issues/283,855369819,MDEyOklzc3VlQ29tbWVudDg1NTM2OTgxOQ==,9599,simonw,2021-06-06T09:40:18Z,2021-06-06T09:40:18Z,OWNER,"> One note on using this pragma I got an error on starting datasette `no such table: pragma_database_list`. > > I diagnosed this to an older version of sqlite3 (3.14.2) and upgrading to a newer version (3.34.2) fixed the issue. That issue is fixed in #1276.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins, https://github.com/simonw/datasette/issues/394#issuecomment-642522285,https://api.github.com/repos/simonw/datasette/issues/394,642522285,MDEyOklzc3VlQ29tbWVudDY0MjUyMjI4NQ==,58298410,LVerneyPEReN,2020-06-11T09:15:19Z,2020-06-11T09:15:19Z,NONE,"Hi @wragge, This looks great, thanks for the share! I refactored it into a self-contained function, binding on a random available TCP port (multi-user context). I am using subprocess API directly since the `%run` magic was leaving defunct process behind :/ ![image](https://user-images.githubusercontent.com/58298410/84367566-b5d0d500-abd4-11ea-96e2-f5c05a28e506.png) ```python import socket from signal import SIGINT from subprocess import Popen, PIPE from IPython.display import display, HTML from notebook.notebookapp import list_running_servers def get_free_tcp_port(): """""" Get a free TCP port. """""" tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM) tcp.bind(('', 0)) _, port = tcp.getsockname() tcp.close() return port def datasette(database): """""" Run datasette on an SQLite database. """""" # Get current running servers servers = list_running_servers() # Get the current base url base_url = next(servers)['base_url'] # Get a free port port = get_free_tcp_port() # Create a base url for Datasette suing the proxy path proxy_url = f'{base_url}proxy/absolute/{port}/' # Display a link to Datasette display(HTML(f'
View Datasette (Click on the stop button to close the Datasette server)
')) # Launch Datasette with Popen( [ 'python', '-m', 'datasette', '--', database, '--port', str(port), '--config', f'base_url:{proxy_url}' ], stdout=PIPE, stderr=PIPE, bufsize=1, universal_newlines=True ) as p: print(p.stdout.readline(), end='') while True: try: line = p.stderr.readline() if not line: break print(line, end='') exit_code = p.poll() except KeyboardInterrupt: p.send_signal(SIGINT) ``` Ideally, I'd like some extra magic to notify users when they are leaving the closing the notebook tab and make them terminate the running datasette processes. I'll be looking for it.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",396212021,base_url configuration setting, https://github.com/simonw/datasette/issues/509#issuecomment-749749948,https://api.github.com/repos/simonw/datasette/issues/509,749749948,MDEyOklzc3VlQ29tbWVudDc0OTc0OTk0OA==,9599,simonw,2020-12-22T20:03:10Z,2020-12-22T20:03:10Z,OWNER,"If you open multiple files with the same filename, e.g. like this: datasette fixtures.db templates/fixtures.db plugins/fixtures.db You'll now get this: ","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",456568880,Support opening multiple databases with the same stem, https://github.com/simonw/datasette/issues/526#issuecomment-1259693536,https://api.github.com/repos/simonw/datasette/issues/526,1259693536,IC_kwDOBm6k_c5LFWXg,9599,simonw,2022-09-27T15:42:55Z,2022-09-27T15:42:55Z,OWNER,"It's interesting to note WHY the time limit works against this so well. The time limit as-implemented looks like this: https://github.com/simonw/datasette/blob/5f9f567acbc58c9fcd88af440e68034510fb5d2b/datasette/utils/__init__.py#L181-L201 The key here is `conn.set_progress_handler(handler, n)` - which specifies that the handler function should be called every `n` SQLite operations. The handler function then checks to see if too much time has transpired and conditionally cancels the query. This also doubles up as a ""maximum number of operations"" guard, which is what's happening when you attempt to fetch an infinite number of rows from an infinite table. That limit code could even be extended to say ""exit the query after either 5s or 50,000,000 operations"". I don't think that's necessary though. To be honest I'm having trouble with the idea of dropping `max_returned_rows` mainly because what Datasette does (allow arbitrary untrusted SQL queries) is dangerous, so I've designed in multiple redundant defence-in-depth mechanisms right from the start.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/594#issuecomment-547373739,https://api.github.com/repos/simonw/datasette/issues/594,547373739,MDEyOklzc3VlQ29tbWVudDU0NzM3MzczOQ==,2680980,willingc,2019-10-29T11:21:52Z,2019-10-29T11:21:52Z,NONE,"Just an FYI for folks wishing to run datasette with Python 3.8, I was able to successfully use datasette with the following in a virtual environment: ``` pip install uvloop==0.14.0rc1 pip install uvicorn==0.9.1 ``` ","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",506297048,upgrade to uvicorn-0.9 to be Python-3.8 friendly, https://github.com/simonw/datasette/issues/594#issuecomment-552276247,https://api.github.com/repos/simonw/datasette/issues/594,552276247,MDEyOklzc3VlQ29tbWVudDU1MjI3NjI0Nw==,9599,simonw,2019-11-11T03:13:00Z,2019-11-11T03:13:00Z,OWNER,#622,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",506297048,upgrade to uvicorn-0.9 to be Python-3.8 friendly, https://github.com/simonw/datasette/issues/670#issuecomment-797158641,https://api.github.com/repos/simonw/datasette/issues/670,797158641,MDEyOklzc3VlQ29tbWVudDc5NzE1ODY0MQ==,9599,simonw,2021-03-12T00:59:49Z,2021-03-12T00:59:49Z,OWNER,"> Challenge: what's the equivalent for PostgreSQL of opening a database in read only mode? Will I have to talk users through creating read only credentials? It looks like the answer to this is yes - I'll need users to setup read-only credentials. Here's a TIL about that: https://til.simonwillison.net/postgresql/read-only-postgresql-user","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",564833696,Prototoype for Datasette on PostgreSQL, https://github.com/simonw/datasette/issues/716#issuecomment-609455243,https://api.github.com/repos/simonw/datasette/issues/716,609455243,MDEyOklzc3VlQ29tbWVudDYwOTQ1NTI0Mw==,9599,simonw,2020-04-05T17:47:33Z,2020-04-05T17:47:33Z,OWNER,"You start `git bisect` by giving it a known bad commit and a known good one: ``` git bisect start master 286ed28 ``` Then you tell it to start running your script: ``` git bisect run python ../datasette-issue-716/check_view_name.py ``` Here's what I got: ``` (datasette) ~/Dropbox/Development/datasette $ git bisect start master 286ed28 Bisecting: 30 revisions left to test after this (roughly 5 steps) [dc80e779a2e708b2685fc641df99e6aae9ad6f97] Handle scope path if it is a string (datasette) ~/Dropbox/Development/datasette $ git bisect run python ../datasette-issue-716/check_view_name.py running python ../datasette-issue-716/check_view_name.py Traceback (most recent call last): ... Bisecting: 15 revisions left to test after this (roughly 4 steps) [7c6a9c35299f251f9abfb03fd8e85143e4361709] Better tests for prepare_connection() plugin hook, refs #678 running python ../datasette-issue-716/check_view_name.py Traceback (most recent call last): ... Bisecting: 7 revisions left to test after this (roughly 3 steps) [0091dfe3e5a3db94af8881038d3f1b8312bb857d] More reliable tie-break ordering for facet results running python ../datasette-issue-716/check_view_name.py Traceback (most recent call last): ... Bisecting: 3 revisions left to test after this (roughly 2 steps) [ce12244037b60ba0202c814871218c1dab38d729] Release notes for 0.35 running python ../datasette-issue-716/check_view_name.py Traceback (most recent call last): ... Bisecting: 0 revisions left to test after this (roughly 1 step) [4d7dae9eb75e5430c3ee3c369bb5cd9ba0a148bc] Added a bunch more plugins to the Ecosystem page running python ../datasette-issue-716/check_view_name.py Traceback (most recent call last): ... 70b915fb4bc214f9d064179f87671f8a378aa127 is the first bad commit commit 70b915fb4bc214f9d064179f87671f8a378aa127 Author: Simon Willison