{"html_url": "https://github.com/simonw/datasette/issues/651#issuecomment-579675357", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/651", "id": 579675357, "node_id": "MDEyOklzc3VlQ29tbWVudDU3OTY3NTM1Nw==", "user": {"value": 2181410, "label": "clausjuhl"}, "created_at": "2020-01-29T09:45:00Z", "updated_at": "2021-07-14T19:26:06Z", "author_association": "NONE", "body": "Hi Simon\r\n\r\nThank you for adding the escape_function, but it does not work on my datasette-installation (0.33). I've added the following file to my datasette-dir: `/plugins/sql_functions.py`:\r\n\r\n```python\r\nfrom datasette import hookimpl\r\n\r\ndef escape_fts_query(query):\r\n    bits = query.split()\r\n    return ' '.join('\"{}\"'.format(bit.replace('\"', '')) for bit in bits)\r\n\r\n@hookimpl\r\ndef prepare_connection(conn):\r\n    conn.create_function(\"escape_fts_query\", 1, escape_fts_query)`\r\n```\r\nIt has no effect on the standard queries to the tables though, as they still produce errors when including any characters like '-', '/', '+' or '?'\r\n\r\nDoes the function only work when using costum queries, where I can include the escape_fts-function explicitly in the sql-query?\r\n\r\nPS. I'm calling datasette with --plugins=plugins, and my other plugins work just fine.\r\nPPS. The fts5 virtual table is created with 'sqlite3' like so:\r\n\r\n`CREATE VIRTUAL TABLE \"cases_fts\" USING FTS5(\r\n  title,\r\n  subtitle,\r\n  resume,\r\n  suggestion,\r\n  presentation,\r\n  detail = full,\r\n  content_rowid = 'id',\r\n  content = 'cases',\r\n  tokenize='unicode61', 'remove_diacritics 2', 'tokenchars \"-_\"'\r\n);`\r\n\r\nThanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 539590148, "label": "fts5 syntax error when using punctuation"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-880150755", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 880150755, "node_id": "MDEyOklzc3VlQ29tbWVudDg4MDE1MDc1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-14T19:26:47Z", "updated_at": "2021-07-14T19:29:08Z", "author_association": "OWNER", "body": "> What are the side-effects of turning that on in the query string, or even by default as you suggested? I see that you stated in the docs... \"to ensure they do not cause any confusion for users who are not aware of them\", but I'm not sure what those could be.\r\n\r\nMainly that it's possible to generate SQL queries that crash with an error. This was the example that convinced me to default to escaping:\r\n\r\n- https://www.niche-museums.com/browse/museums?_search=park.&_searchmode=raw (returns `fts5: syntax error near \".\"`)\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-880153069", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 880153069, "node_id": "MDEyOklzc3VlQ29tbWVudDg4MDE1MzA2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-14T19:31:00Z", "updated_at": "2021-07-14T19:31:00Z", "author_association": "OWNER", "body": "... though interestingly I can't replicate that error on `latest.datasette.io` - https://latest.datasette.io/fixtures/searchable?_search=park.&_searchmode=raw\r\n\r\nThat's running https://latest.datasette.io/-/versions SQLite 3.35.4 whereas https://www.niche-museums.com/-/versions is running 3.27.2 (the most recent version available with Vercel) - but there's nothing in the SQLite changelog between those two versions that suggests changes to how the FTS5 parser works. https://www.sqlite.org/changes.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880256058", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 880256058, "node_id": "MDEyOklzc3VlQ29tbWVudDg4MDI1NjA1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-14T22:40:01Z", "updated_at": "2021-07-14T22:40:47Z", "author_association": "OWNER", "body": "Full docs here: https://www.sqlite.org/draft/cli.html#csv\r\n\r\nOne catch: how this works has changed in recent SQLite versions: https://www.sqlite.org/changes.html\r\n\r\n- 2020-12-01 (3.34.0) - \"Table name quoting works correctly for the .import dot-command\"\r\n- 2020-05-22 (3.32.0) - \"Add options to the .import command: --csv, --ascii, --skip\"\r\n- 2017-08-01 (3.20.0) - \" The \".import\" command ignores an initial UTF-8 BOM.\"\r\n\r\nThe \"skip\" feature is particularly important to understand. https://www.sqlite.org/draft/cli.html#csv says:\r\n\r\n> There are two cases to consider: (1) Table \"tab1\" does not previously exist and (2) table \"tab1\" does already exist.\r\n>\r\n> In the first case, when the table does not previously exist, the table is automatically created and the content of the first row of the input CSV file is used to determine the name of all the columns in the table. In other words, if the table does not previously exist, the first row of the CSV file is interpreted to be column names and the actual data starts on the second row of the CSV file.\r\n>\r\n> For the second case, when the table already exists, every row of the CSV file, including the first row, is assumed to be actual content. If the CSV file contains an initial row of column labels, you can cause the .import command to skip that initial row using the \"--skip 1\" option.\r\n\r\nBut the `--skip 1` option is only available in 3.32.0 and higher.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880256865", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 880256865, "node_id": "MDEyOklzc3VlQ29tbWVudDg4MDI1Njg2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-14T22:42:11Z", "updated_at": "2021-07-14T22:42:11Z", "author_association": "OWNER", "body": "Potential workaround for missing `--skip` implementation is that the filename can be a command instead, so maybe it could shell out to `tail -n +1 filename`:\r\n\r\n> The source argument is the name of a file to be read or, if it begins with a \"|\" character, specifies a command which will be run to produce the input CSV data.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880257587", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 880257587, "node_id": "MDEyOklzc3VlQ29tbWVudDg4MDI1NzU4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-14T22:44:05Z", "updated_at": "2021-07-14T22:44:05Z", "author_association": "OWNER", "body": "https://unix.stackexchange.com/a/642364 suggests you can also use this to import from stdin, like so:\r\n\r\n    sqlite3 -csv $database_file_name \".import '|cat -' $table_name\"\r\n\r\nHere the `sqlite3 -csv` is an alternative to using `.mode csv`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880259255", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 880259255, "node_id": "MDEyOklzc3VlQ29tbWVudDg4MDI1OTI1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-14T22:48:41Z", "updated_at": "2021-07-14T22:48:41Z", "author_association": "OWNER", "body": "Should also take advantage of `.mode tabs` to support `sqlite-utils insert blah.db blah blah.csv --tsv --fast`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1394#issuecomment-880278256", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1394", "id": 880278256, "node_id": "MDEyOklzc3VlQ29tbWVudDg4MDI3ODI1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-14T23:35:18Z", "updated_at": "2021-07-14T23:35:18Z", "author_association": "OWNER", "body": "The challenge here is that faceting doesn't currently modify the inner SQL at all - it wraps it so that it can work against any SQL statement (though Datasette itself does not yet take advantage of that ability, only offering faceting on table pages).\r\n\r\nSo just removing the order by wouldn't be appropriate if the inner query looked something like this:\r\n\r\n```sql\r\nselect * from items order by created desc limit 100\r\n```\r\nSince the intent there would be to return facet counts against only the most recent 100 items.\r\n\r\nIn SQLite the `limit` has to come after the `order by` though, so the fix here could be as easy as using a regular expression to identify queries that end with `order by COLUMN (desc)?` and stripping off that clause.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944870799, "label": "Big performance boost on faceting: skip the inner order by"}, "performed_via_github_app": null}