issue_comments
9,947 rows sorted by id descending
This data as json, CSV (advanced)
id ▲ | html_url | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
1074478299 | https://github.com/simonw/datasette/issues/1671#issuecomment-1074478299 | https://api.github.com/repos/simonw/datasette/issues/1671 | IC_kwDOBm6k_c5ACzzb | simonw 9599 | 2022-03-21T22:20:26Z | 2022-03-21T22:20:26Z | OWNER | Thinking about options for fixing this... The following query works fine: ```sql select * from test_view where cast(has_expired as text) = '1' ``` I don't want to start using this for every query, because one of the goals of Datasette is to help people who are learning SQL: - #1613 If someone clicks on "View and edit SQL" from a filtered table page I don't want them to have to wonder why that `cast` is there. But... for querying views, the `cast` turns out to be necessary. So one fix would be to get the SQL generating logic to use casts like this any time it is operating against a view. An even better fix would be to detect which columns in a view come from a table and which ones might not, and only use casts for the columns that aren't definitely from a table. The trick I was exploring here might be able to help with that: - #1293 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 1174655187 | |
1074470568 | https://github.com/simonw/datasette/issues/1671#issuecomment-1074470568 | https://api.github.com/repos/simonw/datasette/issues/1671 | IC_kwDOBm6k_c5ACx6o | simonw 9599 | 2022-03-21T22:11:14Z | 2022-03-21T22:12:49Z | OWNER | I wonder if this will be a problem with generated columns, or with SQLite strict tables? My hunch is that strict tables will continue to work without any changes, because https://www.sqlite.org/stricttables.html says nothing about their impact on comparison operations. I should test this to make absolutely sure though. Generated columns have a type, so my hunch is they will continue to work fine too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 1174655187 | |
1074468450 | https://github.com/simonw/datasette/issues/1671#issuecomment-1074468450 | https://api.github.com/repos/simonw/datasette/issues/1671 | IC_kwDOBm6k_c5ACxZi | simonw 9599 | 2022-03-21T22:08:35Z | 2022-03-21T22:10:00Z | OWNER | Relevant section of the SQLite documentation: [3.2. Affinity Of Expressions](https://www.sqlite.org/datatype3.html#affinity_of_expressions): > When an expression is a simple reference to a column of a real table (not a [VIEW](https://www.sqlite.org/lang_createview.html) or subquery) then the expression has the same affinity as the table column. In your example, `has_expired` is no longer a simple reference to a column of a real table, hence the bug. Then [4.2. Type Conversions Prior To Comparison](https://www.sqlite.org/datatype3.html#type_conversions_prior_to_comparison) fills in the rest: > SQLite may attempt to convert values between the storage classes INTEGER, REAL, and/or TEXT before performing a comparison. Whether or not any conversions are attempted before the comparison takes place depends on the type affinity of the operands. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 1174655187 | |
1074465536 | https://github.com/simonw/datasette/issues/1671#issuecomment-1074465536 | https://api.github.com/repos/simonw/datasette/issues/1671 | IC_kwDOBm6k_c5ACwsA | simonw 9599 | 2022-03-21T22:04:31Z | 2022-03-21T22:04:31Z | OWNER | Oh this is fascinating! I replicated the bug (thanks for the steps to reproduce) and it looks like this is down to the following: <img width="1276" alt="image" src="https://user-images.githubusercontent.com/9599/159370986-1e2fc513-6d6c-4a2f-96dd-dccf5a680fe4.png"> Against views, `where has_expired = 1` returns different results from `where has_expired = '1'` This doesn't happen against tables because of SQLite's [type affinity](https://www.sqlite.org/datatype3.html#type_affinity) mechanism, which handles the type conversion automatically. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 1174655187 | |
1074459746 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074459746 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACvRi | simonw 9599 | 2022-03-21T21:55:45Z | 2022-03-21T21:55:45Z | OWNER | I'm going to change the original logic to set n=1 for times that are `<= 20ms` - and update the comments to make it more obvious what is happening. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074458506 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074458506 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACu-K | simonw 9599 | 2022-03-21T21:53:47Z | 2022-03-21T21:53:47Z | OWNER | Oh interesting, it turns out there is ONE place in the code that sets the `ms` to less than 20 - this test fixture: https://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/tests/fixtures.py#L224-L226 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074454687 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074454687 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACuCf | simonw 9599 | 2022-03-21T21:48:02Z | 2022-03-21T21:48:02Z | OWNER | Here's another microbenchmark that measures how many nanoseconds it takes to run 1,000 vmops: ```python import sqlite3 import time db = sqlite3.connect(":memory:") i = 0 out = [] def count(): global i i += 1000 out.append(((i, time.perf_counter_ns()))) db.set_progress_handler(count, 1000) print("Start:", time.perf_counter_ns()) all = db.execute(""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """).fetchall() print("End:", time.perf_counter_ns()) print() print("So how long does it take to execute 1000 ops?") prev_time_ns = None for i, time_ns in out: if prev_time_ns is not None: print(time_ns - prev_time_ns, "ns") prev_time_ns = time_ns ``` Running it: ``` % python nanobench.py Start: 330877620374821 End: 330877632515822 So how long does it take to execute 1000 ops? 47290 ns 49573 ns 48226 ns 45674 ns 53238 ns 47313 ns 52346 ns 48689 ns 47092 ns 87596 ns 69999 ns 52522 ns 52809 ns 53259 ns 52478 ns 53478 ns 65812 ns ``` 87596ns is 0.087596ms - so even a measure rate of every 1000 ops is easily finely grained enough to capture differences of less than 0.1ms. If anything I could bump that default 1000 up - and I can definitely eliminate the `if ms < 50` branch entirely. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074446576 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074446576 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACsDw | simonw 9599 | 2022-03-21T21:38:27Z | 2022-03-21T21:38:27Z | OWNER | OK here's a microbenchmark script: ```python import sqlite3 import timeit db = sqlite3.connect(":memory:") db_with_progress_handler_1 = sqlite3.connect(":memory:") db_with_progress_handler_1000 = sqlite3.connect(":memory:") db_with_progress_handler_1.set_progress_handler(lambda: None, 1) db_with_progress_handler_1000.set_progress_handler(lambda: None, 1000) def execute_query(db): cursor = db.execute(""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """) list(cursor.fetchall()) print("Without progress_handler") print(timeit.timeit(lambda: execute_query(db), number=100)) print("progress_handler every 1000 ops") print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1000), number=100)) print("progress_handler every 1 op") print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1), number=100)) ``` Results: ``` % python3 bench.py Without progress_handler 0.8789225700311363 progress_handler every 1000 ops 0.8829826560104266 progress_handler every 1 op 2.8892734259716235 ``` So running every 1000 ops makes almost no difference at all, but running every single op is a 3.2x performance degradation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074439309 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074439309 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACqSN | simonw 9599 | 2022-03-21T21:28:58Z | 2022-03-21T21:28:58Z | OWNER | David Raymond solved it there: https://sqlite.org/forum/forumpost/330c8532d8a88bcd > Don't forget to step through the results. All .execute() has done is prepared it. > > db.execute(query).fetchall() Sure enough, adding that gets the VM steps number up to 190,007 which is close enough that I'm happy. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074378472 | https://github.com/simonw/datasette/issues/1676#issuecomment-1074378472 | https://api.github.com/repos/simonw/datasette/issues/1676 | IC_kwDOBm6k_c5ACbbo | simonw 9599 | 2022-03-21T20:18:10Z | 2022-03-21T20:18:10Z | OWNER | Maybe there is a better name for this method that helps emphasize its cascading nature. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Reconsider ensure_permissions() logic, can it be less confusing? 1175690070 | |
1074347023 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074347023 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACTwP | simonw 9599 | 2022-03-21T19:48:59Z | 2022-03-21T19:48:59Z | OWNER | Posed a question about that here: https://sqlite.org/forum/forumpost/de9ff10fa7 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074341924 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074341924 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACSgk | simonw 9599 | 2022-03-21T19:42:08Z | 2022-03-21T19:42:08Z | OWNER | Here's the Python-C implementation of `set_progress_handler`: https://github.com/python/cpython/blob/4674fd4e938eb4a29ccd5b12c15455bd2a41c335/Modules/_sqlite/connection.c#L1177-L1201 It calls `sqlite3_progress_handler(self->db, n, progress_callback, ctx);` https://www.sqlite.org/c3ref/progress_handler.html says: > The parameter N is the approximate number of [virtual machine instructions](https://www.sqlite.org/opcode.html) that are evaluated between successive invocations of the callback X So maybe VM-steps and virtual machine instructions are different things? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074337997 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074337997 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACRjN | simonw 9599 | 2022-03-21T19:37:08Z | 2022-03-21T19:37:08Z | OWNER | This is weird: ```python import sqlite3 db = sqlite3.connect(":memory:") i = 0 def count(): global i i += 1 db.set_progress_handler(count, 1) db.execute(""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """) print(i) ``` Outputs `24`. But if you try the same thing in the SQLite console: ``` sqlite> .stats vmstep sqlite> with recursive counter(x) as ( ...> select 0 ...> union ...> select x + 1 from counter ...> ) ...> select * from counter limit 10000; ... VM-steps: 200007 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074332718 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074332718 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACQQu | simonw 9599 | 2022-03-21T19:31:10Z | 2022-03-21T19:31:10Z | OWNER | How long does it take for SQLite to execute 1000 opcodes anyway? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074332325 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074332325 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACQKl | simonw 9599 | 2022-03-21T19:30:44Z | 2022-03-21T19:30:44Z | OWNER | So it looks like even for facet suggestion `n=1000` always - it's never reduced to `n=1`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074331743 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074331743 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACQBf | simonw 9599 | 2022-03-21T19:30:05Z | 2022-03-21T19:30:05Z | OWNER | https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/app.py#L118-L122 sets it to 50ms for facet suggestion but that's not going to pass `ms < 50`: ```python Setting( "facet_suggest_time_limit_ms", 50, "Time limit for calculating a suggested facet", ), ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074321862 | https://github.com/simonw/datasette/issues/1660#issuecomment-1074321862 | https://api.github.com/repos/simonw/datasette/issues/1660 | IC_kwDOBm6k_c5ACNnG | simonw 9599 | 2022-03-21T19:19:01Z | 2022-03-21T19:19:01Z | OWNER | I've simplified this a ton now. I'm going to keep working on this in the long-term but I think this issue can be closed. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor and simplify Datasette routing and views 1170144879 | |
1074302559 | https://github.com/simonw/datasette/issues/1678#issuecomment-1074302559 | https://api.github.com/repos/simonw/datasette/issues/1678 | IC_kwDOBm6k_c5ACI5f | simonw 9599 | 2022-03-21T19:04:03Z | 2022-03-21T19:04:03Z | OWNER | Documentation: https://docs.datasette.io/en/latest/internals.html#await-check-visibility-actor-action-resource-none | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Make `check_visibility()` a documented API 1175715988 | |
1074287177 | https://github.com/simonw/datasette/issues/1660#issuecomment-1074287177 | https://api.github.com/repos/simonw/datasette/issues/1660 | IC_kwDOBm6k_c5ACFJJ | simonw 9599 | 2022-03-21T18:51:42Z | 2022-03-21T18:51:42Z | OWNER | `BaseView` is looking a LOT slimmer now that I've moved all of the permissions stuff out of it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor and simplify Datasette routing and views 1170144879 | |
1074256603 | https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074256603 | https://api.github.com/repos/simonw/sqlite-utils/issues/417 | IC_kwDOCGYnMM5AB9rb | blaine 9954 | 2022-03-21T18:19:41Z | 2022-03-21T18:19:41Z | NONE | That makes sense; just a little hint that points folks towards doing the right thing might be helpful! fwiw, the reason I was using jq in the first place was just a quick way to extract one attribute from an actual JSON array. When I initially imported it, I got a table with a bunch of embedded JSON values, rather than a native table, because each array entry had two attributes, one with the data I _actually_ wanted. Not sure how common a use-case this is, though (and easily fixed, aside from the jq weirdness!) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | insert fails on JSONL with whitespace 1175744654 | |
1074243540 | https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074243540 | https://api.github.com/repos/simonw/sqlite-utils/issues/417 | IC_kwDOCGYnMM5AB6fU | simonw 9599 | 2022-03-21T18:08:03Z | 2022-03-21T18:08:03Z | OWNER | I've not really thought about standards as much here as I should. It looks like there are two competing specs for newline-delimited JSON! http://ndjson.org/ is the one I've been using in `sqlite-utils` - and https://github.com/ndjson/ndjson-spec#31-serialization says: > The JSON texts MUST NOT contain newlines or carriage returns. https://jsonlines.org/ is the other one. It is slightly less clear, but it does say this: > 2. Each Line is a Valid JSON Value > > The most common values will be objects or arrays, but any JSON value is permitted. My interpretation of both of these is that newlines in the middle of a JSON object shouldn't be allowed. So what's `jq` doing here? It looks to me like that `jq` format is its own thing - it's not actually compatible with either of those two loose specs described above. The `jq` docs seem to call this "whitespace-separated JSON": https://stedolan.github.io/jq/manual/v1.6/#Invokingjq The thing I like about newline-delimited JSON is that it's really trivial to parse - loop through each line, run it through `json.loads()` and that's it. No need to try and unwrap JSON objects that might span multiple lines. Unless someone has written a robust Python implementation of a `jq`-compatible whitespace-separated JSON parser, I'm inclined to leave this as is. I'd be fine adding some documentation that helps point people towards `jq -c` though. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | insert fails on JSONL with whitespace 1175744654 | |
1074184240 | https://github.com/simonw/datasette/issues/1677#issuecomment-1074184240 | https://api.github.com/repos/simonw/datasette/issues/1677 | IC_kwDOBm6k_c5ABsAw | simonw 9599 | 2022-03-21T17:20:17Z | 2022-03-21T17:20:17Z | OWNER | https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/views/base.py#L69-L77 This is weirdly different from how `check_permissions()` used to work, in that it doesn't differentiate between `None` and `False`. https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L79-L103 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove `check_permission()` from `BaseView` 1175694248 | |
1074180312 | https://github.com/simonw/datasette/issues/1676#issuecomment-1074180312 | https://api.github.com/repos/simonw/datasette/issues/1676 | IC_kwDOBm6k_c5ABrDY | simonw 9599 | 2022-03-21T17:16:45Z | 2022-03-21T17:16:45Z | OWNER | When looking at this code earlier I assumed that the following would check each permission in turn and fail if any of them failed: ```python await self.ds.ensure_permissions( request.actor, [ ("view-table", (database, table)), ("view-database", database), "view-instance", ] ) ``` But it's not quite that simple: if any of them fail, it fails... but if an earlier one returns `True` the whole stack passes even if there would have been a failure later on! If that is indeed the right abstraction, I need to work to make the documentation as clear as possible. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Reconsider ensure_permissions() logic, can it be less confusing? 1175690070 | |
1074178865 | https://github.com/simonw/datasette/issues/1676#issuecomment-1074178865 | https://api.github.com/repos/simonw/datasette/issues/1676 | IC_kwDOBm6k_c5ABqsx | simonw 9599 | 2022-03-21T17:15:27Z | 2022-03-21T17:15:27Z | OWNER | This method here: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/app.py#L632-L664 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Reconsider ensure_permissions() logic, can it be less confusing? 1175690070 | |
1074177827 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074177827 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABqcj | simonw 9599 | 2022-03-21T17:14:31Z | 2022-03-21T17:14:31Z | OWNER | Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions > This method allows multiple permissions to be checked at onced. It raises a `datasette.Forbidden` exception if any of the checks are denied before one of them is explicitly granted. > > This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns `True` or not a single one of them returns `False`: That's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Extract out `check_permissions()` from `BaseView 1175648453 | |
1074161523 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074161523 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABmdz | simonw 9599 | 2022-03-21T16:59:55Z | 2022-03-21T17:00:03Z | OWNER | Also calling that function `permissions_allowed()` is confusing because there is a plugin hook with a similar name already: https://docs.datasette.io/en/stable/plugin_hooks.html#permission-allowed-datasette-actor-action-resource | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Extract out `check_permissions()` from `BaseView 1175648453 | |
1074158890 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074158890 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABl0q | simonw 9599 | 2022-03-21T16:57:15Z | 2022-03-21T16:57:15Z | OWNER | Idea: `ds.permission_allowed()` continues to just return `True` or `False`. A new `ds.ensure_permissions(...)` method is added which raises a `Forbidden` exception if a check fails (hence the different name)`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Extract out `check_permissions()` from `BaseView 1175648453 | |
1074156779 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074156779 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABlTr | simonw 9599 | 2022-03-21T16:55:08Z | 2022-03-21T16:56:02Z | OWNER | One benefit of the current design of `check_permissions` that raises an exception is that the exception includes information on WHICH of the permission checks failed. Returning just `True` or `False` loses that information. I could return an object which evaluates to `False` but also carries extra information? Bit weird, I've never seen anything like that in other Python code. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Extract out `check_permissions()` from `BaseView 1175648453 | |
1074143209 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074143209 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABh_p | simonw 9599 | 2022-03-21T16:46:05Z | 2022-03-21T16:46:05Z | OWNER | The other difference though is that `ds.permission_allowed(...)` works against an actor, while `check_permission()` works against a request (though just to access `request.actor`). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Extract out `check_permissions()` from `BaseView 1175648453 | |
1074142617 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074142617 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABh2Z | simonw 9599 | 2022-03-21T16:45:27Z | 2022-03-21T16:45:27Z | OWNER | Though at that point `check_permission` is such a light wrapper around `self.ds.permission_allowed()` that there's little point in it existing at all. So maybe `check_permisions()` becomes `ds.permissions_allowed()`. `permission_allowed()` v.s. `permissions_allowed()` is a bit of a subtle naming difference, but I think it works. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Extract out `check_permissions()` from `BaseView 1175648453 | |
1074141457 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074141457 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABhkR | simonw 9599 | 2022-03-21T16:44:09Z | 2022-03-21T16:44:09Z | OWNER | A slightly odd thing about these methods is that they either fail silently or they raise a `Forbidden` exception. Maybe they should instead return `True` or `False` and the calling code could decide if it wants to raise the exception? That would make them more usable and a little less surprising. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Extract out `check_permissions()` from `BaseView 1175648453 | |
1074136176 | https://github.com/simonw/datasette/issues/1660#issuecomment-1074136176 | https://api.github.com/repos/simonw/datasette/issues/1660 | IC_kwDOBm6k_c5ABgRw | simonw 9599 | 2022-03-21T16:38:46Z | 2022-03-21T16:38:46Z | OWNER | I'm going to refactor this stuff out and document it so it can be easily used by plugins: https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L69-L103 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor and simplify Datasette routing and views 1170144879 | |
1074019047 | https://github.com/simonw/datasette/issues/526#issuecomment-1074019047 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5ABDrn | simonw 9599 | 2022-03-21T15:09:56Z | 2022-03-21T15:09:56Z | OWNER | I should research how much overhead creating a new connection costs - it may be that an easy way to solve this is to create A dedicated connection for the query and then close that connection at the end. | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Stream all results for arbitrary SQL and canned queries 459882902 | |
1074017633 | https://github.com/simonw/datasette/issues/1177#issuecomment-1074017633 | https://api.github.com/repos/simonw/datasette/issues/1177 | IC_kwDOBm6k_c5ABDVh | simonw 9599 | 2022-03-21T15:08:51Z | 2022-03-21T15:08:51Z | OWNER | Related: - #1062 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to stream all rows as newline-delimited JSON 780153562 | |
1073468996 | https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073468996 | https://api.github.com/repos/simonw/sqlite-utils/issues/415 | IC_kwDOCGYnMM4_-9ZE | simonw 9599 | 2022-03-21T04:14:42Z | 2022-03-21T04:14:42Z | OWNER | I can fix this like so: ``` % sqlite-utils convert demo.db demo foo '{"foo": "bar"}' --multi --dry-run abc --- becomes: {"foo": "bar"} Would affect 1 row ``` Diff is this: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 0cf0468..b2a0440 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -2676,7 +2676,10 @@ def convert( raise click.ClickException(str(e)) if dry_run: # Pull first 20 values for first column and preview them - db.conn.create_function("preview_transform", 1, lambda v: fn(v) if v else v) + preview = lambda v: fn(v) if v else v + if multi: + preview = lambda v: json.dumps(fn(v), default=repr) if v else v + db.conn.create_function("preview_transform", 1, preview) sql = """ select [{column}] as value, ``` | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Convert with `--multi` and `--dry-run` flag does not work 1171599874 | |
1073463375 | https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073463375 | https://api.github.com/repos/simonw/sqlite-utils/issues/415 | IC_kwDOCGYnMM4_-8BP | simonw 9599 | 2022-03-21T04:02:36Z | 2022-03-21T04:02:36Z | OWNER | Thanks for the really clear steps to reproduce! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Convert with `--multi` and `--dry-run` flag does not work 1171599874 | |
1073456222 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456222 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-6Re | simonw 9599 | 2022-03-21T03:45:52Z | 2022-03-21T03:45:52Z | OWNER | Needs tests and documentation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073456155 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456155 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-6Qb | simonw 9599 | 2022-03-21T03:45:37Z | 2022-03-21T03:45:37Z | OWNER | Prototype: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 8255b56..0a3693e 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -2583,7 +2583,11 @@ def _generate_convert_help(): """ ).strip() recipe_names = [ - n for n in dir(recipes) if not n.startswith("_") and n not in ("json", "parser") + n + for n in dir(recipes) + if not n.startswith("_") + and n not in ("json", "parser") + and callable(getattr(recipes, n)) ] for name in recipe_names: fn = getattr(recipes, name) diff --git a/sqlite_utils/recipes.py b/sqlite_utils/recipes.py index 6918661..569c30d 100644 --- a/sqlite_utils/recipes.py +++ b/sqlite_utils/recipes.py @@ -1,17 +1,38 @@ from dateutil import parser import json +IGNORE = object() +SET_NULL = object() -def parsedate(value, dayfirst=False, yearfirst=False): + +def parsedate(value, dayfirst=False, yearfirst=False, errors=None): "Parse a date and convert it to ISO date format: yyyy-mm-dd" - return ( - parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() - ) + try: + return ( + parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst) + .date() + .isoformat() + ) + except parser.ParserError: + if errors is IGNORE: + return value + elif errors is SET_NULL: + return None + else: + raise -def parsedatetime(value, dayfirst=False, yearfirst=False): +def parsedatetime(value, dayfirst=False, yearfirst=False, errors=None): "Parse a datetime and convert it to ISO datetime format: yyyy-mm-ddTHH:MM:SS" - return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + try: + return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + except parser.ParserError: + if errors is IGNORE: + return value + elif erro… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073455905 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073455905 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-6Mh | simonw 9599 | 2022-03-21T03:44:47Z | 2022-03-21T03:45:00Z | OWNER | This is quite nice: ``` % sqlite-utils convert test-dates.db dates date "r.parsedate(value, errors=r.IGNORE)" [####################################] 100% % sqlite-utils rows test-dates.db dates [{"id": 1, "date": "2016-03-15"}, {"id": 2, "date": "2016-03-16"}, {"id": 3, "date": "2016-03-17"}, {"id": 4, "date": "2016-03-18"}, {"id": 5, "date": "2016-03-19"}, {"id": 6, "date": "2016-03-20"}, {"id": 7, "date": "2016-03-21"}, {"id": 8, "date": "2016-03-22"}, {"id": 9, "date": "2016-03-23"}, {"id": 10, "date": "//"}, {"id": 11, "date": "2016-03-25"}, {"id": 12, "date": "2016-03-26"}, {"id": 13, "date": "2016-03-27"}, {"id": 14, "date": "2016-03-28"}, {"id": 15, "date": "2016-03-29"}, {"id": 16, "date": "2016-03-30"}, {"id": 17, "date": "2016-03-31"}, {"id": 18, "date": "2016-04-01"}] % sqlite-utils convert test-dates.db dates date "r.parsedate(value, errors=r.SET_NULL)" [####################################] 100% % sqlite-utils rows test-dates.db dates [{"id": 1, "date": "2016-03-15"}, {"id": 2, "date": "2016-03-16"}, {"id": 3, "date": "2016-03-17"}, {"id": 4, "date": "2016-03-18"}, {"id": 5, "date": "2016-03-19"}, {"id": 6, "date": "2016-03-20"}, {"id": 7, "date": "2016-03-21"}, {"id": 8, "date": "2016-03-22"}, {"id": 9, "date": "2016-03-23"}, {"id": 10, "date": null}, {"id": 11, "date": "2016-03-25"}, {"id": 12, "date": "2016-03-26"}, {"id": 13, "date": "2016-03-27"}, {"id": 14, "date": "2016-03-28"}, {"id": 15, "date": "2016-03-29"}, {"id": 16, "date": "2016-03-30"}, {"id": 17, "date": "2016-03-31"}, {"id": 18, "date": "2016-04-01"}] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073453370 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453370 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-5k6 | simonw 9599 | 2022-03-21T03:41:06Z | 2022-03-21T03:41:06Z | OWNER | I'm going to try the `errors=r.IGNORE` option and see what that looks like once implemented. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073453230 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453230 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-5iu | simonw 9599 | 2022-03-21T03:40:37Z | 2022-03-21T03:40:37Z | OWNER | I think the options here should be: - On error, raise an exception and revert the transaction (the current default) - On error, leave the value as-is - On error, set the value to `None` These need to be indicated by parameters to the `r.parsedate()` function. Some design options: - `ignore=True` to ignore errors - but how does it know if it should leave the value or set it to `None`? This is similar to other `ignore=True` parameters elsewhere in the Python API. - `errors="ignore"`, `errors="set-null"` - I don't like magic string values very much, but this is similar to Python's `str.encode(errors=)` mechanism - `errors=r.IGNORE` - using constants, which at least avoids magic strings. The other one could be `errors=r.SET_NULL` - `error=lambda v: None` or `error=lambda v: v` - this is a bit confusing though, introducing another callback that gets to have a go at converting the error if the first callback failed? And what happens if that lambda itself raises an error? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073451659 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073451659 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-5KL | simonw 9599 | 2022-03-21T03:35:01Z | 2022-03-21T03:35:01Z | OWNER | I confirmed that if it fails for any value ALL values are left alone, since it runs in a transaction. Here's the code that does that: https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L2523-L2526 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073450588 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073450588 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-45c | simonw 9599 | 2022-03-21T03:32:58Z | 2022-03-21T03:32:58Z | OWNER | Then I ran this to convert `2016-03-27` etc to `2016/03/27` so I could see which ones were later converted: sqlite-utils convert test-dates.db dates date 'value.replace("-", "/")' | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073448904 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073448904 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-4fI | simonw 9599 | 2022-03-21T03:28:12Z | 2022-03-21T03:30:37Z | OWNER | Generating a test database using a pattern from https://www.geekytidbits.com/date-range-table-sqlite/ ``` sqlite-utils create-database test-dates.db sqlite-utils create-table test-dates.db dates id integer date text --pk id sqlite-utils test-dates.db "WITH RECURSIVE cnt(x) AS ( SELECT 0 UNION ALL SELECT x+1 FROM cnt LIMIT (SELECT ((julianday('2016-04-01') - julianday('2016-03-15'))) + 1) ) insert into dates (date) select date(julianday('2016-03-15'), '+' || x || ' days') as date FROM cnt;" ``` After running that: ``` % sqlite-utils rows test-dates.db dates [{"id": 1, "date": "2016-03-15"}, {"id": 2, "date": "2016-03-16"}, {"id": 3, "date": "2016-03-17"}, {"id": 4, "date": "2016-03-18"}, {"id": 5, "date": "2016-03-19"}, {"id": 6, "date": "2016-03-20"}, {"id": 7, "date": "2016-03-21"}, {"id": 8, "date": "2016-03-22"}, {"id": 9, "date": "2016-03-23"}, {"id": 10, "date": "2016-03-24"}, {"id": 11, "date": "2016-03-25"}, {"id": 12, "date": "2016-03-26"}, {"id": 13, "date": "2016-03-27"}, {"id": 14, "date": "2016-03-28"}, {"id": 15, "date": "2016-03-29"}, {"id": 16, "date": "2016-03-30"}, {"id": 17, "date": "2016-03-31"}, {"id": 18, "date": "2016-04-01"}] ``` Then to make one of them invalid: sqlite-utils test-dates.db "update dates set date = '//' where id = 10" | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073366630 | https://github.com/simonw/datasette/issues/1510#issuecomment-1073366630 | https://api.github.com/repos/simonw/datasette/issues/1510 | IC_kwDOBm6k_c4_-kZm | simonw 9599 | 2022-03-20T22:59:33Z | 2022-03-20T22:59:33Z | OWNER | I really like the idea of making this effectively the same thing as the fully documented, stable JSON API that comes as part of 1.0. If you want to know what will be available to your templates, consult the API documentation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Datasette 1.0 documented template context (maybe via API docs) 1054244712 | |
1073366436 | https://github.com/simonw/datasette/issues/1674#issuecomment-1073366436 | https://api.github.com/repos/simonw/datasette/issues/1674 | IC_kwDOBm6k_c4_-kWk | simonw 9599 | 2022-03-20T22:58:40Z | 2022-03-20T22:58:40Z | OWNER | This will probably happen as part of turning this into an officially documented API that serves the template context for the homepage: - #1510 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tweak design of /.json 1174717287 | |
1073362979 | https://github.com/simonw/datasette/issues/1355#issuecomment-1073362979 | https://api.github.com/repos/simonw/datasette/issues/1355 | IC_kwDOBm6k_c4_-jgj | simonw 9599 | 2022-03-20T22:38:53Z | 2022-03-20T22:38:53Z | OWNER | Built a research prototype: ```diff diff --git a/datasette/app.py b/datasette/app.py index 5c8101a..5cd3e63 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1,6 +1,7 @@ import asyncio import asgi_csrf import collections +import contextlib import datetime import functools import glob @@ -1490,3 +1491,11 @@ class DatasetteClient: return await client.request( method, self._fix(path, avoid_path_rewrites), **kwargs ) + + @contextlib.asynccontextmanager + async def stream(self, method, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + print("async with as client") + async with client.stream(method, self._fix(path), **kwargs) as response: + print("async with client.stream about to yield response") + yield response diff --git a/datasette/cli.py b/datasette/cli.py index 3c6e1b2..3025ead 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -585,11 +585,19 @@ def serve( asyncio.get_event_loop().run_until_complete(check_databases(ds)) if get: - client = TestClient(ds) - response = client.get(get) - click.echo(response.text) - exit_code = 0 if response.status == 200 else 1 - sys.exit(exit_code) + + async def _run_get(): + print("_run_get") + async with ds.client.stream("GET", get) as response: + print("Got response:", response) + async for chunk in response.aiter_bytes(chunk_size=1024): + print(" chunk") + sys.stdout.buffer.write(chunk) + sys.stdout.buffer.flush() + exit_code = 0 if response.status_code == 200 else 1 + sys.exit(exit_code) + + asyncio.get_event_loop().run_until_complete(_run_get()) return # Start the server ``` But for some reason it didn't appear to stream out the response - it would print thi… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | datasette --get should efficiently handle streaming CSV 910088936 | |
1073361986 | https://github.com/simonw/datasette/issues/1673#issuecomment-1073361986 | https://api.github.com/repos/simonw/datasette/issues/1673 | IC_kwDOBm6k_c4_-jRC | simonw 9599 | 2022-03-20T22:31:41Z | 2022-03-20T22:34:06Z | OWNER | Maybe it's because `supports_table_xinfo()` creates a brand new in-memory SQLite connection every time you call it? https://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/sqlite.py#L22-L35 Actually no, I'm caching that already: https://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/sqlite.py#L12-L19 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Streaming CSV spends a lot of time in `table_column_details` 1174708375 | |
1073355818 | https://github.com/simonw/datasette/issues/1672#issuecomment-1073355818 | https://api.github.com/repos/simonw/datasette/issues/1672 | IC_kwDOBm6k_c4_-hwq | simonw 9599 | 2022-03-20T21:52:38Z | 2022-03-20T21:52:38Z | OWNER | That means taking on these issues: - https://github.com/simonw/datasette/issues/1101 - https://github.com/simonw/datasette/issues/1096 - https://github.com/simonw/datasette/issues/1062 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor CSV handling code out of DataView 1174697144 | |
1073355032 | https://github.com/simonw/datasette/issues/1660#issuecomment-1073355032 | https://api.github.com/repos/simonw/datasette/issues/1660 | IC_kwDOBm6k_c4_-hkY | simonw 9599 | 2022-03-20T21:46:43Z | 2022-03-20T21:46:43Z | OWNER | I think the way to get rid of most of the remaining complexity in `DataView` is to refactor how CSV stuff works - pulling it in line with other export factors and extracting the streaming mechanism. Opening a fresh issue for that. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor and simplify Datasette routing and views 1170144879 | |
1073330388 | https://github.com/simonw/sqlite-utils/issues/140#issuecomment-1073330388 | https://api.github.com/repos/simonw/sqlite-utils/issues/140 | IC_kwDOCGYnMM4_-bjU | simonw 9599 | 2022-03-20T19:44:39Z | 2022-03-20T19:45:45Z | OWNER | Alternative idea for specifying types: accept a Python expression, then use Python type literal syntax. For example: ``` sqlite-utils insert-files gifs.db images *.gif \ -c path -c md5 -c last_modified:mtime \ -a file_type '"gif"' ``` Where `-a` indicates an additional column. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Idea: insert-files mechanism for adding extra columns with fixed values 688351054 | |
1073152522 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/10#issuecomment-1073152522 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/10 | IC_kwDODFE5qs4_9wIK | csusanu 9290214 | 2022-03-20T02:38:07Z | 2022-03-20T02:38:07Z | NONE | [This line](https://github.com/dogsheep/google-takeout-to-sqlite/blob/e54e544427f1cc3ea8189f0e95f54046301a8645/google_takeout_to_sqlite/utils.py) needs to say `"MyActivity.json"` instead of `"My Activity.json"`. Google must have changed the file name. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite3.OperationalError: no such table: main.my_activity 1123393829 | |
1073143413 | https://github.com/simonw/datasette/issues/1669#issuecomment-1073143413 | https://api.github.com/repos/simonw/datasette/issues/1669 | IC_kwDOBm6k_c4_9t51 | simonw 9599 | 2022-03-20T01:24:36Z | 2022-03-20T01:24:36Z | OWNER | https://github.com/simonw/datasette/releases/tag/0.61a0 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Release 0.61 alpha 1174404647 | |
1073139067 | https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1073139067 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14 | IC_kwDOC8tyDs4_9s17 | lchski 343884 | 2022-03-20T00:54:18Z | 2022-03-20T00:54:18Z | NONE | Update: this appears to be because of running the command twice without clearing the DB in between. Tries to insert a Workout that already exists, causing a collision on the (auto-generated) `id` column. Had a different error with a clean DB, likely due to the workout points format; will make a new issue for that. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | UNIQUE constraint failed: workouts.id 771608692 | |
1073137170 | https://github.com/simonw/datasette/issues/1669#issuecomment-1073137170 | https://api.github.com/repos/simonw/datasette/issues/1669 | IC_kwDOBm6k_c4_9sYS | simonw 9599 | 2022-03-20T00:35:52Z | 2022-03-20T00:35:52Z | OWNER | https://github.com/simonw/datasette/compare/0.60.2...main | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Release 0.61 alpha 1174404647 | |
1073136896 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073136896 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9sUA | simonw 9599 | 2022-03-20T00:33:23Z | 2022-03-20T00:33:23Z | OWNER | I'm going to release this as a 0.61 alpha so I can more easily depend on it from `datasette-hashed-urls`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073136686 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073136686 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9sQu | simonw 9599 | 2022-03-20T00:31:13Z | 2022-03-20T00:31:13Z | OWNER | That demo is now live: - https://latest.datasette.io/alternative-route - https://latest.datasette.io/alternative-route/attraction_characteristic | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073135433 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073135433 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9r9J | simonw 9599 | 2022-03-20T00:20:36Z | 2022-03-20T00:20:36Z | OWNER | Building this plugin instantly revealed that all of the links - on the homepage and the database page and so on - are incorrect: ```python from datasette import hookimpl @hookimpl def startup(datasette): db = datasette.get_database("fixtures2") db.route = "alternative-route" ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073134816 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073134816 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9rzg | simonw 9599 | 2022-03-20T00:16:22Z | 2022-03-20T00:16:22Z | OWNER | I'm going to add a `fixtures2.db` database which has that as the name but `alternative-route` as the route. I'll set that up using a custom plugin in the `plugins/` folder that gets deployed by https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073134206 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073134206 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9rp- | simonw 9599 | 2022-03-20T00:12:03Z | 2022-03-20T00:12:03Z | OWNER | I'd like to have a live demo of this up on `latest.datasette.io` too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073126264 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073126264 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9pt4 | simonw 9599 | 2022-03-19T22:59:30Z | 2022-03-19T22:59:30Z | OWNER | Also need to update the `datasette.urls` methods that construct the URL to a database/table/row - they take the database name but they need to know to look for the route. Need to add tests that check the links in the HTML and can confirm this is working correctly. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073125334 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073125334 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9pfW | simonw 9599 | 2022-03-19T22:53:55Z | 2022-03-19T22:53:55Z | OWNER | Need to update documentation in a few places - e.g. https://docs.datasette.io/en/stable/internals.html#remove-database-name > This removes a database that has been previously added. `name=` is the unique name of that database, used in its URL path. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073123231 | https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1073123231 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14 | IC_kwDOC8tyDs4_9o-f | lchski 343884 | 2022-03-19T22:39:29Z | 2022-03-19T22:39:29Z | NONE | I have this issue, too, with a fresh export. None of my `Workout` entries in `export.xml` have an `id` key, though [the sample `export.xml` in the tests folder doesn’t either](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/tests/zip_contents/apple_health_export/export.xml#L14-L21), so I don’t think this is the culprit. Indeed, it seems @simonw is using the [`hash_id` function from `sqlite_utils`](https://sqlite-utils.datasette.io/en/stable/python-api.html#setting-an-id-based-on-the-hash-of-the-row-contents), which creates a column (`id`, in this case) based on a hash of the row’s contents. When I run the script, a `workouts` table is created, with one entry: my first workout. No `workout_points` table is created, as [I’d expect from `utils.py`](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/healthkit_to_sqlite/utils.py#L89-L90). I then get essentially the same error as noted in this thread: ```Importing from HealthKit [###################################-] 98% 00:00:01 Traceback (most recent call last): File "/Users/lchski/.pyenv/versions/3.10.3/bin/healthkit-to-sqlite", line 8, in <module> sys.exit(cli()) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/cli.py", line 57, in cli convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/health… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | UNIQUE constraint failed: workouts.id 771608692 | |
1073112104 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073112104 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9mQo | simonw 9599 | 2022-03-19T21:08:21Z | 2022-03-19T21:08:21Z | OWNER | I think I've got this working but I need to write a test for it that covers the rare case when the route is not the same thing as the database name. I'll do that with a new test. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073097394 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073097394 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9iqy | simonw 9599 | 2022-03-19T20:56:35Z | 2022-03-19T20:56:35Z | OWNER | I'm trying to think if there's any reason not to use `route` for this. Would I possibly want to use that noun for something else in the future? I like it more than `route_path` because it has no underscore. Decision made: I'm going with `route`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073076624 | https://github.com/simonw/datasette/issues/1667#issuecomment-1073076624 | https://api.github.com/repos/simonw/datasette/issues/1667 | IC_kwDOBm6k_c4_9dmQ | simonw 9599 | 2022-03-19T20:31:44Z | 2022-03-19T20:31:44Z | OWNER | I can now read `format` from `request.url_vars` and delete this code entirely: https://github.com/simonw/datasette/blob/b9c2b1cfc8692b9700416db98721fa3ec982f6be/datasette/views/base.py#L375-L381 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Make route matched pattern groups more consistent 1174302994 | |
1073076187 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073076187 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9dfb | simonw 9599 | 2022-03-19T20:28:20Z | 2022-03-19T20:28:20Z | OWNER | I'm going to keep `path` as the path to the file on disk. I'll pick a new name for what is currently `path` in that undocumented JSON API. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073076136 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073076136 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9deo | simonw 9599 | 2022-03-19T20:27:44Z | 2022-03-19T20:27:44Z | OWNER | Pretty sure changing it will break some existing plugins though, including likely Datasette Desktop. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073076110 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073076110 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9deO | simonw 9599 | 2022-03-19T20:27:22Z | 2022-03-19T20:27:22Z | OWNER | The docs do currently describe `path` as the filesystem path here: https://docs.datasette.io/en/stable/internals.html#database-class <img width="720" alt="image" src="https://user-images.githubusercontent.com/9599/159137373-5bbc7fea-7544-42b3-8423-be4e11eb4d52.png"> Good thing I'm not at 1.0 yet so I can change that! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073076015 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073076015 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9dcv | simonw 9599 | 2022-03-19T20:26:32Z | 2022-03-19T20:26:32Z | OWNER | I'm inclined to redefine `ds.path` to `ds.file_path` to fix this. Or `ds.filepath`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073075913 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073075913 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9dbJ | simonw 9599 | 2022-03-19T20:25:46Z | 2022-03-19T20:26:08Z | OWNER | The output of `/.json` DOES use `path` to mean the URL path, not the path to the file on disk: ``` { "fixtures.dot": { "name": "fixtures.dot", "hash": null, "color": "631f11", "path": "/fixtures~2Edot", ``` So that's a problem already: having `db.path` refer to something different from that JSON is inconsistent. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073075697 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073075697 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9dXx | simonw 9599 | 2022-03-19T20:24:06Z | 2022-03-19T20:24:06Z | OWNER | Right now if a database has a `.` in its name e.g. `fixtures.dot` the URL to that database is: /fixtures~2Edot But the output on `/-/databases` doesn't reflect that, it still shows the name with the dot. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073073599 | https://github.com/simonw/datasette/issues/1660#issuecomment-1073073599 | https://api.github.com/repos/simonw/datasette/issues/1660 | IC_kwDOBm6k_c4_9c2_ | simonw 9599 | 2022-03-19T20:06:40Z | 2022-03-19T20:06:40Z | OWNER | This blocks: - #1668 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor and simplify Datasette routing and views 1170144879 | |
1073073579 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073073579 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9c2r | simonw 9599 | 2022-03-19T20:06:27Z | 2022-03-19T20:06:27Z | OWNER | Marking this as blocked until #1660 is done. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073073547 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073073547 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9c2L | simonw 9599 | 2022-03-19T20:06:07Z | 2022-03-19T20:06:07Z | OWNER | Implementing this is a little tricky because there's a whole lot of code that expects the `database` captured by the URL routing to be the name used to look up the database in `datasette.databases` - or via `.get_database()`. The `DataView.get()` method is a good example of the trickyness here. It even has code that dispatches out to plugin hooks that take `database` as a parameter. https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/views/base.py#L383-L555 All the more reason to get rid of that `BaseView -> DataView -> TableView` hierarchy entirely: - #1660 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073043713 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073043713 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9VkB | simonw 9599 | 2022-03-19T16:56:19Z | 2022-03-19T16:56:19Z | OWNER | Worth noting that the `name` right now is picked automatically to avoid conflicts: https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/app.py#L397-L413 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073043433 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073043433 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9Vfp | simonw 9599 | 2022-03-19T16:54:55Z | 2022-03-19T20:01:19Z | OWNER | Options: - `route_path` - `url_path` - `route` I like `route_path`, or maybe `route`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073043350 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073043350 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9VeW | simonw 9599 | 2022-03-19T16:54:26Z | 2022-03-19T16:54:26Z | OWNER | The `Database` class already has a `path` property but it means something else - it's the path to the `.db` file on disk: https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/database.py#L29-L50 So need a different name for the path-that-is-used-in-the-URL. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Introduce concept of a database `route`, separate from its name 1174306154 | |
1073042554 | https://github.com/simonw/datasette/issues/1667#issuecomment-1073042554 | https://api.github.com/repos/simonw/datasette/issues/1667 | IC_kwDOBm6k_c4_9VR6 | simonw 9599 | 2022-03-19T16:50:01Z | 2022-03-19T16:52:35Z | OWNER | OK, I've made this more consistent - I still need to address the fact that `format` can be `.json` or `json` or not used at all before I close this issue. https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/tests/test_routes.py#L15-L35 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Make route matched pattern groups more consistent 1174302994 | |
1073040072 | https://github.com/simonw/datasette/issues/1667#issuecomment-1073040072 | https://api.github.com/repos/simonw/datasette/issues/1667 | IC_kwDOBm6k_c4_9UrI | simonw 9599 | 2022-03-19T16:34:02Z | 2022-03-19T16:34:02Z | OWNER | I called it `as_format` to avoid clashing with the Python built-in `format()` function when these things were turned into keyword arguments, but now that they're not I can use `format` instead. I think I'm going to go with `database`, `table`, `format` and `pks`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Make route matched pattern groups more consistent 1174302994 | |
1073039670 | https://github.com/simonw/datasette/issues/1666#issuecomment-1073039670 | https://api.github.com/repos/simonw/datasette/issues/1666 | IC_kwDOBm6k_c4_9Uk2 | simonw 9599 | 2022-03-19T16:31:08Z | 2022-03-19T16:31:57Z | OWNER | This does make it more interesting - it also highlights how inconsistent the way the capturing works is. Especially `as_format` which can be `None` or `""` or `.json` or `json` or not used at all in the case of `TableView`. https://github.com/simonw/datasette/blob/764738dfcb16cd98b0987d443f59d5baa9d3c332/tests/test_routes.py#L12-L36 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor URL routing to enable testing 1174162781 | |
1073039241 | https://github.com/simonw/datasette/issues/1666#issuecomment-1073039241 | https://api.github.com/repos/simonw/datasette/issues/1666 | IC_kwDOBm6k_c4_9UeJ | simonw 9599 | 2022-03-19T16:28:15Z | 2022-03-19T16:28:15Z | OWNER | This is more interesting if it also asserts against the captured matches from the pattern. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor URL routing to enable testing 1174162781 | |
1073037939 | https://github.com/simonw/datasette/issues/878#issuecomment-1073037939 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c4_9UJz | simonw 9599 | 2022-03-19T16:19:30Z | 2022-03-19T16:19:30Z | OWNER | On revisiting https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 a few months later I'm having second thoughts about using `@inject` on the `main()` method. But I still like the pattern as a way to resolve more complex cases like "to generate GeoJSON of the expanded view with labels, the label expansion code needs to run once at some before the GeoJSON formatting code does". So I'm going to stick with it a tiny bit longer, but maybe try to make it a lot more explicit when it's going to happen rather than having the main view methods themselves also use async DI. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
1072954795 | https://github.com/simonw/datasette/issues/1228#issuecomment-1072954795 | https://api.github.com/repos/simonw/datasette/issues/1228 | IC_kwDOBm6k_c4_8_2r | Kabouik 7107523 | 2022-03-19T06:44:40Z | 2022-03-19T06:44:40Z | NONE | > ... unless your data had a column called `n`? Exactly, that's highly likely even though I can't double check from this computer just now. Thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 500 error caused by faceting if a column called `n` exists 810397025 | |
1072939780 | https://github.com/simonw/datasette/issues/1561#issuecomment-1072939780 | https://api.github.com/repos/simonw/datasette/issues/1561 | IC_kwDOBm6k_c4_88ME | simonw 9599 | 2022-03-19T04:45:40Z | 2022-03-19T04:45:40Z | OWNER | I ended up moving hashed URL mode out to a plugin in: - #647 If you're still interested in using it with `_memory` please open an issue in that repo here: https://github.com/simonw/datasette-hashed-urls | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | add hash id to "_memory" url if hashed url mode is turned on and crossdb is also turned on 1082765654 | |
1072933875 | https://github.com/simonw/datasette/issues/1666#issuecomment-1072933875 | https://api.github.com/repos/simonw/datasette/issues/1666 | IC_kwDOBm6k_c4_86vz | simonw 9599 | 2022-03-19T04:03:42Z | 2022-03-19T04:03:42Z | OWNER | Tests so far: https://github.com/simonw/datasette/blob/711767bcd3c1e76a0861fe7f24069ff1c8efc97a/tests/test_routes.py#L12-L34 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor URL routing to enable testing 1174162781 | |
1072915936 | https://github.com/simonw/datasette/issues/1228#issuecomment-1072915936 | https://api.github.com/repos/simonw/datasette/issues/1228 | IC_kwDOBm6k_c4_82Xg | simonw 9599 | 2022-03-19T01:50:27Z | 2022-03-19T01:50:27Z | OWNER | Demo: https://latest.datasette.io/fixtures/facetable - which now has a column called `n`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 500 error caused by faceting if a column called `n` exists 810397025 | |
1072908029 | https://github.com/simonw/datasette/issues/1228#issuecomment-1072908029 | https://api.github.com/repos/simonw/datasette/issues/1228 | IC_kwDOBm6k_c4_80b9 | simonw 9599 | 2022-03-19T00:57:54Z | 2022-03-19T00:57:54Z | OWNER | Yes! That's the problem. I was able to replicate it like so: ``` echo '[{ "n": "one", "abc": 1 }, { "n": "one", "abc": 2 }, { "n": "two", "abc": 3 }]' | sqlite-utils insert column-called-n.db t - ``` <img width="564" alt="image" src="https://user-images.githubusercontent.com/9599/159100494-9a072e3c-7bad-4fc5-b90e-b5078c11fc44.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 500 error caused by faceting if a column called `n` exists 810397025 | |
1072907680 | https://github.com/simonw/datasette/issues/1228#issuecomment-1072907680 | https://api.github.com/repos/simonw/datasette/issues/1228 | IC_kwDOBm6k_c4_80Wg | simonw 9599 | 2022-03-19T00:55:48Z | 2022-03-19T00:55:48Z | OWNER | ... unless your data had a column called `n`? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 500 error caused by faceting if a column called `n` exists 810397025 | |
1072907610 | https://github.com/simonw/datasette/issues/1228#issuecomment-1072907610 | https://api.github.com/repos/simonw/datasette/issues/1228 | IC_kwDOBm6k_c4_80Va | simonw 9599 | 2022-03-19T00:55:29Z | 2022-03-19T00:55:29Z | OWNER | It looks to me like something is causing the faceting query here to return a string when it was expected to return a number: https://github.com/simonw/datasette/blob/32963018e7edfab1233de7c7076c428d0e5c7813/datasette/facets.py#L153-L170 I can't think of any way that a `count(*) as n` would turn into a string though! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 500 error caused by faceting if a column called `n` exists 810397025 | |
1072907200 | https://github.com/simonw/datasette/issues/1605#issuecomment-1072907200 | https://api.github.com/repos/simonw/datasette/issues/1605 | IC_kwDOBm6k_c4_80PA | simonw 9599 | 2022-03-19T00:52:54Z | 2022-03-19T00:53:45Z | OWNER | Had a thought about the implementation of this: it could make a really neat plugin. Something like `datasette-export` which adds a `export` command using https://docs.datasette.io/en/stable/plugin_hooks.html#register-commands-cli - then you could run: datasette export my-export-dir mydatabase.db -m metadata.json --template-dir templates/ And the command would then: - Create a `Datasette()` instance with those databases/metadata/etc - Execute`await datasette.client.get("/")` to get the homepage HTML - Parse the HTML using BeautifulSoup to find all `a[href]`, `link[href]`, `script[src]`, `img[src]` elements that reference a relative path as opposed to one that starts with `http://` - Write out the homepage to `my-export-dir/index.html` - Recursively fetch and dump all of the other pages and assets that it found too All of that HTML parsing may be over-complicating things. It could alternatively accept options for which pages you want to export: ``` datasette export my-export-dir \ mydatabase.db -m metadata.json --template-dir templates/ \ --path / \ --path /mydatabase ... ``` Or a really wild option: it could allow you to define the paths you want to export using a SQL query: ``` datasette export my-export-dir \ mydatabase.db -m metadata.json --template-dir templates/ \ --sql " select '/' as path, 'index.html' as filename union all select '/mydatabase/articles/' || id as path, 'article-' || id || '.html' as filename from articles union all select '/mydatabase/tags/' || tag as path, 'tag-' || tag || '.html' as filename from tags " ``` Which would save these files: - `index.html` as the content of `/` - `article-1.html` (and more) as the content of `/mydatabase/articles/1` - `tag-python.html` (and more) as the content of `/mydatabase/tags/python` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Scripted exports 1108671952 | |
1072905467 | https://github.com/simonw/datasette/issues/1662#issuecomment-1072905467 | https://api.github.com/repos/simonw/datasette/issues/1662 | IC_kwDOBm6k_c4_8zz7 | simonw 9599 | 2022-03-19T00:42:23Z | 2022-03-19T00:42:23Z | OWNER | Those client-side SQLite tricks are _really_ neat. `datasette publish` defaults to configuring it so the raw SQLite database can be downloaded from `/fixtures.db` - and this issue updated it to be served with a CORS header that would allow client-side scripts to load the file: - #1057 If you're not going to run any server-side code at all you don't need Datasette for this - you can upload the SQLite database file to any static hosting with CORS headers and load it into the client that way. In terms of static publishing, I do think there's something interesting about using Datasette to generate static sites. There's an issue discussing options for that over here: - #1605 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | [feature request] Publish to fully static website 1170497629 | |
1072904703 | https://github.com/simonw/datasette/issues/1661#issuecomment-1072904703 | https://api.github.com/repos/simonw/datasette/issues/1661 | IC_kwDOBm6k_c4_8zn_ | simonw 9599 | 2022-03-19T00:37:36Z | 2022-03-19T00:37:36Z | OWNER | Updated docs: https://docs.datasette.io/en/latest/performance.html#datasette-hashed-urls | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove Hashed URL mode 1170355774 | |
1072901159 | https://github.com/simonw/datasette/issues/1661#issuecomment-1072901159 | https://api.github.com/repos/simonw/datasette/issues/1661 | IC_kwDOBm6k_c4_8ywn | simonw 9599 | 2022-03-19T00:20:27Z | 2022-03-19T00:20:27Z | OWNER | I can remove the `default_cache_ttl_hashed` setting too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove Hashed URL mode 1170355774 | |
1072898923 | https://github.com/simonw/datasette/pull/1664#issuecomment-1072898923 | https://api.github.com/repos/simonw/datasette/issues/1664 | IC_kwDOBm6k_c4_8yNr | simonw 9599 | 2022-03-19T00:11:33Z | 2022-03-19T00:11:33Z | OWNER | I'm going to land this and handle those in separate commits. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove hashed URL mode 1173017980 | |
1072898797 | https://github.com/simonw/datasette/pull/1664#issuecomment-1072898797 | https://api.github.com/repos/simonw/datasette/issues/1664 | IC_kwDOBm6k_c4_8yLt | simonw 9599 | 2022-03-19T00:11:09Z | 2022-03-19T00:11:09Z | OWNER | Still need to remove it from the documentation and do something about that `hash_urls` setting. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove hashed URL mode 1173017980 | |
1072890524 | https://github.com/simonw/datasette/pull/1664#issuecomment-1072890524 | https://api.github.com/repos/simonw/datasette/issues/1664 | IC_kwDOBm6k_c4_8wKc | simonw 9599 | 2022-03-18T23:44:33Z | 2022-03-19T00:06:51Z | OWNER | Looks like that was set here: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L490-L492 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove hashed URL mode 1173017980 | |
1072890205 | https://github.com/simonw/datasette/pull/1664#issuecomment-1072890205 | https://api.github.com/repos/simonw/datasette/issues/1664 | IC_kwDOBm6k_c4_8wFd | simonw 9599 | 2022-03-18T23:43:15Z | 2022-03-18T23:43:15Z | OWNER | Now almost everything is working except for foreign key expansion: ![CleanShot 2022-03-18 at 16 41 39@2x](https://user-images.githubusercontent.com/9599/159097349-6f41dfdf-5bab-449b-a148-5cda3df6534c.png) Using the debugger I tracked it down to this code: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/table.py#L708-L715 Turns out `default_labels` there is `None` - and it's a parameter to that `data()` method: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/table.py#L325-L334 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove hashed URL mode 1173017980 | |
1072834273 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072834273 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_8ibh | simonw 9599 | 2022-03-18T21:36:05Z | 2022-03-18T21:36:05Z | OWNER | Python's `str.encode()` method has a `errors=` parameter that does something along these lines: https://docs.python.org/3/library/stdtypes.html#str.encode > *errors* may be given to set a different error handling scheme. The default for *errors* is `'strict'`, meaning that encoding errors raise a [`UnicodeError`](https://docs.python.org/3/library/exceptions.html#UnicodeError "UnicodeError"). Other possible values are `'ignore'`, `'replace'`, `'xmlcharrefreplace'`, `'backslashreplace'` and any other name registered via [`codecs.register_error()`](https://docs.python.org/3/library/codecs.html#codecs.register_error "codecs.register_error"), Imitating this might be the way to go. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1072833174 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072833174 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_8iKW | simonw 9599 | 2022-03-18T21:34:06Z | 2022-03-18T21:34:06Z | OWNER | Good call-out: right now the `parsedate()` and `parsedatetime()` functions both terminate with an exception if they hit something invalid: https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes It would be better if this was configurable by the user (and properly documented) - options could include "set null if date is invalid" and "leave the value as it is if invalid" in addition to throwing an error. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
reactions 15 ✖