issue_comments
53 rows where "updated_at" is on date 2020-09-22
This data as json, CSV (advanced)
Suggested facets: issue_url, issue, created_at (date)
| id ▼ | html_url | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 513262013 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513262013 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDUxMzI2MjAxMw== | simonw 9599 | 2019-07-19T14:58:23Z | 2020-09-22T18:12:11Z | OWNER | CLI design idea: $ sqlite-utils extract my.db \ dea_sales company_name Here we just specify the original table and column - the new extracted table will automatically be called "company_name" and will have "id" and "value" columns, by default. To set a custom extract table: $ sqlite-utils extract my.db \ dea_sales company_name \ --table companies And for extracting multiple columns and renaming them on the created table, maybe something like this: $ sqlite-utils extract my.db \ dea_sales company_name company_address \ --table companies \ --column company_name name \ --column company_address address | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 693009048 | https://github.com/simonw/datasette/issues/943#issuecomment-693009048 | https://api.github.com/repos/simonw/datasette/issues/943 | MDEyOklzc3VlQ29tbWVudDY5MzAwOTA0OA== | simonw 9599 | 2020-09-15T22:17:30Z | 2020-09-22T14:37:00Z | OWNER | Maybe instead of implementing `datasette.get()` and `datasette.post()` and `datasette.request()` and `datasette.stream()` I could instead have a nested object called `datasette.client` which is a preconfigured `AsyncClient` instance. ```python response = await datasette.client.get("/") ``` Or perhaps this should be a method in case I ever need to be able to `await` it: ```python response = await (await datasette.client()).get("/") ``` This is a bit cosmetically ugly though, I'd rather avoid that if possible. Maybe I could get this working by returning an object from `.client()` which provides a `await obj.get()` method: ```python response = await datasette.client().get("/") ``` I don't think there's any benefit to that over `await datasette.client.get()` though. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | await datasette.client.get(path) mechanism for executing internal requests 681375466 | |
| 696442621 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696442621 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0MjYyMQ== | simonw 9599 | 2020-09-22T00:00:23Z | 2020-09-22T00:00:23Z | OWNER | I still need to figure out what to do about these various other table properties: https://github.com/simonw/sqlite-utils/blob/b34c9b40c206d7a9d7ee57a8c1f198ff1f522735/sqlite_utils/db.py#L775-L787 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696443042 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443042 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0MzA0Mg== | simonw 9599 | 2020-09-22T00:01:50Z | 2020-09-22T00:01:50Z | OWNER | When you transform a table, it should keep its primary key, foreign keys, not_null and defaults. I don't think it needs to care about `hash_id` or `extracts=` since those don't affect the structure of the table as it is being created - well, `hash_id` does but if we are transforming an existing table we will get the `hash_id` column for free. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696443190 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443190 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0MzE5MA== | simonw 9599 | 2020-09-22T00:02:22Z | 2020-09-22T00:02:22Z | OWNER | How would I detect which columns are `not_null` and what their defaults are? I don`t think my introspection logic handles that yet. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696443845 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443845 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0Mzg0NQ== | simonw 9599 | 2020-09-22T00:04:31Z | 2020-09-22T00:04:44Z | OWNER | Good news: the `.columns` introspection does tell me those things: ``` >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db.create_table("foo", {"id": int, "name": str, "age": int}, defaults={"age": 1}, not_null={"name", "age"}) <Table foo (id, name, age)> >>> db["foo"] <Table foo (id, name, age)> >>> print(db["foo"].schema) CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT NOT NULL, [age] INTEGER NOT NULL DEFAULT 1 ) >>> db["foo"].columns [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value=None, is_pk=0), Column(cid=2, name='age', type='INTEGER', notnull=1, default_value='1', is_pk=0)] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696444353 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444353 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0NDM1Mw== | simonw 9599 | 2020-09-22T00:06:12Z | 2020-09-22T00:06:12Z | OWNER | I should support `not_null=` and `default=` arguments to the `.transform()` method because it looks like you can't use `ALTER TABLE` to change those. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696444842 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444842 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0NDg0Mg== | simonw 9599 | 2020-09-22T00:07:43Z | 2020-09-22T00:09:05Z | OWNER | Syntax challenge: I could use `.transform(defaults={"age": None})` to indicate that the `age` column should have its default removed, but how would I tell `.transform()` that the `age` column, currently `not null`, should have the `not null` removed from it? I could do this: `.transform(not_not_null={"age"})` - it's a bit gross but it's also kind of funny. I actually like it! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696445766 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696445766 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0NTc2Ng== | simonw 9599 | 2020-09-22T00:10:50Z | 2020-09-22T00:11:12Z | OWNER | A less horrible interface might be the following: ```python # Ensure the 'age' column is not null: table.transform(not_null={"age"}) # The 'age' column is not null but I don't want it to be: table.transform(not_null={"age": False}) ``` So if the argument is a set it means "make sure these are all not null" - if the argument is a dictionary it means "set these to be null or not null depending on if their dictionary value is true or false". | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696446658 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696446658 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0NjY1OA== | simonw 9599 | 2020-09-22T00:13:55Z | 2020-09-22T00:14:21Z | OWNER | Idea: allow a `conversions=` parameter, as seen on `.insert_all()` and friends, which lets you apply a SQL transformation function as part of the operation. E.g.: ```python table.transform({"age": int}, conversions={"name": "upper(?)"}) ``` https://sqlite-utils.readthedocs.io/en/stable/python-api.html#converting-column-values-using-sql-functions | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696449345 | https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696449345 | https://api.github.com/repos/simonw/sqlite-utils/issues/162 | MDEyOklzc3VlQ29tbWVudDY5NjQ0OTM0NQ== | simonw 9599 | 2020-09-22T00:22:46Z | 2020-09-22T00:22:46Z | OWNER | Inspired by the idea of adding `conversions=` to #114 - since this would make it easy to register custom Python functions that can be used to convert the values in a table. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | A decorator for registering custom SQL functions 705995722 | |
| 696454084 | https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696454084 | https://api.github.com/repos/simonw/sqlite-utils/issues/162 | MDEyOklzc3VlQ29tbWVudDY5NjQ1NDA4NA== | simonw 9599 | 2020-09-22T00:40:44Z | 2020-09-22T00:40:44Z | OWNER | Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#registering-custom-sql-functions | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | A decorator for registering custom SQL functions 705995722 | |
| 696454485 | https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696454485 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | MDEyOklzc3VlQ29tbWVudDY5NjQ1NDQ4NQ== | simonw 9599 | 2020-09-22T00:42:35Z | 2020-09-22T00:42:35Z | OWNER | The reason I'm working on this now is that I'd like to support many more options for data cleanup in the Datasette ecosystem - so being able to do things like convert the type of existing columns becomes increasingly important. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method for advanced alter table 621989740 | |
| 696465788 | https://github.com/simonw/sqlite-utils/issues/163#issuecomment-696465788 | https://api.github.com/repos/simonw/sqlite-utils/issues/163 | MDEyOklzc3VlQ29tbWVudDY5NjQ2NTc4OA== | simonw 9599 | 2020-09-22T01:33:04Z | 2020-09-22T01:33:04Z | OWNER | This would apply to `.transform()` in #114 too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Idea: conversions= could take Python functions 706001517 | |
| 696473559 | https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696473559 | https://api.github.com/repos/simonw/sqlite-utils/issues/164 | MDEyOklzc3VlQ29tbWVudDY5NjQ3MzU1OQ== | simonw 9599 | 2020-09-22T02:10:37Z | 2020-09-22T02:10:37Z | OWNER | Maybe something like this: sqlite-utils transform mydb.db mytable -c age integer --rename age dog_age | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils transform sub-command 706017416 | |
| 696480925 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696480925 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ4MDkyNQ== | simonw 9599 | 2020-09-22T02:45:47Z | 2020-09-22T02:45:47Z | OWNER | I'm not going to do `conversions=` because it would be inconsistent with how they work elsewhere. The SQL generated by this function looks like this: INSERT INTO dogs_new_tmp VALUES (a, b) SELECT a, b from dogs; So passing `conversions={"name": "upper(?)"})` wouldn't make sense, since we're not using arguments hence there is no-where for that `?` to go. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696485791 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696485791 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ4NTc5MQ== | simonw 9599 | 2020-09-22T03:10:15Z | 2020-09-22T03:10:15Z | OWNER | Design decision needed on foreign keys: what does the syntax look like for removing an existing foreign key? Since I already have a good implementation of `add_foreign_key()` I'm tempted to only support dropping them. Maybe like this: ```python table.transform(drop_foreign_keys=[("author_id", "author", "id")]) ``` It's a bit crufty but it's such a rare use-case that I think this will be good enough. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696488201 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696488201 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ4ODIwMQ== | simonw 9599 | 2020-09-22T03:21:16Z | 2020-09-22T03:21:16Z | OWNER | Just needs documentation now. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696490851 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696490851 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ5MDg1MQ== | simonw 9599 | 2020-09-22T03:33:54Z | 2020-09-22T03:33:54Z | OWNER | It would be neat if `.transform(pk=None)` converted a primary key table to a rowid table. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696494070 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696494070 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ5NDA3MA== | simonw 9599 | 2020-09-22T03:48:58Z | 2020-09-22T03:48:58Z | OWNER | One last thing. https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_change says that the first step should be: > If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. And the last steps should be: > If foreign key constraints were originally enabled then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints. > > Commit the transaction started in step 2. > > If foreign keys constraints were originally enabled, reenable them now. I need to implement that. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
| 696500767 | https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696500767 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | MDEyOklzc3VlQ29tbWVudDY5NjUwMDc2Nw== | simonw 9599 | 2020-09-22T04:21:45Z | 2020-09-22T04:21:45Z | OWNER | Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method for advanced alter table 621989740 | |
| 696500922 | https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696500922 | https://api.github.com/repos/simonw/sqlite-utils/issues/164 | MDEyOklzc3VlQ29tbWVudDY5NjUwMDkyMg== | simonw 9599 | 2020-09-22T04:22:40Z | 2020-09-22T04:22:40Z | OWNER | Documentation for the `.transform()` method #114 (now landed) is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils transform sub-command 706017416 | |
| 696520928 | https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696520928 | https://api.github.com/repos/simonw/sqlite-utils/issues/164 | MDEyOklzc3VlQ29tbWVudDY5NjUyMDkyOA== | simonw 9599 | 2020-09-22T05:50:17Z | 2020-09-22T05:50:17Z | OWNER | Idea for CLI options: ``` --type age integer --drop colname --rename oldname newname --not-null col --not-null-false col --pk new_id --pk-none --default col value --default-none column --drop-foreign-key col other_table other_column ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils transform sub-command 706017416 | |
| 696565981 | https://github.com/simonw/sqlite-utils/issues/167#issuecomment-696565981 | https://api.github.com/repos/simonw/sqlite-utils/issues/167 | MDEyOklzc3VlQ29tbWVudDY5NjU2NTk4MQ== | simonw 9599 | 2020-09-22T07:53:13Z | 2020-09-22T07:53:13Z | OWNER | Confirmed this is a bug, https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes explicitly says you should do the `PRAGMA foreign_keys` bits before and after the transaction, not during. Right now my code does this INSIDE the transaction: https://github.com/simonw/sqlite-utils/blob/f29f6821f2d08e91c5c6d65d885a1bbc0c743bdd/sqlite_utils/db.py#L790-L793 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Review the foreign key pragma stuff 706098005 | |
| 696566750 | https://github.com/simonw/sqlite-utils/issues/26#issuecomment-696566750 | https://api.github.com/repos/simonw/sqlite-utils/issues/26 | MDEyOklzc3VlQ29tbWVudDY5NjU2Njc1MA== | simonw 9599 | 2020-09-22T07:55:00Z | 2020-09-22T07:55:00Z | OWNER | Problem: `extract` means something else now, see #47 and the upcoming work in #42. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for turning nested JSON into foreign keys / many-to-many 455486286 | |
| 696567460 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696567460 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5NjU2NzQ2MA== | simonw 9599 | 2020-09-22T07:56:42Z | 2020-09-22T07:56:42Z | OWNER | `.transform()` has landed now which should make this a lot easier to solve. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 696567988 | https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696567988 | https://api.github.com/repos/simonw/sqlite-utils/issues/164 | MDEyOklzc3VlQ29tbWVudDY5NjU2Nzk4OA== | simonw 9599 | 2020-09-22T07:57:50Z | 2020-09-22T07:57:50Z | OWNER | Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#transforming-tables | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils transform sub-command 706017416 | |
| 696573944 | https://github.com/simonw/sqlite-utils/issues/168#issuecomment-696573944 | https://api.github.com/repos/simonw/sqlite-utils/issues/168 | MDEyOklzc3VlQ29tbWVudDY5NjU3Mzk0NA== | simonw 9599 | 2020-09-22T08:11:30Z | 2020-09-22T08:11:30Z | OWNER | Huh... maybe I don't need to do anything here? It looks like it's been kept up to date: https://github.com/Homebrew/homebrew-core/commits/master/Formula/sqlite-utils.rb | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Automate (as much as possible) updates published to Homebrew 706167456 | |
| 696769501 | https://github.com/simonw/datasette/issues/943#issuecomment-696769501 | https://api.github.com/repos/simonw/datasette/issues/943 | MDEyOklzc3VlQ29tbWVudDY5Njc2OTUwMQ== | simonw 9599 | 2020-09-22T14:45:49Z | 2020-09-22T14:45:49Z | OWNER | I put together a minimal prototype of this and it feels pretty good: ```diff diff --git a/datasette/app.py b/datasette/app.py index 20aae7d..fb3bdad 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -4,6 +4,7 @@ import collections import datetime import glob import hashlib +import httpx import inspect import itertools from itsdangerous import BadSignature @@ -312,6 +313,7 @@ class Datasette: self._register_renderers() self._permission_checks = collections.deque(maxlen=200) self._root_token = secrets.token_hex(32) + self.client = DatasetteClient(self) async def invoke_startup(self): for hook in pm.hook.startup(datasette=self): @@ -1209,3 +1211,25 @@ def route_pattern_from_filepath(filepath): class NotFoundExplicit(NotFound): pass + + +class DatasetteClient: + def __init__(self, ds): + self.app = ds.app() + + def _fix(self, path): + if path.startswith("/"): + path = "http://localhost{}".format(path) + return path + + async def get(self, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + return await client.get(self._fix(path), **kwargs) + + async def post(self, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + return await client.post(self._fix(path), **kwargs) + + async def options(self, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + return await client.options(self._fix(path), **kwargs) ``` Used like this in `ipython`: ``` In [1]: from datasette.app import Datasette In [2]: ds = Datasette(["fixtures.db"]) In [3]: (await ds.client.get("/-/config.json")).json() Out[3]: {'default_page_size': 100, 'max_returned_rows': 1000, 'num_sql_threads': 3, 'sql_time_limit_ms': 1000, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'hash_urls': False, 'allow_facet': True,… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | await datasette.client.get(path) mechanism for executing internal requests 681375466 | |
| 696769853 | https://github.com/simonw/datasette/issues/943#issuecomment-696769853 | https://api.github.com/repos/simonw/datasette/issues/943 | MDEyOklzc3VlQ29tbWVudDY5Njc2OTg1Mw== | simonw 9599 | 2020-09-22T14:46:21Z | 2020-09-22T14:46:21Z | OWNER | This adds `httpx` as a dependency - I think I'm OK with that. I use it for testing in all of my plugins anyway. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | await datasette.client.get(path) mechanism for executing internal requests 681375466 | |
| 696774711 | https://github.com/simonw/datasette/issues/943#issuecomment-696774711 | https://api.github.com/repos/simonw/datasette/issues/943 | MDEyOklzc3VlQ29tbWVudDY5Njc3NDcxMQ== | simonw 9599 | 2020-09-22T14:53:56Z | 2020-09-22T14:53:56Z | OWNER | How important is it to use `httpx.AsyncClient` with a context manager? https://www.python-httpx.org/async/#opening-and-closing-clients says: > Alternatively, use `await client.aclose()` if you want to close a client explicitly: > > ``` > client = httpx.AsyncClient() > ... > await client.aclose() > ``` The `.aclose()` method has a comment saying "Close transport and proxies" - I'm not using proxies, so the relevant implementation seems to be a call to `await self._transport.aclose()` in https://github.com/encode/httpx/blob/f932af9172d15a803ad40061a4c2c0cd891645cf/httpx/_client.py#L1741-L1751 The transport I am using is a class called `ASGITransport` in https://github.com/encode/httpx/blob/master/httpx/_transports/asgi.py The `aclose()` method on that class does nothing. So it looks like I can instantiate a client without bothering with the `async with httpx.AsyncClient` bit. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | await datasette.client.get(path) mechanism for executing internal requests 681375466 | |
| 696775516 | https://github.com/simonw/datasette/issues/943#issuecomment-696775516 | https://api.github.com/repos/simonw/datasette/issues/943 | MDEyOklzc3VlQ29tbWVudDY5Njc3NTUxNg== | simonw 9599 | 2020-09-22T14:55:10Z | 2020-09-22T14:55:10Z | OWNER | Even smaller `DatasetteClient` implementation: ```python class DatasetteClient: def __init__(self, ds): self._client = httpx.AsyncClient(app=ds.app()) def _fix(self, path): if path.startswith("/"): path = "http://localhost{}".format(path) return path async def get(self, path, **kwargs): return await self._client.get(self._fix(path), **kwargs) async def post(self, path, **kwargs): return await self._client.post(self._fix(path), **kwargs) async def options(self, path, **kwargs): return await self._client.options(self._fix(path), **kwargs) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | await datasette.client.get(path) mechanism for executing internal requests 681375466 | |
| 696776828 | https://github.com/simonw/datasette/issues/943#issuecomment-696776828 | https://api.github.com/repos/simonw/datasette/issues/943 | MDEyOklzc3VlQ29tbWVudDY5Njc3NjgyOA== | simonw 9599 | 2020-09-22T14:57:13Z | 2020-09-22T14:57:13Z | OWNER | I may as well implement all of the HTTP methods supported by the `httpx` client: - get - options - head - post - put - patch - delete | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | await datasette.client.get(path) mechanism for executing internal requests 681375466 | |
| 696777886 | https://github.com/simonw/datasette/issues/943#issuecomment-696777886 | https://api.github.com/repos/simonw/datasette/issues/943 | MDEyOklzc3VlQ29tbWVudDY5Njc3Nzg4Ng== | simonw 9599 | 2020-09-22T14:58:54Z | 2020-09-22T14:58:54Z | OWNER | ```python class DatasetteClient: def __init__(self, ds): self._client = httpx.AsyncClient(app=ds.app()) def _fix(self, path): if path.startswith("/"): path = "http://localhost{}".format(path) return path async def get(self, path, **kwargs): return await self._client.get(self._fix(path), **kwargs) async def options(self, path, **kwargs): return await self._client.options(self._fix(path), **kwargs) async def head(self, path, **kwargs): return await self._client.head(self._fix(path), **kwargs) async def post(self, path, **kwargs): return await self._client.post(self._fix(path), **kwargs) async def put(self, path, **kwargs): return await self._client.put(self._fix(path), **kwargs) async def patch(self, path, **kwargs): return await self._client.patch(self._fix(path), **kwargs) async def delete(self, path, **kwargs): return await self._client.delete(self._fix(path), **kwargs) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | await datasette.client.get(path) mechanism for executing internal requests 681375466 | |
| 696778735 | https://github.com/simonw/datasette/issues/943#issuecomment-696778735 | https://api.github.com/repos/simonw/datasette/issues/943 | MDEyOklzc3VlQ29tbWVudDY5Njc3ODczNQ== | simonw 9599 | 2020-09-22T15:00:13Z | 2020-09-22T15:00:39Z | OWNER | Am I going to rewrite ALL of my tests to use this instead? It would clean up a lot of test code, at the cost of quite a bit of work. It would make for much neater plugin tests too, and neater testing documentation: https://docs.datasette.io/en/stable/testing_plugins.html | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | await datasette.client.get(path) mechanism for executing internal requests 681375466 | |
| 696788109 | https://github.com/simonw/datasette/issues/969#issuecomment-696788109 | https://api.github.com/repos/simonw/datasette/issues/969 | MDEyOklzc3VlQ29tbWVudDY5Njc4ODEwOQ== | simonw 9599 | 2020-09-22T15:15:14Z | 2020-09-22T15:15:14Z | OWNER | I don't think a standard "pass these extra arguments to the publish tool" mechanism will work because there's no guarantee that a publisher uses a CLI tool - or if it does, it might make several calls to different CLI tools. The Cloud Run one runs a couple of commands, as illustrated by this test: https://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/tests/test_publish_cloudrun.py#L63-L73 Adding a `--tar` option for `datasette publish heroku` is a good fix for this though. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add --tar option to "datasette publish heroku" 705057955 | |
| 696798114 | https://github.com/simonw/datasette/issues/973#issuecomment-696798114 | https://api.github.com/repos/simonw/datasette/issues/973 | MDEyOklzc3VlQ29tbWVudDY5Njc5ODExNA== | simonw 9599 | 2020-09-22T15:31:25Z | 2020-09-22T15:31:25Z | OWNER | D'oh because I have a new variable called `open`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 'bool' object is not callable error 706486323 | |
| 696800410 | https://github.com/simonw/datasette/issues/973#issuecomment-696800410 | https://api.github.com/repos/simonw/datasette/issues/973 | MDEyOklzc3VlQ29tbWVudDY5NjgwMDQxMA== | simonw 9599 | 2020-09-22T15:35:28Z | 2020-09-22T15:35:28Z | OWNER | Confirmed in local dev: ``` % datasette fixtures.db --inspect-file inspect.json Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette", line 11, in <module> load_entry_point('datasette', 'console_scripts', 'datasette')() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 406, in serve inspect_data = json.load(open(inspect_file)) TypeError: 'bool' object is not callable ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 'bool' object is not callable error 706486323 | |
| 696893244 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893244 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5Njg5MzI0NA== | simonw 9599 | 2020-09-22T18:14:33Z | 2020-09-22T18:14:45Z | OWNER | Thinking more about this one: ``` $ sqlite-utils extract my.db \ dea_sales company_name company_address \ --table companies ``` The goal here is to pull the company name and address pair out into a separate table. Some questions: - should this first verify that every company_name has just one company_address? I like the idea of a unique constraint on the created table for this. - what should the foreign key column that gets added to the `companies` table be called? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 696893774 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893774 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5Njg5Mzc3NA== | simonw 9599 | 2020-09-22T18:15:33Z | 2020-09-22T18:15:33Z | OWNER | I think the new foreign key column is called `company_name_id` by default in this example but can be customized by passing `--fk-column=xxx` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 696976678 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696976678 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5Njk3NjY3OA== | simonw 9599 | 2020-09-22T20:57:57Z | 2020-09-22T20:57:57Z | OWNER | I think I understand the shape of this feature now. It lets you specify one or more columns on the source table which will be extracted into another table. It uses the `.lookup()` mechanism to populate that other table, which means each unique column value / pair / triple will be assigned an integer ID. That integer ID gets written back into the first of the columns that are being transformed. A `.transform()` call then converts that column to an integer (and drops the additional columns). Finally we set up the new foreign key relationship. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 696979168 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979168 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5Njk3OTE2OA== | simonw 9599 | 2020-09-22T21:02:24Z | 2020-09-22T21:02:24Z | OWNER | In Python it looks like this: ```python # Simple case - species column species_id pointing to species table db["trees"].extract("species") # Setting a custom table db["trees"].extract("species", table="Species") # Custom foreign key column on trees db["trees"].extract("species", fk_column="species") # Extracting multiple columns db["trees"].extract(["common_name", "latin_name"]) # (this creates a lookup table called common_name_latin_name ref'd by common_name_latin_name_id) # Or with explicit table (fk_column here defaults to species_id because of the table name) db["trees"].extract(["common_name", "latin_name"], table="species") ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 696979626 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979626 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5Njk3OTYyNg== | simonw 9599 | 2020-09-22T21:03:11Z | 2020-09-22T21:03:11Z | OWNER | And if you want to rename some of the columns in the new table: ```python db["trees"].extract(["common_name", "latin_name"], table="species", rename={"common_name": "name"}) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 696980503 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980503 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5Njk4MDUwMw== | simonw 9599 | 2020-09-22T21:04:45Z | 2020-09-22T21:04:45Z | OWNER | `table.extract()` can take an optional `progress=` argument which is a callback which will be used to report progress - called after each batch with `(num_done, total)`. It will get called with `(0, total)` once at the start to allow progress bars to be initialized. The command-line progress bar will use this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 696980709 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980709 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5Njk4MDcwOQ== | simonw 9599 | 2020-09-22T21:05:07Z | 2020-09-22T21:05:07Z | OWNER | So `.extract()` probably takes a `batch_size=` argument too, which defaults to maybe 1000. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 696987257 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987257 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5Njk4NzI1Nw== | simonw 9599 | 2020-09-22T21:17:34Z | 2020-09-22T21:17:34Z | OWNER | What to do if the table already exists? The `.lookup()` function already knows how to modify an existing table to create the correct constraints etc, so I'll rely on that mechanism. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 696987925 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987925 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5Njk4NzkyNQ== | simonw 9599 | 2020-09-22T21:19:04Z | 2020-09-22T21:19:04Z | OWNER | Need to make sure this works correctly for `rowid` tables. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 697012111 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697012111 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5NzAxMjExMQ== | simonw 9599 | 2020-09-22T22:18:13Z | 2020-09-22T22:18:13Z | OWNER | Here's how I'm generating the examples for the documentation: ``` In [2]: import sqlite_utils In [3]: db = sqlite_utils.Database(memory=True) In [4]: db["Trees"].insert({"id": 1, "TreeAddress": "52 Vine St", "CommonName": ...: "Palm", "LatinName": "foo"}, pk="id") Out[4]: <Table Trees (id, TreeAddress, CommonName, LatinName)> In [5]: db["Trees"].extract(["CommonName", "LatinName"], table="Species", fk_col ...: umn="species_id") In [6]: print(db["Trees"].schema) CREATE TABLE "Trees" ( [id] INTEGER PRIMARY KEY, [TreeAddress] TEXT, [species_id] INTEGER, FOREIGN KEY(species_id) REFERENCES Species(id) ) In [7]: print(db["Species"].schema) CREATE TABLE [Species] ( [id] INTEGER PRIMARY KEY, [CommonName] TEXT, [LatinName] TEXT ) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 697013681 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697013681 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5NzAxMzY4MQ== | simonw 9599 | 2020-09-22T22:22:49Z | 2020-09-22T22:22:49Z | OWNER | The command-line version of this needs to accept a table and one or more columns, then a `--table` and `--fk-column` option. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 697019944 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697019944 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5NzAxOTk0NA== | simonw 9599 | 2020-09-22T22:40:00Z | 2020-09-22T22:40:00Z | OWNER | I tried out the prototype of the CLI on the Global Power Plants data: ``` wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv sqlite-utils extract global.db power_plants country country_long ``` This threw an error because `rowid` columns are not yet supported. I fixed that like so: ``` sqlite-utils transform global.db power_plants --rename rowid id sqlite-utils extract global.db power_plants country country_long ``` That worked! But it didn't play great with Datasette, because the resulting extracted table had columns `country` and `country_long` and neither of those are called `name` or `value` or `title`. Based on this I need to add `rowid` table support AND I need to implement the proposed `rename=` argument for renaming columns on their way into the new table. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 697025403 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697025403 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5NzAyNTQwMw== | simonw 9599 | 2020-09-22T22:57:53Z | 2020-09-22T22:57:53Z | OWNER | The documentation for the `.extract()` method is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#extracting-columns-into-a-separate-table | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 697031174 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697031174 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5NzAzMTE3NA== | simonw 9599 | 2020-09-22T23:16:00Z | 2020-09-22T23:16:00Z | OWNER | Trying this demo again: ``` wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv sqlite-utils extract global.db power_plants country country_long --table countries --rename country_long name ``` It worked! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 | |
| 697037974 | https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697037974 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | MDEyOklzc3VlQ29tbWVudDY5NzAzNzk3NA== | simonw 9599 | 2020-09-22T23:39:31Z | 2020-09-22T23:39:31Z | OWNER | Documentation for `sqlite-utils extract`: https://sqlite-utils.readthedocs.io/en/latest/cli.html#extracting-columns-into-a-separate-table | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.extract(...) method and "sqlite-utils extract" command 470345929 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] (
[html_url] TEXT,
[issue_url] TEXT,
[id] INTEGER PRIMARY KEY,
[node_id] TEXT,
[user] INTEGER REFERENCES [users]([id]),
[created_at] TEXT,
[updated_at] TEXT,
[author_association] TEXT,
[body] TEXT,
[reactions] TEXT,
[issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
ON [issue_comments] ([user]);