issue_comments
7,931 rows where author_association = "OWNER" sorted by html_url descending
This data as json, CSV (advanced)
id | html_url ▲ | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
982126665 | https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982126665 | https://api.github.com/repos/simonw/sqlite-utils/issues/347 | IC_kwDOCGYnMM46ihBJ | simonw 9599 | 2021-11-29T23:26:01Z | 2021-11-29T23:33:48Z | OWNER | https://github.com/simonw/sqlite-utils/blob/93b059dd230eae9eaae472b7fbabd4a66feeb79d/.github/workflows/test.yml#L11-L20 This configuration means that the numpy=0, Python=3.10, os=Ubuntu build will additionally use `pysqlite3` with the SQLite 3.37.0. It's failing right now: https://github.com/simonw/sqlite-utils/runs/4360593156 - because `pysqlite3` doesn't provide `.iterdump()`. I can use the workaround from this comment: https://github.com/coleifer/pysqlite3/issues/24#issuecomment-982081267 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Test against pysqlite3 running SQLite 3.37 1066603133 | |
979442854 | https://github.com/simonw/sqlite-utils/pull/333#issuecomment-979442854 | https://api.github.com/repos/simonw/sqlite-utils/issues/333 | IC_kwDOCGYnMM46YRym | simonw 9599 | 2021-11-25T19:47:26Z | 2021-11-25T19:47:26Z | OWNER | I just remembered that there's one other place that this could fit: as a Datasette "insert" plugin. This is vaporware at the moment, but the idea is that Datasette itself could grow a mechanism for importing data, that's driven by plugins. Out of the box Datasette would be able to import CSV and CSV files, similar to `sqlite-utils insert ... --csv` - but plugins would then be able to add support for additional format such as GeoJSON or - in this case - Parquet. The neat thing about having it as a Datasette plugin is that one plugin would enable three different ways of importing data: 1. Via a new `datasette insert ...` CLI option (similar to `sqlite-utils`) 2. Via a web form upload interface, where authenticated Datasette users would be able to upload files 3. Via an API interface, where files could be programatically submitted to a running Datasette server I started fleshing out this idea quite a while ago but didn't make much concrete progress, maybe I should revisit it: - https://github.com/simonw/datasette/issues/1160 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add functionality to read Parquet files. 1039037439 | |
974754412 | https://github.com/simonw/sqlite-utils/pull/333#issuecomment-974754412 | https://api.github.com/repos/simonw/sqlite-utils/issues/333 | IC_kwDOCGYnMM46GZJs | simonw 9599 | 2021-11-21T04:35:32Z | 2021-11-21T04:35:32Z | OWNER | Some other recent projects (like trying to get this library to work in JupyterLite) have made me much more cautious about adding new dependencies, especially dependencies like `pyarrow` which require custom C/Rust extensions. There are a few ways this could work though: - Have this as an optional dependency feature - so it only works if the user installs `pyarrow` as well - Implement this as a separate tool, `parquet-to-sqlite` - which could itself depend on `sqlite-utils` - Add a concept of "plugins" to `sqlite-utils`, similar to how those work in Datasette: https://docs.datasette.io/en/stable/plugins.html My favourite option is `parquet-to-sqlite` because that can be built without any additional changes to `sqlite-utils` at all! I find the concept of plugins for `sqlite-utils` interesting. I've so far not had quite enough potential use-cases to convince me this is worthwhile (especially since it should be very easy to build out separate tools entirely), but I'm ready to be convinced that a plugin mechanism would be worthwhile. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add functionality to read Parquet files. 1039037439 | |
968384988 | https://github.com/simonw/sqlite-utils/pull/324#issuecomment-968384988 | https://api.github.com/repos/simonw/sqlite-utils/issues/324 | IC_kwDOCGYnMM45uGHc | simonw 9599 | 2021-11-14T23:25:16Z | 2021-11-14T23:25:16Z | OWNER | Yes this was absolutely the intention! Thanks, I wonder how often I've made that mistake in other projects? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Use python-dateutil package instead of dateutils 988013247 | |
968401459 | https://github.com/simonw/sqlite-utils/pull/322#issuecomment-968401459 | https://api.github.com/repos/simonw/sqlite-utils/issues/322 | IC_kwDOCGYnMM45uKIz | simonw 9599 | 2021-11-15T00:26:42Z | 2021-11-15T00:26:42Z | OWNER | This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947 So the `COLUMN_TYPE_MAPPING` should include those too - right now it looks like this: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L165-L188 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add dict type to be mapped as TEXT in sqllite 979612115 | |
905040902 | https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040902 | https://api.github.com/repos/simonw/sqlite-utils/issues/321 | IC_kwDOCGYnMM418dQG | simonw 9599 | 2021-08-24T23:25:03Z | 2021-08-24T23:25:03Z | OWNER | I'm going to skip this test on windows. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to insert file contents as text, in addition to blob 978537855 | |
905040307 | https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040307 | https://api.github.com/repos/simonw/sqlite-utils/issues/321 | IC_kwDOCGYnMM418dGz | simonw 9599 | 2021-08-24T23:23:36Z | 2021-08-24T23:23:36Z | OWNER | https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819 says: > Currently, `TextIOWrapper` uses `locale.getpreferredencoding(False)` (hereinafter called “locale encoding”) when encoding is not specified. > ... > Package authors using macOS or Linux may forget that the default encoding is not always UTF-8. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to insert file contents as text, in addition to blob 978537855 | |
905039576 | https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905039576 | https://api.github.com/repos/simonw/sqlite-utils/issues/321 | IC_kwDOCGYnMM418c7Y | simonw 9599 | 2021-08-24T23:21:29Z | 2021-08-24T23:21:29Z | OWNER | Hah, the error here is actually: ``` > assert result.exit_code == 1, result.output E AssertionError: E E assert 0 == 1 E + where 0 = <Result okay>.exit_code ``` So I was expecting an error, but instead the command worked. I suspect this is because on Windows the default character set may not be UTF-8? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to insert file contents as text, in addition to blob 978537855 | |
905037323 | https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905037323 | https://api.github.com/repos/simonw/sqlite-utils/issues/321 | IC_kwDOCGYnMM418cYL | simonw 9599 | 2021-08-24T23:15:29Z | 2021-08-24T23:15:29Z | OWNER | Huh, tests are failing but only on Windows! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to insert file contents as text, in addition to blob 978537855 | |
506784912 | https://github.com/simonw/sqlite-utils/pull/32#issuecomment-506784912 | https://api.github.com/repos/simonw/sqlite-utils/issues/32 | MDEyOklzc3VlQ29tbWVudDUwNjc4NDkxMg== | simonw 9599 | 2019-06-28T15:55:13Z | 2019-06-28T15:55:13Z | OWNER | Documentation, to be added as a subsection of https://sqlite-utils.readthedocs.io/en/latest/python-api.html#adding-foreign-key-constraints ### Adding multiple foreign key constraints at once The final step in adding a new foreign key to a SQLite database is to run VACUUM, to ensure the new foreign key is available in future introspection queries. VACUUM against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using table.add_foreign_key(...) these can quickly add up. Instead, you can use db.add_foreign_keys(...) to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a table, column, other_table and other_column. Here's an example adding two foreign keys at once: db.add_foreign_keys([ ("dogs", "breed_id", "breeds", "id"), ("dogs", "home_town_id", "towns", "id") ]) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | db.add_foreign_keys() method 462094937 | |
896284722 | https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896284722 | https://api.github.com/repos/simonw/sqlite-utils/issues/312 | IC_kwDOCGYnMM41bDgy | simonw 9599 | 2021-08-10T20:08:03Z | 2021-08-10T20:08:21Z | OWNER | Spotted a rogue backtick: ![A0147E27-7506-49B0-BEFB-20D99BBFEBAD](https://user-images.githubusercontent.com/9599/128927930-b3333dee-a385-409b-a945-f108e6ea40df.jpeg) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add reference page to documentation using Sphinx autodoc 965143346 | |
896200682 | https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896200682 | https://api.github.com/repos/simonw/sqlite-utils/issues/312 | IC_kwDOCGYnMM41au_q | simonw 9599 | 2021-08-10T18:03:40Z | 2021-08-10T18:03:40Z | OWNER | Adding type signatures to `create_table()` and `.create_table_sql()` is a bit too involved, I'll do that in a separate issue. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add reference page to documentation using Sphinx autodoc 965143346 | |
896186025 | https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896186025 | https://api.github.com/repos/simonw/sqlite-utils/issues/312 | IC_kwDOCGYnMM41arap | simonw 9599 | 2021-08-10T17:42:51Z | 2021-08-10T17:42:51Z | OWNER | That worked! https://sqlite-utils.datasette.io/en/autodoc/reference.html | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add reference page to documentation using Sphinx autodoc 965143346 | |
896182934 | https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896182934 | https://api.github.com/repos/simonw/sqlite-utils/issues/312 | IC_kwDOCGYnMM41aqqW | simonw 9599 | 2021-08-10T17:38:44Z | 2021-08-10T17:38:44Z | OWNER | From https://docs.readthedocs.io/en/stable/config-file/v2.html#packages it looks like I can tell Read The Docs to run `pip install -e .` using a `.readthedocs.yaml` configuration: ```yaml version: 2 sphinx: configuration: docs/conf.py python: version: "3.9" install: - method: pip path: . extra_requirements: - docs ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add reference page to documentation using Sphinx autodoc 965143346 | |
896180956 | https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896180956 | https://api.github.com/repos/simonw/sqlite-utils/issues/312 | IC_kwDOCGYnMM41aqLc | simonw 9599 | 2021-08-10T17:35:51Z | 2021-08-10T17:35:51Z | OWNER | Reading the rest of https://sphinx-rtd-tutorial.readthedocs.io/en/latest/sphinx-config.html#autodoc-configuration it suggests using a `requirements.txt` file to install dependencies - but I use `setup.py` for that so I need to figure out a different pattern here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add reference page to documentation using Sphinx autodoc 965143346 | |
896175438 | https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896175438 | https://api.github.com/repos/simonw/sqlite-utils/issues/312 | IC_kwDOCGYnMM41ao1O | simonw 9599 | 2021-08-10T17:28:19Z | 2021-08-10T17:28:19Z | OWNER | https://sphinx-rtd-tutorial.readthedocs.io/en/latest/sphinx-config.html#autodoc-configuration says do something like this at the top of `conf.py`: ```python import os import sys sys.path.insert(0, os.path.abspath('../../simpleble/')) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add reference page to documentation using Sphinx autodoc 965143346 | |
896174456 | https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896174456 | https://api.github.com/repos/simonw/sqlite-utils/issues/312 | IC_kwDOCGYnMM41aol4 | simonw 9599 | 2021-08-10T17:27:01Z | 2021-08-10T17:27:01Z | OWNER | Docs are now building at https://sqlite-utils.datasette.io/en/autodoc/reference.html But there's a problem! The page is semi-blank: <img width="663" alt="Reference_—_sqlite-utils_3_15-6-gc11ff89_documentation" src="https://user-images.githubusercontent.com/9599/128906373-7463e0c9-3b08-4b15-a73a-84b8ba6ab342.png"> I need to teach Read The Docs how to ensure `sqlite_utils` is available for introspection. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add reference page to documentation using Sphinx autodoc 965143346 | |
896156971 | https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896156971 | https://api.github.com/repos/simonw/sqlite-utils/issues/312 | IC_kwDOCGYnMM41akUr | simonw 9599 | 2021-08-10T17:04:22Z | 2021-08-10T17:05:59Z | OWNER | I'm going to get Read The Docs to build the docs for this branch too - on https://readthedocs.org/projects/sqlite-utils/versions/ I am clicking this button: <img width="881" alt="Versions___Read_the_Docs" src="https://user-images.githubusercontent.com/9599/128903015-0f2649e7-34a8-4dfd-984c-b1b5bd8a98be.png"> I then set it to "active" (so pushes to the branch will build it) and "hidden" (so it wouldn't show up in search or in the navigation menu). https://docs.readthedocs.io/en/stable/versions.html#version-states <img width="846" alt="autodoc_-_sqlite-utils___Read_the_Docs" src="https://user-images.githubusercontent.com/9599/128903217-efe827c6-4f54-42dd-943e-ddf9ab3a05d8.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add reference page to documentation using Sphinx autodoc 965143346 | |
896154028 | https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896154028 | https://api.github.com/repos/simonw/sqlite-utils/issues/312 | IC_kwDOCGYnMM41ajms | simonw 9599 | 2021-08-10T17:01:06Z | 2021-08-10T17:01:06Z | OWNER | On Python 3.6: ``` sqlite_utils/db.py:366: in Database def tables(self) -> List[Table]: E NameError: name 'Table' is not defined ``` Python 3.7 can fix this with `from __future__ import annotations` but since we still support 3.6 I'll have to use a string instead. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add reference page to documentation using Sphinx autodoc 965143346 | |
868728092 | https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868728092 | https://api.github.com/repos/simonw/sqlite-utils/issues/293 | MDEyOklzc3VlQ29tbWVudDg2ODcyODA5Mg== | simonw 9599 | 2021-06-25T17:39:35Z | 2021-06-25T17:39:35Z | OWNER | Here's more about this problem: https://github.com/numpy/numpy/issues/15947 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Test against Python 3.10-dev 929748885 | |
868134040 | https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868134040 | https://api.github.com/repos/simonw/sqlite-utils/issues/293 | MDEyOklzc3VlQ29tbWVudDg2ODEzNDA0MA== | simonw 9599 | 2021-06-25T01:49:44Z | 2021-06-25T01:50:33Z | OWNER | Test failed on 3.10 with `numpy` on macOS: ``` sqlite_utils/__init__.py:1: in <module> 11 from .db import Database 12 sqlite_utils/db.py:48: in <module> 13 import numpy as np # type: ignore 14 ../../../hostedtoolcache/Python/3.10.0-beta.3/x64/lib/python3.10/site-packages/numpy/__init__.py:391: in <module> 15 raise RuntimeError(msg) 16 E RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, more information is available at https://numpy.org/doc/stable/user/building.html#accelerated-blas-lapack-libraries Otherwise report this to the vendor that provided NumPy. 17 E RankWarning: Polyfit may be poorly conditioned 18 Error: Process completed with exit code 4. ``` | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1} | Test against Python 3.10-dev 929748885 | |
864092515 | https://github.com/simonw/sqlite-utils/pull/277#issuecomment-864092515 | https://api.github.com/repos/simonw/sqlite-utils/issues/277 | MDEyOklzc3VlQ29tbWVudDg2NDA5MjUxNQ== | simonw 9599 | 2021-06-18T14:47:57Z | 2021-06-18T14:47:57Z | OWNER | This is a neat improvement. | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0} | add -h support closes #276 923612361 | |
864099764 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-864099764 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2NDA5OTc2NA== | simonw 9599 | 2021-06-18T14:59:27Z | 2021-06-18T14:59:27Z | OWNER | I'm going to merge this as-is and work on the JSON/TSV support in a separate issue. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
862605436 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862605436 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2MjYwNTQzNg== | simonw 9599 | 2021-06-16T18:19:05Z | 2021-06-16T18:19:05Z | OWNER | `--attach` documentation: https://github.com/simonw/sqlite-utils/blob/192dc2c5b73bd836ab8e2e5fed4b36c6ea02f250/docs/cli.rst#joining-in-memory-data-against-existing-databases-using-attach | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
862046009 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862046009 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2MjA0NjAwOQ== | simonw 9599 | 2021-06-16T05:15:38Z | 2021-06-16T05:15:38Z | OWNER | I'm going to add a `--encoding` option - it will affect ALL CSV input files, so if you have CSV files with different encodings you'll need to sort that mess out yourself (likely by importing each CSV file separately into a database using `sqlite-utils insert` with different `--encoding` values). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
862045639 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045639 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2MjA0NTYzOQ== | simonw 9599 | 2021-06-16T05:14:38Z | 2021-06-16T05:14:38Z | OWNER | Can't share much code though since a bunch of that `insert` stuff is specific to that command - showing progress bars, returning errors on illegal option combinations etc. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
862045438 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045438 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2MjA0NTQzOA== | simonw 9599 | 2021-06-16T05:14:00Z | 2021-06-16T05:14:00Z | OWNER | I should probably refactor the CSV/JSON/loading stuff into a function in `utils.py` in order to share some of the implementation with the existing `sqlite-utils insert` code: https://github.com/simonw/sqlite-utils/blob/287cdcae8908916687f2ecccc87c38549d004ac6/sqlite_utils/cli.py#L691-L734 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
862043974 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862043974 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2MjA0Mzk3NA== | simonw 9599 | 2021-06-16T05:10:12Z | 2021-06-16T05:10:12Z | OWNER | I can stop promoting `:memory:` here and promote `memory` instead: https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#L83-L86 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
862042110 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862042110 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2MjA0MjExMA== | simonw 9599 | 2021-06-16T05:05:51Z | 2021-06-16T05:06:11Z | OWNER | Initial documentation is here: https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#running-queries-directly-against-csv-data It only talks about CSV at the moment - needs to be updated to mention JSON too once that is implemented. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
843702392 | https://github.com/simonw/sqlite-utils/pull/258#issuecomment-843702392 | https://api.github.com/repos/simonw/sqlite-utils/issues/258 | MDEyOklzc3VlQ29tbWVudDg0MzcwMjM5Mg== | simonw 9599 | 2021-05-19T02:47:37Z | 2021-05-19T02:47:37Z | OWNER | I'm going to merge this and add a test - thanks! | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fixing insert from JSON containing strings with non-ascii characters … 868191959 | |
843705533 | https://github.com/simonw/sqlite-utils/pull/254#issuecomment-843705533 | https://api.github.com/repos/simonw/sqlite-utils/issues/254 | MDEyOklzc3VlQ29tbWVudDg0MzcwNTUzMw== | simonw 9599 | 2021-05-19T02:57:22Z | 2021-05-19T02:57:22Z | OWNER | Thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix incorrect create-table cli description 857280617 | |
901338988 | https://github.com/simonw/sqlite-utils/pull/247#issuecomment-901338988 | https://api.github.com/repos/simonw/sqlite-utils/issues/247 | IC_kwDOCGYnMM41uVds | simonw 9599 | 2021-08-18T18:33:39Z | 2021-08-18T18:33:39Z | OWNER | This was also requested in #296. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | FTS quote functionality from datasette 832687563 | |
843705721 | https://github.com/simonw/sqlite-utils/pull/245#issuecomment-843705721 | https://api.github.com/repos/simonw/sqlite-utils/issues/245 | MDEyOklzc3VlQ29tbWVudDg0MzcwNTcyMQ== | simonw 9599 | 2021-05-19T02:58:01Z | 2021-05-19T02:58:01Z | OWNER | Thanks very much. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Correct some typos 830803173 | |
843705806 | https://github.com/simonw/sqlite-utils/pull/244#issuecomment-843705806 | https://api.github.com/repos/simonw/sqlite-utils/issues/244 | MDEyOklzc3VlQ29tbWVudDg0MzcwNTgwNg== | simonw 9599 | 2021-05-19T02:58:18Z | 2021-05-19T02:58:18Z | OWNER | Thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Typo in upsert example 820468864 | |
778841547 | https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778841547 | https://api.github.com/repos/simonw/sqlite-utils/issues/225 | MDEyOklzc3VlQ29tbWVudDc3ODg0MTU0Nw== | simonw 9599 | 2021-02-14T21:04:13Z | 2021-02-14T21:04:13Z | OWNER | I added a test and fixed this in #234 - thanks for the fix. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | fix for problem in Table.insert_all on search for columns per chunk of rows 797159961 | |
778834504 | https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778834504 | https://api.github.com/repos/simonw/sqlite-utils/issues/225 | MDEyOklzc3VlQ29tbWVudDc3ODgzNDUwNA== | simonw 9599 | 2021-02-14T20:09:30Z | 2021-02-14T20:09:30Z | OWNER | Thanks for this. I'm going to try and get the test suite to run in Windows on GitHub Actions. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | fix for problem in Table.insert_all on search for columns per chunk of rows 797159961 | |
778828495 | https://github.com/simonw/sqlite-utils/pull/224#issuecomment-778828495 | https://api.github.com/repos/simonw/sqlite-utils/issues/224 | MDEyOklzc3VlQ29tbWVudDc3ODgyODQ5NQ== | simonw 9599 | 2021-02-14T19:31:06Z | 2021-02-14T19:31:06Z | OWNER | I'm going to add a `offset=` parameter to support this case. Thanks for the suggestion! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add fts offset docs. 792297010 | |
743956666 | https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743956666 | https://api.github.com/repos/simonw/sqlite-utils/issues/208 | MDEyOklzc3VlQ29tbWVudDc0Mzk1NjY2Ng== | simonw 9599 | 2020-12-13T05:44:49Z | 2020-12-13T05:44:49Z | OWNER | Example output: ``` % sqlite-utils analyze-tables github.db tags tags.repo: (1/3) Total rows: 261 Null rows: 0 Blank rows: 0 Distinct values: 14 Most common: 88: 107914493 75: 140912432 27: 206156866 21: 207052882 17: 197431109 8: 197882382 5: 256834907 5: 205429375 4: 248903544 3: 206202864 Least common: 1: 209590345 2: 206649770 2: 303218369 3: 206202864 3: 213286752 4: 248903544 5: 205429375 5: 256834907 8: 197882382 17: 197431109 tags.name: (2/3) Total rows: 261 Null rows: 0 Blank rows: 0 Distinct values: 175 Most common: 10: 0.2 9: 0.1 7: 0.3 6: 0.4 5: 0.7 5: 0.5 5: 0.1a 4: 0.9 4: 0.8 4: 0.6 Least common: 1: 0.1.1 1: 0.11.1 1: 0.1a2 1: 0.20.1 1: 0.21.1 1: 0.21.2 1: 0.21.3 1: 0.22 1: 0.22.1 1: 0.23 tags.sha: (3/3) Total rows: 261 Null rows: 0 Blank rows: 0 Distinct values: 261 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils analyze-tables command and table.analyze_column() method 763320133 | |
743708524 | https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708524 | https://api.github.com/repos/simonw/sqlite-utils/issues/208 | MDEyOklzc3VlQ29tbWVudDc0MzcwODUyNA== | simonw 9599 | 2020-12-12T05:48:20Z | 2020-12-12T05:48:32Z | OWNER | ``` % sqlite-utils analyze-tables ../datasette/fixtures.db facetable --column pk 1/1: ColumnDetails(table='facetable', column='pk', total_rows=15, num_null=0, num_blank=0, num_distinct=15, most_common=None, least_common=None) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils analyze-tables command and table.analyze_column() method 763320133 | |
743708325 | https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708325 | https://api.github.com/repos/simonw/sqlite-utils/issues/208 | MDEyOklzc3VlQ29tbWVudDc0MzcwODMyNQ== | simonw 9599 | 2020-12-12T05:46:27Z | 2020-12-12T05:46:27Z | OWNER | It would be neat if you could optionally specify a subset of columns to analyze, using `-c` or `--column`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils analyze-tables command and table.analyze_column() method 763320133 | |
743708169 | https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708169 | https://api.github.com/repos/simonw/sqlite-utils/issues/208 | MDEyOklzc3VlQ29tbWVudDc0MzcwODE2OQ== | simonw 9599 | 2020-12-12T05:44:46Z | 2020-12-12T05:44:46Z | OWNER | If there are less than ten values is it worth outputting them twice, once in `most_common` and then in reverse in `least_common`? Feels redundant - I think I should leave `least_common` empty in that case. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils analyze-tables command and table.analyze_column() method 763320133 | |
743708080 | https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708080 | https://api.github.com/repos/simonw/sqlite-utils/issues/208 | MDEyOklzc3VlQ29tbWVudDc0MzcwODA4MA== | simonw 9599 | 2020-12-12T05:43:45Z | 2020-12-12T05:43:45Z | OWNER | CLI output looks like this at the moment, which is bad: ``` % sqlite-utils analyze-tables ../datasette/fixtures.db facetable 1/10: ColumnDetails(table='facetable', column='pk', total_rows=15, num_null=0, num_blank=0, num_distinct=15, most_common=None, least_common=None) 2/10: ColumnDetails(table='facetable', column='created', total_rows=15, num_null=0, num_blank=0, num_distinct=4, most_common=[('2019-01-17 08:00:00', 4), ('2019-01-15 08:00:00', 4), ('2019-01-14 08:00:00', 4), ('2019-01-16 08:00:00', 3)], least_common=[('2019-01-16 08:00:00', 3), ('2019-01-14 08:00:00', 4), ('2019-01-15 08:00:00', 4), ('2019-01-17 08:00:00', 4)]) 3/10: ColumnDetails(table='facetable', column='planet_int', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[(1, 14), (2, 1)], least_common=[(2, 1), (1, 14)]) 4/10: ColumnDetails(table='facetable', column='on_earth', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[(1, 14), (0, 1)], least_common=[(0, 1), (1, 14)]) 5/10: ColumnDetails(table='facetable', column='state', total_rows=15, num_null=0, num_blank=0, num_distinct=3, most_common=[('CA', 10), ('MI', 4), ('MC', 1)], least_common=[('MC', 1), ('MI', 4), ('CA', 10)]) 6/10: ColumnDetails(table='facetable', column='city_id', total_rows=15, num_null=0, num_blank=0, num_distinct=4, most_common=[(1, 6), (3, 4), (2, 4), (4, 1)], least_common=[(4, 1), (2, 4), (3, 4), (1, 6)]) 7/10: ColumnDetails(table='facetable', column='neighborhood', total_rows=15, num_null=0, num_blank=0, num_distinct=14, most_common=[('Downtown', 2), ('Tenderloin', 1), ('SOMA', 1), ('Mission', 1), ('Mexicantown', 1), ('Los Feliz', 1), ('Koreatown', 1), ('Hollywood', 1), ('Hayes Valley', 1), ('Greektown', 1)], least_common=[('Arcadia Planitia', 1), ('Bernal Heights', 1), ('Corktown', 1), ('Dogpatch', 1), ('Greektown', 1), ('Hayes Valley', 1), ('Hollywood', 1), ('Koreatown', 1), ('Los Feliz', 1), ('Mexicantown', 1)]) 8/10: ColumnDetails(table='facetable', column='tags', total_rows=15, num_null=0, num_blank=0, num_distinct=3,… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils analyze-tables command and table.analyze_column() method 763320133 | |
743707969 | https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743707969 | https://api.github.com/repos/simonw/sqlite-utils/issues/208 | MDEyOklzc3VlQ29tbWVudDc0MzcwNzk2OQ== | simonw 9599 | 2020-12-12T05:42:26Z | 2020-12-12T05:43:06Z | OWNER | Should truncate values in the least/most common JSON array to a sensible length, otherwise you end up with stuff like this: ```json [ [ "b'\\x00\\x05barry\\x03\\x01\\x02\\x00\\x00\\x03cat\\x03\\x01\\x03\\x00\\x00\\x03dog\\x08\\x01\\x01\\x01\\x03\\x00\\x01\\x03\\x00\\x00\\x07panther\\x05\\x01\\x01\\x02\\x02\\x00\\x01\\x03uma\\x05\\x02\\x01\\x02\\x02\\x00\\x00\\x04sara\\x05\\x02\\x01\\x01\\x02\\x00\\x00\\x05terry\\x08\\x01\\x01\\x01\\x02\\x00\\x01\\x02\\x00\\x00\\x06weasel\\x05\\x02\\x01\\x01\\x03\\x00'", 1 ] ] ``` This example also shows that binary values (like those in `_fts` tables) look a bit weird, but I think I'm OK with that since binary data can't be represented neatly in JSON anyway. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | sqlite-utils analyze-tables command and table.analyze_column() method 763320133 | |
740796067 | https://github.com/simonw/sqlite-utils/pull/204#issuecomment-740796067 | https://api.github.com/repos/simonw/sqlite-utils/issues/204 | MDEyOklzc3VlQ29tbWVudDc0MDc5NjA2Nw== | simonw 9599 | 2020-12-08T17:49:22Z | 2020-12-08T17:49:22Z | OWNER | Great catch, thank you. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | use jsonify_if_need for sql updates 752888228 | |
753567932 | https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932 | https://api.github.com/repos/simonw/sqlite-utils/issues/203 | MDEyOklzc3VlQ29tbWVudDc1MzU2NzkzMg== | simonw 9599 | 2021-01-03T04:54:43Z | 2021-01-03T04:54:43Z | OWNER | Another option: expand the `ForeignKey` object to have `.columns` and `.other_columns` properties in addition to the existing `.column` and `.other_column` properties. These new plural properties would always return a tuple, which would be a one-item tuple for a non-compound-foreign-key. The question then is what should `.column` and `.other_column` return for compound foreign keys? I'd be inclined to say they should return `None` - which would trigger errors in code that encounters a compound foreign key for the first time, but those errors would at least be a strong indicator as to what had gone wrong. We can label `.column` and `.other_column` as deprecated and then remove them in `sqlite-utils 4.0`. Since this would still be a breaking change in some minor edge-cases I'm thinking maybe 4.0 needs to happen in order to land this feature. I'm not opposed to doing that, I was just hoping it might be avoidable. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | changes to allow for compound foreign keys 743384829 | |
753567744 | https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567744 | https://api.github.com/repos/simonw/sqlite-utils/issues/203 | MDEyOklzc3VlQ29tbWVudDc1MzU2Nzc0NA== | simonw 9599 | 2021-01-03T04:51:44Z | 2021-01-03T04:51:44Z | OWNER | One way that this could avoid a breaking change would be to have `fk.column` and `fk.other_column` remain as strings for non-compound-foreign-keys, but turn into tuples for a compound foreign key. This is a bit of an ugly API design, and it could still break existing code that encounters a compound foreign key for the first time - but it would leave code working for the more common case of a non-compound-foreign-key. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | changes to allow for compound foreign keys 743384829 | |
753567508 | https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567508 | https://api.github.com/repos/simonw/sqlite-utils/issues/203 | MDEyOklzc3VlQ29tbWVudDc1MzU2NzUwOA== | simonw 9599 | 2021-01-03T04:48:17Z | 2021-01-03T04:48:17Z | OWNER | Sorry for taking so long to review this! This approach looks great to me - being able to optionally pass a tuple anywhere the API currently expects a column is smart, and it's consistent with how the `pk=` parameter works elsewhere. There's just one problem I can see with this: the way it changes the `ForeignKey(...)` interface to always return a tuple for `.column` and `.other_column`, even if that tuple only contains a single item. This represents a breaking change to the existing API - any code that expects `ForeignKey.column` to be a single string (which is any code that has been written against that) will break. As such, I'd have to bump the major version of `sqlite-utils` to `4.0` in order to ship this. Ideally I'd like to make this change in a way that doesn't represent an API compatibility break. I need to think a bit harder about how that might be achieved. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | changes to allow for compound foreign keys 743384829 | |
743966289 | https://github.com/simonw/sqlite-utils/pull/203#issuecomment-743966289 | https://api.github.com/repos/simonw/sqlite-utils/issues/203 | MDEyOklzc3VlQ29tbWVudDc0Mzk2NjI4OQ== | simonw 9599 | 2020-12-13T07:20:51Z | 2020-12-13T07:20:51Z | OWNER | Sorry for not reviewing this yet! I'll try to carve out time to look at it in the next few days. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | changes to allow for compound foreign keys 743384829 | |
723148906 | https://github.com/simonw/sqlite-utils/pull/195#issuecomment-723148906 | https://api.github.com/repos/simonw/sqlite-utils/issues/195 | MDEyOklzc3VlQ29tbWVudDcyMzE0ODkwNg== | simonw 9599 | 2020-11-06T15:43:51Z | 2020-11-06T15:43:51Z | OWNER | Thanks to #198 (introducing a `rank_bm25()` custom function for FTS4) this feature will be able to offer relevance search for both FTS5 AND FTS4 tables. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.search() improvements plus sqlite-utils search command 735663855 | |
722542895 | https://github.com/simonw/sqlite-utils/pull/195#issuecomment-722542895 | https://api.github.com/repos/simonw/sqlite-utils/issues/195 | MDEyOklzc3VlQ29tbWVudDcyMjU0Mjg5NQ== | simonw 9599 | 2020-11-05T18:01:33Z | 2020-11-05T18:01:33Z | OWNER | Latest test failure: ``` 114 -> assert [("racoons are biting trash pandas", "USA", "bar")] == table.search( 115 "bite", order="rowid" 116 ) 117 118 119 def test_optimize_fts(fresh_db): (Pdb) table.search("bite") [(2, 'racoons are biting trash pandas', 'USA', 'bar', -9.641434262948206e-07)] ``` The problem here is that the `table.search()` method now behaves differently for FTS4 v.s. FTS5 tables. With FTS4 you get back just the table columns. With FTS5 you also get back the `rowid` as the first column and the `rank` score as the last column. This is weird. It also makes me question whether having `.search()` return a list of tuples is the right API design. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.search() improvements plus sqlite-utils search command 735663855 | |
721397665 | https://github.com/simonw/sqlite-utils/pull/195#issuecomment-721397665 | https://api.github.com/repos/simonw/sqlite-utils/issues/195 | MDEyOklzc3VlQ29tbWVudDcyMTM5NzY2NQ== | simonw 9599 | 2020-11-03T22:02:57Z | 2020-11-03T22:02:57Z | OWNER | Documentation so far: https://github.com/simonw/sqlite-utils/blob/6cadc6103ff1ba58c6409ce7fba74259e72965d9/docs/cli.rst#executing-searches | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.search() improvements plus sqlite-utils search command 735663855 | |
717361487 | https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717361487 | https://api.github.com/repos/simonw/sqlite-utils/issues/189 | MDEyOklzc3VlQ29tbWVudDcxNzM2MTQ4Nw== | simonw 9599 | 2020-10-27T16:24:04Z | 2020-10-27T16:24:04Z | OWNER | This is great, thank you very much. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Allow iterables other than Lists in m2m records 729818242 | |
716756103 | https://github.com/simonw/sqlite-utils/pull/189#issuecomment-716756103 | https://api.github.com/repos/simonw/sqlite-utils/issues/189 | MDEyOklzc3VlQ29tbWVudDcxNjc1NjEwMw== | simonw 9599 | 2020-10-26T18:56:19Z | 2020-10-26T18:56:19Z | OWNER | This is a great fix, thanks! If you add a unit test somewhere in here I'll merge the PR: https://github.com/simonw/sqlite-utils/blob/main/tests/test_m2m.py | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Allow iterables other than Lists in m2m records 729818242 | |
701627158 | https://github.com/simonw/sqlite-utils/pull/178#issuecomment-701627158 | https://api.github.com/repos/simonw/sqlite-utils/issues/178 | MDEyOklzc3VlQ29tbWVudDcwMTYyNzE1OA== | simonw 9599 | 2020-09-30T20:29:11Z | 2020-09-30T20:29:11Z | OWNER | Thanks for the fix! | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Update README.md 709043182 | |
698400790 | https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698400790 | https://api.github.com/repos/simonw/sqlite-utils/issues/174 | MDEyOklzc3VlQ29tbWVudDY5ODQwMDc5MA== | simonw 9599 | 2020-09-24T14:59:50Z | 2020-09-24T14:59:50Z | OWNER | For reusing the lookup table: I'm going to raise an error if a lookup table exists but without the correct columns. The caller can then add those columns and try again. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Much, much faster extract() implementation 707944044 | |
698184166 | https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698184166 | https://api.github.com/repos/simonw/sqlite-utils/issues/174 | MDEyOklzc3VlQ29tbWVudDY5ODE4NDE2Ng== | simonw 9599 | 2020-09-24T08:01:07Z | 2020-09-24T08:01:07Z | OWNER | I may revert the now unnecessary undocumented tweaks to the `.update()` method made in 66d506587eba9f0715267d6560b97c1fa44cc781 as well. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Much, much faster extract() implementation 707944044 | |
698182656 | https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698182656 | https://api.github.com/repos/simonw/sqlite-utils/issues/174 | MDEyOklzc3VlQ29tbWVudDY5ODE4MjY1Ng== | simonw 9599 | 2020-09-24T07:58:08Z | 2020-09-24T07:58:08Z | OWNER | The way the lookup table works here differs from the previous implementation. In the previous implementation the usage of `.lookup()` meant that an existing table would be modified to fit the new purpose. That no longer happens in this version. Need to make a design decision about how this should work. It should definitely be possible to use an existing lookup table - imagine a database where several tables have a "Departments" column and we want to extract all of those values out to a single shared "Departments" table. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Much, much faster extract() implementation 707944044 | |
698182037 | https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698182037 | https://api.github.com/repos/simonw/sqlite-utils/issues/174 | MDEyOklzc3VlQ29tbWVudDY5ODE4MjAzNw== | simonw 9599 | 2020-09-24T07:56:50Z | 2020-09-24T07:56:50Z | OWNER | I could also be a bit smarter about transaction handling. I think it may be possible to run this entire operation in a single transaction now. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Much, much faster extract() implementation 707944044 | |
698181478 | https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698181478 | https://api.github.com/repos/simonw/sqlite-utils/issues/174 | MDEyOklzc3VlQ29tbWVudDY5ODE4MTQ3OA== | simonw 9599 | 2020-09-24T07:55:45Z | 2020-09-24T07:55:45Z | OWNER | `import functools` is no longer needed. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Much, much faster extract() implementation 707944044 | |
698180705 | https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698180705 | https://api.github.com/repos/simonw/sqlite-utils/issues/174 | MDEyOklzc3VlQ29tbWVudDY5ODE4MDcwNQ== | simonw 9599 | 2020-09-24T07:54:10Z | 2020-09-24T07:54:10Z | OWNER | After running through the steps in https://simonwillison.net/2020/Sep/23/sqlite-utils-extract/ I get a table that looks like this: <img width="1699" alt="salaries__salaries__683_558_rows" src="https://user-images.githubusercontent.com/9599/94116875-666b8900-fe00-11ea-9e97-2b9ccbfeae29.png"> The foreign key columns are all at the end of the table. It would be nicer if they were arranged in the same order as the columns they replaced. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Much, much faster extract() implementation 707944044 | |
698180113 | https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698180113 | https://api.github.com/repos/simonw/sqlite-utils/issues/174 | MDEyOklzc3VlQ29tbWVudDY5ODE4MDExMw== | simonw 9599 | 2020-09-24T07:53:03Z | 2020-09-24T07:53:03Z | OWNER | This could do with a little bit more testing - I'm worried there may be column or table name edge cases that are not covered yet. I also need to remove the progress bar code since that no longer makes sense for this implementation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Much, much faster extract() implementation 707944044 | |
696494070 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696494070 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ5NDA3MA== | simonw 9599 | 2020-09-22T03:48:58Z | 2020-09-22T03:48:58Z | OWNER | One last thing. https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_change says that the first step should be: > If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. And the last steps should be: > If foreign key constraints were originally enabled then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints. > > Commit the transaction started in step 2. > > If foreign keys constraints were originally enabled, reenable them now. I need to implement that. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
696490851 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696490851 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ5MDg1MQ== | simonw 9599 | 2020-09-22T03:33:54Z | 2020-09-22T03:33:54Z | OWNER | It would be neat if `.transform(pk=None)` converted a primary key table to a rowid table. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
696488201 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696488201 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ4ODIwMQ== | simonw 9599 | 2020-09-22T03:21:16Z | 2020-09-22T03:21:16Z | OWNER | Just needs documentation now. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
696485791 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696485791 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ4NTc5MQ== | simonw 9599 | 2020-09-22T03:10:15Z | 2020-09-22T03:10:15Z | OWNER | Design decision needed on foreign keys: what does the syntax look like for removing an existing foreign key? Since I already have a good implementation of `add_foreign_key()` I'm tempted to only support dropping them. Maybe like this: ```python table.transform(drop_foreign_keys=[("author_id", "author", "id")]) ``` It's a bit crufty but it's such a rare use-case that I think this will be good enough. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
696480925 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696480925 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ4MDkyNQ== | simonw 9599 | 2020-09-22T02:45:47Z | 2020-09-22T02:45:47Z | OWNER | I'm not going to do `conversions=` because it would be inconsistent with how they work elsewhere. The SQL generated by this function looks like this: INSERT INTO dogs_new_tmp VALUES (a, b) SELECT a, b from dogs; So passing `conversions={"name": "upper(?)"})` wouldn't make sense, since we're not using arguments hence there is no-where for that `?` to go. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
696446658 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696446658 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0NjY1OA== | simonw 9599 | 2020-09-22T00:13:55Z | 2020-09-22T00:14:21Z | OWNER | Idea: allow a `conversions=` parameter, as seen on `.insert_all()` and friends, which lets you apply a SQL transformation function as part of the operation. E.g.: ```python table.transform({"age": int}, conversions={"name": "upper(?)"}) ``` https://sqlite-utils.readthedocs.io/en/stable/python-api.html#converting-column-values-using-sql-functions | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
696445766 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696445766 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0NTc2Ng== | simonw 9599 | 2020-09-22T00:10:50Z | 2020-09-22T00:11:12Z | OWNER | A less horrible interface might be the following: ```python # Ensure the 'age' column is not null: table.transform(not_null={"age"}) # The 'age' column is not null but I don't want it to be: table.transform(not_null={"age": False}) ``` So if the argument is a set it means "make sure these are all not null" - if the argument is a dictionary it means "set these to be null or not null depending on if their dictionary value is true or false". | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
696444842 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444842 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0NDg0Mg== | simonw 9599 | 2020-09-22T00:07:43Z | 2020-09-22T00:09:05Z | OWNER | Syntax challenge: I could use `.transform(defaults={"age": None})` to indicate that the `age` column should have its default removed, but how would I tell `.transform()` that the `age` column, currently `not null`, should have the `not null` removed from it? I could do this: `.transform(not_not_null={"age"})` - it's a bit gross but it's also kind of funny. I actually like it! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
696444353 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444353 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0NDM1Mw== | simonw 9599 | 2020-09-22T00:06:12Z | 2020-09-22T00:06:12Z | OWNER | I should support `not_null=` and `default=` arguments to the `.transform()` method because it looks like you can't use `ALTER TABLE` to change those. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
696443845 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443845 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0Mzg0NQ== | simonw 9599 | 2020-09-22T00:04:31Z | 2020-09-22T00:04:44Z | OWNER | Good news: the `.columns` introspection does tell me those things: ``` >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db.create_table("foo", {"id": int, "name": str, "age": int}, defaults={"age": 1}, not_null={"name", "age"}) <Table foo (id, name, age)> >>> db["foo"] <Table foo (id, name, age)> >>> print(db["foo"].schema) CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT NOT NULL, [age] INTEGER NOT NULL DEFAULT 1 ) >>> db["foo"].columns [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value=None, is_pk=0), Column(cid=2, name='age', type='INTEGER', notnull=1, default_value='1', is_pk=0)] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
696443190 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443190 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0MzE5MA== | simonw 9599 | 2020-09-22T00:02:22Z | 2020-09-22T00:02:22Z | OWNER | How would I detect which columns are `not_null` and what their defaults are? I don`t think my introspection logic handles that yet. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
696443042 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443042 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0MzA0Mg== | simonw 9599 | 2020-09-22T00:01:50Z | 2020-09-22T00:01:50Z | OWNER | When you transform a table, it should keep its primary key, foreign keys, not_null and defaults. I don't think it needs to care about `hash_id` or `extracts=` since those don't affect the structure of the table as it is being created - well, `hash_id` does but if we are transforming an existing table we will get the `hash_id` column for free. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
696442621 | https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696442621 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | MDEyOklzc3VlQ29tbWVudDY5NjQ0MjYyMQ== | simonw 9599 | 2020-09-22T00:00:23Z | 2020-09-22T00:00:23Z | OWNER | I still need to figure out what to do about these various other table properties: https://github.com/simonw/sqlite-utils/blob/b34c9b40c206d7a9d7ee57a8c1f198ff1f522735/sqlite_utils/db.py#L775-L787 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.transform() method 705975133 | |
693199392 | https://github.com/simonw/sqlite-utils/pull/158#issuecomment-693199392 | https://api.github.com/repos/simonw/sqlite-utils/issues/158 | MDEyOklzc3VlQ29tbWVudDY5MzE5OTM5Mg== | simonw 9599 | 2020-09-16T06:21:29Z | 2020-09-16T06:21:29Z | OWNER | Thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix accidental mega long line in docs 697203800 | |
689735140 | https://github.com/simonw/sqlite-utils/pull/156#issuecomment-689735140 | https://api.github.com/repos/simonw/sqlite-utils/issues/156 | MDEyOklzc3VlQ29tbWVudDY4OTczNTE0MA== | simonw 9599 | 2020-09-09T18:21:06Z | 2020-09-09T18:21:06Z | OWNER | Good spot, thanks. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Typos in tests 697030843 | |
689185393 | https://github.com/simonw/sqlite-utils/pull/146#issuecomment-689185393 | https://api.github.com/repos/simonw/sqlite-utils/issues/146 | MDEyOklzc3VlQ29tbWVudDY4OTE4NTM5Mw== | simonw 9599 | 2020-09-08T23:17:42Z | 2020-09-08T23:17:42Z | OWNER | That seems like a reasonable approach to me, especially since this is going to be a pretty rare edge-case. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Handle case where subsequent records (after first batch) include extra columns 688668680 | |
688508510 | https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688508510 | https://api.github.com/repos/simonw/sqlite-utils/issues/146 | MDEyOklzc3VlQ29tbWVudDY4ODUwODUxMA== | simonw 9599 | 2020-09-07T20:56:03Z | 2020-09-07T20:56:24Z | OWNER | The problem with this approach is that it requires us to consume the entire iterator before we can start inserting rows into the table - here on line 1052: https://github.com/simonw/sqlite-utils/blob/bb131793feac16bc7181ab997568f941b0220ef2/sqlite_utils/db.py#L1047-L1054 I designed the `.insert_all()` to avoid doing this, because I want to be able to pass it an iterator (or more likely a generator) that could produce potentially millions of records. Doing things one batch of 100 records at a time means that the Python process doesn't need to pull millions of records into memory at once. `db-to-sqlite` is one example of a tool that uses that characteristic, in https://github.com/simonw/db-to-sqlite/blob/63e4ee972f292de13bb11767c0fb64b35339d954/db_to_sqlite/cli.py#L94-L106 So we need to solve this issue without consuming the entire iterator with a `records = list(records)` call. I think one way to do this is to execute each chunk one at a time and watch out for an exception that indicates that we sent too many parameters - then adjust the chunk size down and try again. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Handle case where subsequent records (after first batch) include extra columns 688668680 | |
683173375 | https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683173375 | https://api.github.com/repos/simonw/sqlite-utils/issues/142 | MDEyOklzc3VlQ29tbWVudDY4MzE3MzM3NQ== | simonw 9599 | 2020-08-28T22:29:02Z | 2020-08-28T22:29:02Z | OWNER | Yeah I think that failure is actually because there's a brand new release of Black out and it subtly changes some of the formatting rules. I'll merge this and then run Black against the entire codebase. | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | insert_all(..., alter=True) should work for new columns introduced after the first 100 records 688386219 | |
683172829 | https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172829 | https://api.github.com/repos/simonw/sqlite-utils/issues/142 | MDEyOklzc3VlQ29tbWVudDY4MzE3MjgyOQ== | simonw 9599 | 2020-08-28T22:27:05Z | 2020-08-28T22:27:05Z | OWNER | Looks like it failed the "black" formatting test - possibly because there's a new release if black out. I'm going to merge despite that failure. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | insert_all(..., alter=True) should work for new columns introduced after the first 100 records 688386219 | |
683172082 | https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172082 | https://api.github.com/repos/simonw/sqlite-utils/issues/142 | MDEyOklzc3VlQ29tbWVudDY4MzE3MjA4Mg== | simonw 9599 | 2020-08-28T22:24:25Z | 2020-08-28T22:24:25Z | OWNER | Thanks very much! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | insert_all(..., alter=True) should work for new columns introduced after the first 100 records 688386219 | |
655289686 | https://github.com/simonw/sqlite-utils/pull/120#issuecomment-655289686 | https://api.github.com/repos/simonw/sqlite-utils/issues/120 | MDEyOklzc3VlQ29tbWVudDY1NTI4OTY4Ng== | simonw 9599 | 2020-07-08T05:13:11Z | 2020-07-08T05:13:11Z | OWNER | This is an excellent fix, thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix query command's support for DML 652816158 | |
655653292 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655653292 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTY1MzI5Mg== | simonw 9599 | 2020-07-08T17:26:02Z | 2020-07-08T17:26:02Z | OWNER | Awesome, thank you very much. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add insert --truncate option 651844316 | |
655286864 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655286864 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTI4Njg2NA== | simonw 9599 | 2020-07-08T05:05:27Z | 2020-07-08T05:05:36Z | OWNER | The only thing missing from this PR is updates to the documentation. Those need to go in two places: - In the Python API docs. I suggest adding a note to this section about bulk inserts: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/python-api.rst#bulk-inserts - In the CLI docs, in this section: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/cli.rst#inserting-json-data Here's an example of a previous commit that includes updates to both CLI and API documentation: https://github.com/simonw/sqlite-utils/commit/f9473ace14878212c1fa968b7bd2f51e4f064dba#diff-e3e2a9bfd88566b05001b02a3f51d286 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add insert --truncate option 651844316 | |
655284168 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284168 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTI4NDE2OA== | simonw 9599 | 2020-07-08T04:58:00Z | 2020-07-08T04:58:00Z | OWNER | Oops didn't mean to click "close" there. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add insert --truncate option 651844316 | |
655284054 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284054 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTI4NDA1NA== | simonw 9599 | 2020-07-08T04:57:38Z | 2020-07-08T04:57:38Z | OWNER | Thoughts on transactions would be much appreciated in #121 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add insert --truncate option 651844316 | |
655283393 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTI4MzM5Mw== | simonw 9599 | 2020-07-08T04:55:18Z | 2020-07-08T04:55:18Z | OWNER | This is a really good idea - and thank you for the detailed discussion in the pull request. I'm keen to discuss how transactions can work better. I tend to use this pattern in my own code: with db.conn: db["table"].insert(...) But it's not documented and I've not though very hard about it! I like having inserts that handle 10,000+ rows commit on every chunk so I can watch their progress from another process, but the library should absolutely support people who want to commit all of the rows in a single transaction - or combine changes with DML. Lots to discuss here. I'll start a new issue. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add insert --truncate option 651844316 | |
612727814 | https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727814 | https://api.github.com/repos/simonw/sqlite-utils/issues/99 | MDEyOklzc3VlQ29tbWVudDYxMjcyNzgxNA== | simonw 9599 | 2020-04-13T03:05:04Z | 2020-04-13T03:05:04Z | OWNER | Bit trick from an implementation point of view this, since we want to be able to handle input that is a generator - so we can't scan through the input to validate that every dictionary has the same exact keys without consuming the entire iterator. The alternative would be to raise an error the first time we spot a dictionary with keys that differ... but that's weird because we commit changes in batches, so we may end up only applying half of the changes before exiting with the error. On that basis, I'm going to leave this as-is and mark this as wontfix. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .upsert_all() should maybe error if dictionaries passed to it do not have the same keys 598640234 | |
612727400 | https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727400 | https://api.github.com/repos/simonw/sqlite-utils/issues/99 | MDEyOklzc3VlQ29tbWVudDYxMjcyNzQwMA== | simonw 9599 | 2020-04-13T03:03:09Z | 2020-04-13T03:03:09Z | OWNER | I think I'm going to leave this as intended behaviour. Or maybe passing multiple dictionaries to `.upsert_all()` with different numbers of keys should raise an error? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .upsert_all() should maybe error if dictionaries passed to it do not have the same keys 598640234 | |
612728047 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612728047 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | MDEyOklzc3VlQ29tbWVudDYxMjcyODA0Nw== | simonw 9599 | 2020-04-13T03:06:10Z | 2020-04-13T03:06:10Z | OWNER | Implementation plan: `.insert_all()` and `.upsert_all()` should only set `.last_rowid` and `last_pk` if they were called with a single item. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
612708274 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612708274 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | MDEyOklzc3VlQ29tbWVudDYxMjcwODI3NA== | simonw 9599 | 2020-04-13T01:25:59Z | 2020-04-13T01:26:11Z | OWNER | In mucking around with `sqlite3` it looks like `result.lastrowid` is indeed populated for `UPDATE` - in this case with the last inserted rowid in the table. This differs from the documented behaviour I linked to above. ``` In [1]: import sqlite3 In [2]: c = sqlite3.connect(":memory:") In [3]: c Out[3]: <sqlite3.Connection at 0x103c4d490> In [4]: c.execute('create table foo (bar integer);') Out[4]: <sqlite3.Cursor at 0x103f760a0> In [5]: c.execute('insert into foo (bar) values (1)') Out[5]: <sqlite3.Cursor at 0x103f76650> In [6]: c.execute('select * from foo').fetchall() Out[6]: [(1,)] In [7]: c.execute('insert into foo (bar) values (1)') Out[7]: <sqlite3.Cursor at 0x103f766c0> In [8]: c.execute('select * from foo').fetchall() Out[8]: [(1,), (1,)] In [9]: c.execute('insert into foo (bar) values (1)').lastrowid Out[9]: 3 In [10]: c.execute('select * from foo').fetchall() Out[10]: [(1,), (1,), (1,)] In [11]: c.execute('select rowid, bar from foo').fetchall() Out[11]: [(1, 1), (2, 1), (3, 1)] In [12]: c.execute('insert into foo (bar) values (1)').lastrowid Out[12]: 4 In [13]: c.execute('select rowid, bar from foo').fetchall() Out[13]: [(1, 1), (2, 1), (3, 1), (4, 1)] In [14]: r = c.execute('update foo set bar =2 where rowid = 1') In [15]: r.lastrowid Out[15]: 4 In [16]: c.execute('select rowid, bar from foo').fetchall() Out[16]: [(1, 2), (2, 1), (3, 1), (4, 1)] In [17]: r = c.execute('select rowid, bar from foo') In… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
612707828 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707828 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | MDEyOklzc3VlQ29tbWVudDYxMjcwNzgyOA== | simonw 9599 | 2020-04-13T01:24:05Z | 2020-04-13T01:24:16Z | OWNER | Why do I even care about `lastrowid` here? I'm trying to ensure that after you insert or upsert a row you can use `table.last_pk` to start doing things like building additional foreign key relationships. So maybe it doesn't make sense to make `.last_pk` available _at all_ for cases where you called `.upsert_all()` or `.insert_all()` - it should just be populated for `.upsert()` and `.insert()`. The documentation doesn't say it should work for `.upsert_all()` - it's only documented for the single actions. https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1113-L1124 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
612707293 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707293 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | MDEyOklzc3VlQ29tbWVudDYxMjcwNzI5Mw== | simonw 9599 | 2020-04-13T01:21:22Z | 2020-04-13T01:21:22Z | OWNER | I have a hunch that the root of the problem here is that accessing `result.lastrowid` during my version of an `.upsert()` doesn't actually make sense: https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1102-L1113 In the bug I'm seeing (which I still haven't reduced to a reproducible test) the debugger shows me this at that point: ``` (Pdb) query 'UPDATE [files] SET [createdAt] = ?, [ext] = ?, [updatedAt] = ?, [uri] = ?, [uriType] = ? WHERE [project] = ? AND [name] = ?' (Pdb) params ['2020-03-04T04:04:40.152000+00:00', 'csv', '2020-03-04T04:04:40.152000+00:00', 'https://storage.googleapis.com/bln_prod/...', 'download', 'UHJvamVjdDo4MTgyMjU2Ny01ZjI0LTQxM2ItYWZmNi05NTlmNGY3MjExMjI=', 'loans_to_documentation.csv'] (Pdb) result.lastrowid 100 ``` But here's the weird thing... there's no row in the table with a rowid of 100! ``` (Pdb) [r['rowid'] for r in self.db.execute_returning_dicts('select rowid, * from files')] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99] ``` So what the heck is going on? The last SQL statement I executed here was an `UPDATE`. The `lastrowid` docs say: https://kite.com/python/docs/sqlite3.Cursor.lastrowid > This read-only attribute provides the rowid of the last modified row. It is only set if you issued a INSERT statement using the execute() method. For operations other than INSERT or when executemany() is called, lastrowid is set to None. So where did that `100` come from? It should be `None`! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
612258687 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612258687 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | MDEyOklzc3VlQ29tbWVudDYxMjI1ODY4Nw== | simonw 9599 | 2020-04-10T23:08:48Z | 2020-04-10T23:08:48Z | OWNER | I need a test that reproduces this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
612173156 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612173156 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | MDEyOklzc3VlQ29tbWVudDYxMjE3MzE1Ng== | simonw 9599 | 2020-04-10T19:03:32Z | 2020-04-10T23:08:28Z | OWNER | Investigate this traceback: ``` Traceback (most recent call last): File "fetch_projects.py", line 60, in <module> fetch_projects(db, token) File "fetch_projects.py", line 41, in fetch_projects db["projects"].upsert(project, pk="id") File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py", line 1139, in upsert conversions=conversions, File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py", line 1168, in upsert_all upsert=True, File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py", line 1107, in insert_all row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0] IndexError: list index out of range ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
612738311 | https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612738311 | https://api.github.com/repos/simonw/sqlite-utils/issues/97 | MDEyOklzc3VlQ29tbWVudDYxMjczODMxMQ== | simonw 9599 | 2020-04-13T03:55:11Z | 2020-04-13T03:55:11Z | OWNER | Shipped in 2.5 - documentation is here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#connecting-to-or-creating-a-database | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Adding a "recreate" flag to the `Database` constructor 593751293 | |
612732453 | https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732453 | https://api.github.com/repos/simonw/sqlite-utils/issues/97 | MDEyOklzc3VlQ29tbWVudDYxMjczMjQ1Mw== | simonw 9599 | 2020-04-13T03:26:46Z | 2020-04-13T03:26:46Z | OWNER | I wonder if it should delete an recreate the file or if it would be safer to drop every table instead? Dropping tables gets messy: then you need to drop triggers and views, and you need to run `vacuum` to clean up the space. My worry with deleting and recreating the file is that it could trigger errors in other processes that are currently attached to that database file. But... if you know that's going to be likely, maybe you shouldn't use the `recreate=True` feature? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Adding a "recreate" flag to the `Database` constructor 593751293 | |
612732129 | https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732129 | https://api.github.com/repos/simonw/sqlite-utils/issues/97 | MDEyOklzc3VlQ29tbWVudDYxMjczMjEyOQ== | simonw 9599 | 2020-04-13T03:25:29Z | 2020-04-13T03:25:29Z | OWNER | Interesting thought. I've run into this myself a lot - many of my scripts intend to create the database from scratch, so I end up running `!rm /tmp/blah.db` in Jupyter and occasionally getting errors if the file doesn't exist. I think adding `recreate=True` could make sense. It could throw an error if you attempt to use it after passing in something other than a path to a file on disk. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Adding a "recreate" flag to the `Database` constructor 593751293 | |
599247833 | https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599247833 | https://api.github.com/repos/simonw/sqlite-utils/issues/92 | MDEyOklzc3VlQ29tbWVudDU5OTI0NzgzMw== | simonw 9599 | 2020-03-15T18:37:28Z | 2020-03-15T18:37:43Z | OWNER | Released in 2.4.2. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .columns_dict doesn't work for all possible column types 581339961 | |
599128891 | https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599128891 | https://api.github.com/repos/simonw/sqlite-utils/issues/92 | MDEyOklzc3VlQ29tbWVudDU5OTEyODg5MQ== | simonw 9599 | 2020-03-14T20:03:45Z | 2020-03-14T20:03:45Z | OWNER | I'm going to keep treating them as `str`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .columns_dict doesn't work for all possible column types 581339961 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
updated_at (date) >30 ✖