issues
2,892 rows sorted by author_association
This data as json, CSV (advanced)
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association ▼ | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
594553553 | MDExOlB1bGxSZXF1ZXN0Mzk5MTY2NDMz | 719 | asgi: check raw_path is not None | cldellow 193185 | closed | 0 | 1 | 2020-04-05T16:53:58Z | 2020-05-04T17:14:26Z | 2020-05-04T17:14:26Z | CONTRIBUTOR | simonw/datasette/pulls/719 | The ASGI spec (https://asgi.readthedocs.io/en/latest/specs/www.html#http) seems to imply that `None` is a valid value, so we need to check the value itself, not just whether the key is present. In particular, the [mangum](https://github.com/erm/mangum) adapter passes `None` for this key's value. This change permits mangum to be used to front datasette in Amazon API Gateway + AWS Lambda deployments. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/719/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
596245802 | MDExOlB1bGxSZXF1ZXN0NDAwNTc4OTc5 | 720 | Update beautifulsoup4 requirement from ~=4.8.1 to >=4.8.1,<4.10.0 | dependabot-preview[bot] 27856297 | closed | 0 | 0 | 2020-04-08T01:24:38Z | 2020-05-04T17:14:51Z | 2020-05-04T17:14:46Z | CONTRIBUTOR | simonw/datasette/pulls/720 | Updates the requirements on [beautifulsoup4](http://www.crummy.com/software/BeautifulSoup/bs4/) to permit the latest version. Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- **Note:** This repo was added to Dependabot recently, so you'll receive a maximum of 5 PRs for your first few update runs. Once an update run creates fewer than 5 PRs we'll remove that limit. You can always request more updates by clicking `Bump now` in your [Dependabot dashboard](https://app.dependabot.com). <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot u… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/720/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
596245923 | MDExOlB1bGxSZXF1ZXN0NDAwNTc5MDc3 | 721 | Update pytest requirement from ~=5.2.2 to >=5.2.2,<5.5.0 | dependabot-preview[bot] 27856297 | closed | 0 | 0 | 2020-04-08T01:25:04Z | 2020-05-04T17:13:49Z | 2020-05-04T17:13:41Z | CONTRIBUTOR | simonw/datasette/pulls/721 | Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/releases">pytest's releases</a>.</em></p> <blockquote> <h2>5.4.1</h2> <h1>pytest 5.4.1 (2020-03-13)</h1> <h2>Bug Fixes</h2> <ul> <li> <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6909">#6909</a>: Revert the change introduced by <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/pull/6330">#6330</a>, which required all arguments to <code>@pytest.mark.parametrize</code> to be explicitly defined in the function signature.</p> <p>The intention of the original change was to remove what was expected to be an unintended/surprising behavior, but it turns out many people relied on it, so the restriction has been reverted.</p> </li> <li> <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6910">#6910</a>: Fix crash when plugins return an unknown stats while using the <code>--reportlog</code> option.</p> </li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst">pytest's changelog</a>.</em></p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pytest-dev/pytest/commit/3d0f3baa2bb89257dfff25ae6ebabd565287240e"><code>3d0f3ba</code></a> Preparing release version 5.4.1</li> <li><a href="https://github.com/pytest-dev/pytest/commit/b9e2cd0a81e043bdb409e7078db9af4c4d38da54"><code>b9e2cd0</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6914">#6914</a> from nicoddemus/revert-6330</li> <li><a href="https://github.com/pytest-dev/pytest/commit/a84fcbf5b2fc007a1630a1ddd60b0a2438b61642"><code>a84fcbf</code></a> Revert "[parametrize] enforce explicit argnames declaration (<a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6330">#6330</a>… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/721/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
596246006 | MDExOlB1bGxSZXF1ZXN0NDAwNTc5MTM2 | 722 | Update jinja2 requirement from ~=2.10.3 to >=2.10.3,<2.12.0 | dependabot-preview[bot] 27856297 | closed | 0 | 0 | 2020-04-08T01:25:24Z | 2020-05-04T17:13:26Z | 2020-05-04T17:13:16Z | CONTRIBUTOR | simonw/datasette/pulls/722 | Updates the requirements on [jinja2](https://github.com/pallets/jinja) to permit the latest version. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/jinja/releases">jinja2's releases</a>.</em></p> <blockquote> <h2>2.11.1</h2> <p>This fixes an issue in async environment when indexing the result of an attribute lookup, like <code>{{ data.items[1:] }}</code>.</p> <ul> <li>Changes: <a href="https://jinja.palletsprojects.com/en/2.11.x/changelog/#version-2-11-1">https://jinja.palletsprojects.com/en/2.11.x/changelog/#version-2-11-1</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/jinja/blob/master/CHANGES.rst">jinja2's changelog</a>.</em></p> <blockquote> <h1>Version 2.11.1</h1> <p>Released 2020-01-30</p> <ul> <li>Fix a bug that prevented looking up a key after an attribute (<code>{{ data.items[1:] }}</code>) in an async template. 1141</li> </ul> <h1>Version 2.11.0</h1> <p>Released 2020-01-27</p> <ul> <li>Drop support for Python 2.6, 3.3, and 3.4. This will be the last version to support Python 2.7 and 3.5.</li> <li>Added a new <code>ChainableUndefined</code> class to support getitem and getattr on an undefined object. 977</li> <li>Allow <code>{%+</code> syntax (with NOP behavior) when <code>lstrip_blocks</code> is disabled. 748</li> <li>Added a <code>default</code> parameter for the <code>map</code> filter. 557</li> <li>Exclude environment globals from meta.find_undeclared_variables. 931</li> <li>Float literals can be written with scientific notation, like 2.56e-3. 912, 922</li> <li>Int and float literals can be written with the '_' separator for legibility, like 12_345. 923</li> <li>Fix a bug causing deadlocks in <code>LRUCache.setdefault</code>. 1000</li> <li>The <code>trim</code> filter takes an optional string of characters to trim. 828</li> <li>A new <code>jinja2.ext.debug</code> extension adds a <code>{% debug %}</code> tag to quickly dump the current context and available filte… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/722/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
598891570 | MDExOlB1bGxSZXF1ZXN0NDAyNjQ1OTg0 | 725 | Update aiofiles requirement from ~=0.4.0 to >=0.4,<0.6 | dependabot-preview[bot] 27856297 | closed | 0 | 3 | 2020-04-13T13:32:47Z | 2020-05-04T18:16:54Z | 2020-05-04T16:17:49Z | CONTRIBUTOR | simonw/datasette/pulls/725 | Updates the requirements on [aiofiles](https://github.com/Tinche/aiofiles) to permit the latest version. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/Tinche/aiofiles/commit/9a2141e6c51013995ed8b72b121e4e0a27af77b5"><code>9a2141e</code></a> 0.5.0</li> <li><a href="https://github.com/Tinche/aiofiles/commit/479b7eea2c54a9daadb26d1073770e5a50f9ca27"><code>479b7ee</code></a> Update README</li> <li><a href="https://github.com/Tinche/aiofiles/commit/6c247a2cfd1e12f8359aeff04bfe3510f5ef0dc2"><code>6c247a2</code></a> Modernize tests</li> <li><a href="https://github.com/Tinche/aiofiles/commit/eec75d3f425d53306b6aaa8a6005344e56350312"><code>eec75d3</code></a> Switch to async def wherever possible</li> <li><a href="https://github.com/Tinche/aiofiles/commit/786c3e9293a37f63ae1c72b115498d038b5828a0"><code>786c3e9</code></a> Prepare for 3.8</li> <li><a href="https://github.com/Tinche/aiofiles/commit/145107595fe72b877005fb4ebf9d77ffb754689a"><code>1451075</code></a> Update README.rst</li> <li><a href="https://github.com/Tinche/aiofiles/commit/5db1e380e307d0f96177d18db778f82ea0ef87ce"><code>5db1e38</code></a> Add several async os functions</li> <li><a href="https://github.com/Tinche/aiofiles/commit/a60f19bb7b6bd5b063f853d8cae2c41a6878841f"><code>a60f19b</code></a> Add async remove function</li> <li><a href="https://github.com/Tinche/aiofiles/commit/9cf2ac8f9d43446ecdf71c95d05ad6a95bec7377"><code>9cf2ac8</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/Tinche/aiofiles/issues/53">#53</a> from graingert/patch-1</li> <li><a href="https://github.com/Tinche/aiofiles/commit/b88912c6a7a13597b8c5ac93093d7eef86ab1ebb"><code>b88912c</code></a> <strong>all</strong> should be a List[str]</li> <li>Additional commits viewable in <a href="https://github.com/Tinche/aiofiles/compare/v0.4.0...v0.5.0">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/725/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
600120439 | MDU6SXNzdWU2MDAxMjA0Mzk= | 726 | Foreign key : case of a link to the associated row not displayed | JBPressac 6371750 | closed | 0 | 1 | 2020-04-15T08:31:27Z | 2020-04-27T22:05:47Z | 2020-04-27T22:05:46Z | CONTRIBUTOR | Hello, I use Datasette to publish tsv files linked together by foreign keys declared thanks to sqlite-utils. In one table, [prelib_personne](http://crbc-dataset.huma-num.fr/prelib/prelib_personne), the foreign keys are properly noticed by a link to the associated row (for instance ville_naissance_id is properly linked to prelib_ville). But every link to the foreign key prelib_oeuvre.id fails. For instance, [prelib_ecritoeuvre](http://crbc-dataset.huma-num.fr/prelib/prelib_ecritoeuvre) has links to prelib_personne but none to prelib_oeuvre. In despite of the schema: CREATE TABLE "prelib_ecritoeuvre" ( "id" INTEGER, "fonction_id" INTEGER, "oeuvre_id" INTEGER, "personne_id" INTEGER ,PRIMARY KEY ([id]), FOREIGN KEY(fonction_id) REFERENCES prelib_fonctionecritoeuvre(id), FOREIGN KEY(personne_id) REFERENCES prelib_personne(id), FOREIGN KEY(oeuvre_id) REFERENCES prelib_oeuvre(id) ); Would you have any clue to investigate the reason of this problem? Thanks, | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/726/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
603242257 | MDExOlB1bGxSZXF1ZXN0NDA2MDY3MDE5 | 728 | Update mergedeep requirement from ~=1.1.1 to >=1.1.1,<1.4.0 | dependabot-preview[bot] 27856297 | closed | 0 | 0 | 2020-04-20T13:33:23Z | 2020-05-04T16:45:58Z | 2020-05-04T16:45:49Z | CONTRIBUTOR | simonw/datasette/pulls/728 | Updates the requirements on [mergedeep](https://github.com/clarketm/mergedeep) to permit the latest version. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/clarketm/mergedeep/commit/3d6e7b4c9c25b64eba0834da6f54c8dd309767ea"><code>3d6e7b4</code></a> v1.3.0 - support additive merging of <code>Counter</code> types</li> <li><a href="https://github.com/clarketm/mergedeep/commit/56a258a0b58366aa205322476db1b3f84aac65d9"><code>56a258a</code></a> v1.2.1 - tidy docs and variable names</li> <li><a href="https://github.com/clarketm/mergedeep/commit/61ab21382ba20d7973a848c942eb36bd33bc7cd0"><code>61ab213</code></a> v1.2.0 - support both TYPESAFE_REPLACE and TYPESAFE_ADDITIVE merge strategies...</li> <li><a href="https://github.com/clarketm/mergedeep/commit/b331bb5f1580d8290dcf5c6e7690fd3b9708b5cd"><code>b331bb5</code></a> cleanup Makefile</li> <li><a href="https://github.com/clarketm/mergedeep/commit/6f577bf29750c073f6764ffbea29ab6d6759a8d3"><code>6f577bf</code></a> officially label support for python3.8</li> <li><a href="https://github.com/clarketm/mergedeep/commit/84faf37d2da81d4356a4ade4c015fc3964262585"><code>84faf37</code></a> use pipenv for managing dev dependencies</li> <li><a href="https://github.com/clarketm/mergedeep/commit/3a8761a736c8cb7a324bd9e6f4d50a58898a3ba9"><code>3a8761a</code></a> Update README.md</li> <li>See full diff in <a href="https://github.com/clarketm/mergedeep/compare/v1.1.1...v1.3.0">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge th… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/728/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
604001627 | MDExOlB1bGxSZXF1ZXN0NDA2Njc3MjA1 | 730 | Update pytest-asyncio requirement from ~=0.10.0 to >=0.10,<0.12 | dependabot-preview[bot] 27856297 | closed | 0 | 1 | 2020-04-21T13:32:35Z | 2020-05-04T13:27:24Z | 2020-05-04T13:27:23Z | CONTRIBUTOR | simonw/datasette/pulls/730 | Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/1026c39495a963ff3e5fee7da2ae9f3a5d21fb83"><code>1026c39</code></a> 0.11.0</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/ab2b14048a691479fa9f8811aaa558018c6db6e3"><code>ab2b140</code></a> Test on Python 3.8, drop 3.3 and 3.4</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/6397a2255e3e9ef858439b164018438a8106f454"><code>6397a22</code></a> plugin: Use pytest 5.4.0 new Function API</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/21a0f9476be84ca0c84af60057f0f24c5fb2fd71"><code>21a0f94</code></a> Replace yield_fixture() by fixture()</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/964b295ba280a6e217159706279b67f8f4cbb5f4"><code>964b295</code></a> Added min hypothesis version so that bugfix for <a href="https://github.com/Hypothesis">https://github.com/Hypothesis</a>...</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/4a11a206fbcf88ee18cbed2d01041e61c20a9a48"><code>4a11a20</code></a> Add max supported pytest version to < 5.4.0 to prevent fails until <a href="https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/141">#141</a> is fi...</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/b3055940db49cc17e36b66631e3d863e15fe34e4"><code>b305594</code></a> Change event_loop to module scope in hypothesis tests, fixing <a href="https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/145">#145</a>.</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/d5a0f4789e7fecb58d509409e2c537b206c4fde2"><code>d5a0f47</code></a> Enable test_subprocess to be run on win, by changing to ProactorEventLoop in ...</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/d07cd2d447cf313446c4e00e25a35cb2adcb2c63"><code>d07cd2d</code><… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/730/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
605546606 | MDExOlB1bGxSZXF1ZXN0NDA3OTI5MTI4 | 734 | Update janus requirement from ~=0.4.0 to >=0.4,<0.6 | dependabot-preview[bot] 27856297 | closed | 0 | 0 | 2020-04-23T13:43:45Z | 2020-05-04T16:48:14Z | 2020-05-04T16:48:04Z | CONTRIBUTOR | simonw/datasette/pulls/734 | Updates the requirements on [janus](https://github.com/aio-libs/janus) to permit the latest version. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/aio-libs/janus/blob/master/CHANGES.rst">janus's changelog</a>.</em></p> <blockquote> <h1>0.5.0 (2020-04-23)</h1> <ul> <li>Remove explicit loop arguments and forbid creating queues outside event loops <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/246">#246</a></li> </ul> <h1>0.4.0 (2018-07-28)</h1> <ul> <li>Add <code>py.typed</code> macro <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/89">#89</a></li> <li>Drop python 3.4 support and fix minimal version python3.5.3 <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/88">#88</a></li> <li>Add property with that indicates if queue is closed <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/86">#86</a></li> </ul> <h1>0.3.2 (2018-07-06)</h1> <ul> <li>Fixed python 3.7 support <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/97">#97</a></li> </ul> <h1>0.3.1 (2018-01-30)</h1> <ul> <li>Fixed bug with join() in case tasks are added by sync_q.put() <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/75">#75</a></li> </ul> <h1>0.3.0 (2017-02-21)</h1> <ul> <li>Expose unfinished_tasks property <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/34">#34</a></li> </ul> <h1>0.2.4 (2016-12-05)</h1> <ul> <li>Restore tarball deploying</li> </ul> <h1>0.2.3 (2016-07-12)</h1> <ul> <li>Fix exception type</li> </ul> <h1>0.2.2 (2016-07-11)</h1> <ul> <li>Update asyncio.async() to use asyncio.ensure_future() <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/6">#6</a></li> </ul> <h1>0.2.1 (2016-03-24)</h1> <ul> <li>Fix python setup.py test command <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/4">#4</a></li> </ul> <h1>0.2.0 (2015-09-20)</h1> </tr></table> ... (truncated) </blockquote> </details> <details> <summary>Commits</summ… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/734/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
610517472 | MDU6SXNzdWU2MTA1MTc0NzI= | 103 | sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns | b0b5h4rp13 32605365 | closed | 0 | 8 | 2020-05-01T02:26:14Z | 2020-05-14T00:18:57Z | 2020-05-14T00:18:57Z | CONTRIBUTOR | If using insert_all to put in 1000 rows of data with varying number of columns, it comes up with this message `sqlite3.OperationalError: too many SQL variables` if the number of columns is larger in later records (past the first row) I've reduced `SQLITE_MAX_VARS` by 100 to 899 at the top of `db.py` to add wiggle room, so that if the column count increases it wont go past SQLite's batch limit as calculated by this line of code based on the count of the first row's dict keys batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns)) | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/103/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
611874514 | MDExOlB1bGxSZXF1ZXN0NDEyOTUxMTkx | 753 | Update pytest-asyncio requirement from ~=0.10.0 to >=0.10,<0.13 | dependabot-preview[bot] 27856297 | closed | 0 | 0 | 2020-05-04T13:27:19Z | 2020-05-04T17:41:01Z | 2020-05-04T17:40:49Z | CONTRIBUTOR | simonw/datasette/pulls/753 | Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/b8e2a45e152a196cef7fdb6ddcf3e2d67a0f01ca"><code>b8e2a45</code></a> 0.12.0</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/06580c6413e905c5fddcc57c2895e9d3d2913847"><code>06580c6</code></a> Update changelog</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/b45de23c0de0acc843c3f871985ba16fdfb1d5fd"><code>b45de23</code></a> Fixed failing test case, 'test_asyncio_marker_without_loop'.</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/238ccedde8a196c16007a8a2b1ea1eeeb8db86a3"><code>238cced</code></a> Put event_loop first among the fixtures of asyncio tests, fixes <a href="https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/154">#154</a>.</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/e5e3dc7a90685a56747a2d4c3c770d74d573c497"><code>e5e3dc7</code></a> Added unittests for issue <a href="https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/154">#154</a>.</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/a7e5795335b823b4d58bb6b38b6e653f0e0d35b0"><code>a7e5795</code></a> 0.12.0 open for business!</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/1026c39495a963ff3e5fee7da2ae9f3a5d21fb83"><code>1026c39</code></a> 0.11.0</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/ab2b14048a691479fa9f8811aaa558018c6db6e3"><code>ab2b140</code></a> Test on Python 3.8, drop 3.3 and 3.4</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/6397a2255e3e9ef858439b164018438a8106f454"><code>6397a22</code></a> plugin: Use pytest 5.4.0 new Function API</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/21a0f9476be84ca0c84af60057f0f24c5fb2fd71"><code>21a0f94</code></a> Replace yield_fixture() by fixture()</li>… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/753/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
620969465 | MDU6SXNzdWU2MjA5Njk0NjU= | 767 | Allow to specify a URL fragment for canned queries | rixx 2657547 | closed | 0 | Datasette 0.43 5471110 | 2 | 2020-05-19T13:17:42Z | 2020-05-27T21:52:25Z | 2020-05-27T21:52:25Z | CONTRIBUTOR | Canned queries are very useful to direct users to prepared data and views. I like to use them with charts using datasette-vega a lot, because people get a direct impression at first glance. datasette-vega doesn't show up by default though, and users have to click through to it. Also, datasette-vega does not always guess the best way to render columns correctly though, so it would be nice if I could specify a URL fragment in my canned queries to make sure people see what I want them to see. My current workaround is to include a fragement link in ``description_html`` and ask people to reload the page, like [here](https://data.rixx.de/songs/show_by_bpm#g.mark=bar&g.x_column=bpm_floor&g.x_type=ordinal&g.y_column=bpm_count&g.y_type=quantitative), which is a bit hacky. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/767/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | |||||
638270441 | MDExOlB1bGxSZXF1ZXN0NDM0MDg1MjM1 | 848 | Reload support for config_dir mode. | amjith 49260 | closed | 0 | 1 | 2020-06-14T02:34:46Z | 2020-07-03T02:44:54Z | 2020-07-03T02:44:53Z | CONTRIBUTOR | simonw/datasette/pulls/848 | A reference implementation for adding support to reload when datasette is in the config_dir mode. This implementation is flawed since it is watching the entire directory and any changes to the database will reload the server and adding unrelated files to the directory will also reload the server. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/848/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
638375985 | MDExOlB1bGxSZXF1ZXN0NDM0MTYyMzE2 | 29 | Fixed bug in SQL query for photo scores | RhetTbull 41546558 | closed | 0 | 1 | 2020-06-14T15:39:22Z | 2020-12-04T22:32:36Z | 2020-12-04T22:32:27Z | CONTRIBUTOR | dogsheep/dogsheep-photos/pulls/29 | The join on ZCOMPUTEDASSETATTRIBUTES used the wrong columns. In most of the Photos database tables, table.ZASSET joins with ZGENERICASSET.Z_PK | dogsheep-photos 256834907 | pull | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/29/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
640330278 | MDU6SXNzdWU2NDAzMzAyNzg= | 851 | Having trouble getting writable canned queries to work | abdusco 3243482 | closed | 0 | 1 | 2020-06-17T10:30:28Z | 2020-06-17T10:33:25Z | 2020-06-17T10:32:33Z | CONTRIBUTOR | Hey, I'm trying to get canned inserts to work. I have an DB with following metadata: ```text sqlite> .mode line sqlite> select name, sql from sqlite_master where name like '%search%'; name = search sql = CREATE TABLE "search" ("id" INTEGER NOT NULL PRIMARY KEY, "name" VARCHAR(255) NOT NULL, "url" VARCHAR(255) NOT NULL) ``` ```yaml # ... queries: add_search: sql: insert into search(name, url) VALUES (:name, :url), write: true ``` which renders a form as expected, but when I submit the form I get `incomplete input` error. ![image](https://user-images.githubusercontent.com/3243482/84885285-7f46fe80-b09b-11ea-8a05-92c8986bbf7a.png) but when submit post the form I've attached a debugger to see where the error comes from, because `incomplete input` string doesn't appear in datasette codebase. Inside `datasette.database.Database.execute_write_fn` https://github.com/simonw/datasette/blob/4fa7cf68536628344356d3ef8c92c25c249067a0/datasette/database.py#L69 ```py result = await reply_queue.async_q.get() ``` this line raises an exception. That led me to believe I had something wrong with my SQL. But running the command in `sqlite3` inserts the record just fine. ```text sqlite> insert into search (name, url) values ('my name', 'my url'); sqlite> SELECT last_insert_rowid(); last_insert_rowid() = 3 ``` So I'm a bit lost here. --- - datasette, version 0.44 - Python 3.8.1 | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/851/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
642572841 | MDU6SXNzdWU2NDI1NzI4NDE= | 859 | Database page loads too slowly with many large tables (due to table counts) | abdusco 3243482 | open | 0 | 21 | 2020-06-21T14:23:17Z | 2021-08-25T21:59:55Z | CONTRIBUTOR | Hey, I have a database that I save in HTML from couple of web scrapers. There are around 200k+, 50+ rows in a couple of tables, with sqlite file weighing around 600MB. The app runs on a VPS with 2 core CPU, 4GB RAM and refreshing database page regularly takes more than 10 seconds. I was suspecting that counting tables was the culprit, but manually running `select count(*) from table_name` for the largest table finishes under a second. I've looked at the source code. There's a check for index page for mutable databases larger than 100MB https://github.com/simonw/datasette/blob/799c5d53570d773203527f19530cf772dc2eeb24/datasette/views/index.py#L15 but this check is not performed for database page. I've manually crippled `Database::table_counts` method ```py async def table_counts(self, limit=10): if not self.is_mutable and self.cached_table_counts is not None: return self.cached_table_counts # Try to get counts for each table, $limit timeout for each count counts = {} for table in await self.table_names(): try: # table_count = ( # await self.execute( # "select count(*) from [{}]".format(table), # custom_time_limit=limit, # ) # ).rows[0][0] counts[table] = 10 # table_count # In some cases I saw "SQL Logic Error" here in addition to # QueryInterrupted - so we catch that too: except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError): counts[table] = None if not self.is_mutable: self.cached_table_counts = counts return counts ``` now the page loads in <100ms. Is it possible to apply size check on database page too? <details> <summary> /-/versions output </summary> <pre> { "python": { "version": "3.8.0", "full": "3.8.0 (default, Oct 28 2019, 16:14:01) \n[GCC 8.3.0]" }, "datasette": { "version": "0.44" }, "asgi": "3.… | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/859/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
644610729 | MDExOlB1bGxSZXF1ZXN0NDM5MjAzODA4 | 866 | Update pytest-asyncio requirement from <0.13,>=0.10 to >=0.10,<0.15 | dependabot-preview[bot] 27856297 | closed | 0 | 1 | 2020-06-24T13:21:47Z | 2020-06-24T18:50:57Z | 2020-06-24T18:50:56Z | CONTRIBUTOR | simonw/datasette/pulls/866 | Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/53f3da7aefc719e62cbaa89e57ab68a7e12cc3c3"><code>53f3da7</code></a> Prepare for release</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/e99569de645c37fc048964ab4c5073529080fd86"><code>e99569d</code></a> A line is added to the changelog.</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/4099b6351793611acabd8b26d93aabd44ce200c5"><code>4099b63</code></a> One import is not needed</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/68513b33616ff2ef93972acf1b992e7b1a18c4d1"><code>68513b3</code></a> Clarify names and comments, according to yanlend comments 26 May</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/907e8f24b9444f50cb30f17265665706882bc01f"><code>907e8f2</code></a> FIX new test_cases on python 3.5 & 3.6</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/51d986cec83fdbc14fa08015424c79397afc7ad9"><code>51d986c</code></a> To solve test cases that fail:</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/f97e900f1fcb51a572a1b861c95ac49e69bbfdf9"><code>f97e900</code></a> 1) Test case (test_async_fixtures_with_finalizer) refactoring to pass on pyth...</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/c1131f8b5313189508dc81d7ef1937ccb136658b"><code>c1131f8</code></a> 1) A new test case that fails with 0.12.0, and pass with this commit.</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/7a255bc4cf82aba7aa4b213c1a97c81d532c1e85"><code>7a255bc</code></a> 0.13.0 open for business</li> <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/b8e2a45e152a196cef7fdb6ddcf3e2d67a0f01ca"><code>b8e2a45</code></a> 0.12.0</li> <li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest-asyncio/compare/v0.10.0...v0.14.0… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/866/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
648749062 | MDExOlB1bGxSZXF1ZXN0NDQyNTA1MDg4 | 883 | Skip counting hidden tables | abdusco 3243482 | open | 0 | 4 | 2020-07-01T07:38:08Z | 2020-07-02T00:25:44Z | CONTRIBUTOR | simonw/datasette/pulls/883 | Potential fix for https://github.com/simonw/datasette/issues/859. Disabling table counts for hidden tables speeds up database page quite a bit. In my setup it reduced load time by 2/3 (~300 -> ~90ms) | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/883/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||||
649702801 | MDU6SXNzdWU2NDk3MDI4MDE= | 888 | URLs in release notes point to 127.0.0.1 | abdusco 3243482 | closed | 0 | 1 | 2020-07-02T07:28:04Z | 2020-09-15T20:39:50Z | 2020-09-15T20:39:49Z | CONTRIBUTOR | Just a quick heads up: Release notes for 0.45 include urls that point to localhost. https://github.com/simonw/datasette/releases/tag/0.45 | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/888/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
649907676 | MDU6SXNzdWU2NDk5MDc2NzY= | 889 | asgi_wrapper plugin hook is crashing at startup | amjith 49260 | closed | 0 | 3 | 2020-07-02T12:53:13Z | 2020-09-15T20:40:52Z | 2020-09-15T20:40:52Z | CONTRIBUTOR | Steps to reproduce: 1. Install datasette-media plugin `pip install datasette-media` 2. Launch datasette `datasette databasename.db` 3. Error ``` INFO: Started server process [927704] INFO: Waiting for application startup. ERROR: Exception in 'lifespan' protocol Traceback (most recent call last): File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/uvicorn/lifespan/on.py", line 48, in main await app(scope, self.receive, self.send) File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/datasette_media/__init__.py", line 9, in wrapped_app path = scope["path"] KeyError: 'path' ERROR: Application startup failed. Exiting. ``` | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/889/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
650305298 | MDExOlB1bGxSZXF1ZXN0NDQzODIzMDQw | 890 | Load only python files from plugins-dir. | amjith 49260 | closed | 0 | 2 | 2020-07-03T02:47:32Z | 2020-07-03T03:08:33Z | 2020-07-03T03:08:33Z | CONTRIBUTOR | simonw/datasette/pulls/890 | The current behavior for `--plugins-dir` is to load every file in that folder as a python module. This can result in errors if there are non-python files in the plugins dir (such as .mypy_cache). This PR restricts the module loading to only python files. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/890/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
651844316 | MDExOlB1bGxSZXF1ZXN0NDQ1MDIzMzI2 | 118 | Add insert --truncate option | tsibley 79913 | closed | 0 | 9 | 2020-07-06T21:58:40Z | 2020-07-08T17:26:21Z | 2020-07-08T17:26:21Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/118 | Deletes all rows in the table (if it exists) before inserting new rows. SQLite doesn't implement a TRUNCATE TABLE statement but does optimize an unqualified DELETE FROM. This can be handy if you want to refresh the entire contents of a table but a) don't have a PK (so can't use --replace), b) don't want the table to disappear (even briefly) for other connections, and c) have to handle records that used to exist being deleted. Ideally the replacement of rows would appear instantaneous to other connections by putting the DELETE + INSERT in a transaction, but this is very difficult without breaking other code as the current transaction handling is inconsistent and non-systematic. There exists the possibility for the DELETE to succeed but the INSERT to fail, leaving an empty table. This is not much worse, however, than the current possibility of one chunked INSERT succeeding and being committed while the next chunked INSERT fails, leaving a partially complete operation. | sqlite-utils 140912432 | pull | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
652816158 | MDExOlB1bGxSZXF1ZXN0NDQ1ODMzOTA4 | 120 | Fix query command's support for DML | tsibley 79913 | closed | 0 | 1 | 2020-07-08T01:36:34Z | 2020-07-08T05:14:04Z | 2020-07-08T05:14:04Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/120 | See commit messages for details. I ran into this while investigating another feature/issue. | sqlite-utils 140912432 | pull | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/120/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
662322234 | MDExOlB1bGxSZXF1ZXN0NDUzODkwMjky | 901 | Use None as a default arg | Alyetama 56323389 | closed | 0 | 1 | 2020-07-20T22:18:38Z | 2020-07-31T18:42:39Z | 2020-07-31T18:42:39Z | CONTRIBUTOR | simonw/datasette/pulls/901 | When passing a mutable value as a default argument in a function, the default argument is mutated anytime that value is mutated. This poses a bug risk. Instead, use None as a default and assign the mutable value inside the function. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/901/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
662439034 | MDExOlB1bGxSZXF1ZXN0NDUzOTk1MTc5 | 902 | Don't install tests package | abeyerpath 32467826 | closed | 0 | 2 | 2020-07-21T01:08:50Z | 2020-07-24T20:39:54Z | 2020-07-24T20:39:54Z | CONTRIBUTOR | simonw/datasette/pulls/902 | The `exclude` argument to `find_packages` needs an iterable of package names. Fixes: #456 | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/902/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
667840539 | MDExOlB1bGxSZXF1ZXN0NDU4NDM1NTky | 910 | Update pytest requirement from <5.5.0,>=5.2.2 to >=5.2.2,<6.1.0 | dependabot-preview[bot] 27856297 | closed | 0 | 1 | 2020-07-29T13:21:17Z | 2020-07-29T21:26:05Z | 2020-07-29T21:26:04Z | CONTRIBUTOR | simonw/datasette/pulls/910 | Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/releases">pytest's releases</a>.</em></p> <blockquote> <h2>6.0.0</h2> <h1>pytest 6.0.0 (2020-07-28)</h1> <p>(<strong>Please see the full set of changes for this release also in the 6.0.0rc1 notes below</strong>)</p> <h2>Breaking Changes</h2> <ul> <li> <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/5584">#5584</a>: <strong>PytestDeprecationWarning are now errors by default.</strong></p> <p>Following our plan to remove deprecated features with as little disruption as possible, all warnings of type <code>PytestDeprecationWarning</code> now generate errors instead of warning messages.</p> <p><strong>The affected features will be effectively removed in pytest 6.1</strong>, so please consult the <a href="https://docs.pytest.org/en/latest/deprecations.html">Deprecations and Removals</a> section in the docs for directions on how to update existing code.</p> <p>In the pytest <code>6.0.X</code> series, it is possible to change the errors back into warnings as a stopgap measure by adding this to your <code>pytest.ini</code> file:</p> <pre lang="{.sourceCode" data-meta=".ini}"><code>[pytest] filterwarnings = ignore::pytest.PytestDeprecationWarning </code></pre> <p>But this will stop working when pytest <code>6.1</code> is released.</p> <p><strong>If you have concerns</strong> about the removal of a specific feature, please add a comment to <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/5584">#5584</a>.</p> </li> <li> <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7472">#7472</a>: The <code>exec_()</code> and <code>is_true()</code> methods of <code>_pytest._code.Frame</code> have been removed.</p> </li> </ul> <h2>Features</h2> <ul> <li><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7464">#7464</a>: Added… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/910/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
681228542 | MDExOlB1bGxSZXF1ZXN0NDY5NjUxNzMy | 48 | Add pull requests | adamjonas 755825 | closed | 0 | 2 | 2020-08-18T17:58:44Z | 2020-11-29T23:51:09Z | 2020-11-29T23:51:09Z | CONTRIBUTOR | dogsheep/github-to-sqlite/pulls/48 | ref #46 Issues don't have merge information on them, which means that PRs need to be pulled separately. Did my best to mimic the API of issues. | github-to-sqlite 207052882 | pull | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
681575714 | MDExOlB1bGxSZXF1ZXN0NDY5OTQ0OTk5 | 49 | Document the use of --stop_after with favorites, refs #20 | mikepqr 370930 | closed | 0 | 1 | 2020-08-19T06:10:52Z | 2021-08-20T00:02:11Z | 2021-08-20T00:02:11Z | CONTRIBUTOR | dogsheep/twitter-to-sqlite/pulls/49 | (I discovered this trawling the issues for how to use --since with favorites) | twitter-to-sqlite 206156866 | pull | {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
686978131 | MDU6SXNzdWU2ODY5NzgxMzE= | 139 | insert_all(..., alter=True) should work for new columns introduced after the first 100 records | simonwiles 96218 | closed | 0 | 7 | 2020-08-27T06:25:25Z | 2020-08-28T22:48:51Z | 2020-08-28T22:30:14Z | CONTRIBUTOR | Is there a way to make `.insert_all()` work properly when new columns are introduced outside the first 100 records (with or without the `alter=True` argument)? I'm using `.insert_all()` to bulk insert ~3-4k records at a time and it is common for records to need to introduce new columns. However, if new columns are introduced after the first 100 records, `sqlite_utils` doesn't even raise the `OperationalError: table ... has no column named ...` exception; it just silently drops the extra data and moves on. It took me a while to find this little snippet in the [documentation for `.insert_all()`](https://sqlite-utils.readthedocs.io/en/stable/python-api.html#bulk-inserts) (it's not mentioned under [Adding columns automatically on insert/update](https://sqlite-utils.readthedocs.io/en/stable/python-api.html#bulk-inserts)): > The column types used in the CREATE TABLE statement are automatically derived from the types of data in that first batch of rows. **_Any additional or missing columns in subsequent batches will be ignored._** I tried changing the `batch_size` argument to the total number of records, but it seems only to effect the number of rows that are committed at a time, and has no influence on this problem. Is there a way around this that you would suggest? It seems like it should raise an exception at least. | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
687245650 | MDExOlB1bGxSZXF1ZXN0NDc0NzAzMDA3 | 952 | Update black requirement from ~=19.10b0 to >=19.10,<21.0 | dependabot-preview[bot] 27856297 | closed | 0 | 1 | 2020-08-27T13:31:36Z | 2020-09-02T22:26:17Z | 2020-09-02T22:26:16Z | CONTRIBUTOR | simonw/datasette/pulls/952 | Updates the requirements on [black](https://github.com/psf/black) to permit the latest version. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/psf/black/blob/master/CHANGES.md">black's changelog</a>.</em></p> <blockquote> <h3>20.8b1</h3> <h4><em>Packaging</em></h4> <ul> <li>explicitly depend on Click 7.1.2 or newer as <code>Black</code> no longer works with versions older than 7.0</li> </ul> <h3>20.8b0</h3> <h4><em>Black</em></h4> <ul> <li> <p>re-implemented support for explicit trailing commas: now it works consistently within any bracket pair, including nested structures (<a href="https://github-redirect.dependabot.com/psf/black/issues/1288">#1288</a> and duplicates)</p> </li> <li> <p><code>Black</code> now reindents docstrings when reindenting code around it (<a href="https://github-redirect.dependabot.com/psf/black/issues/1053">#1053</a>)</p> </li> <li> <p><code>Black</code> now shows colored diffs (<a href="https://github-redirect.dependabot.com/psf/black/issues/1266">#1266</a>)</p> </li> <li> <p><code>Black</code> is now packaged using 'py3' tagged wheels (<a href="https://github-redirect.dependabot.com/psf/black/issues/1388">#1388</a>)</p> </li> <li> <p><code>Black</code> now supports Python 3.8 code, e.g. star expressions in return statements (<a href="https://github-redirect.dependabot.com/psf/black/issues/1121">#1121</a>)</p> </li> <li> <p><code>Black</code> no longer normalizes capital R-string prefixes as those have a community-accepted meaning (<a href="https://github-redirect.dependabot.com/psf/black/issues/1244">#1244</a>)</p> </li> <li> <p><code>Black</code> now uses exit code 2 when specified configuration file doesn't exit (<a href="https://github-redirect.dependabot.com/psf/black/issues/1361">#1361</a>)</p> </li> <li> <p><code>Black</code> now works on AWS Lambda (<a href="https://github-redirect.dependabot.com/psf/black/issues/1141">#1141</a>)</p> </li> <li> <p>added <code>--force-exclude</code> argument (<a href="https://github-redirect.dependabot.com/p… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/952/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
688386219 | MDExOlB1bGxSZXF1ZXN0NDc1NjY1OTg0 | 142 | insert_all(..., alter=True) should work for new columns introduced after the first 100 records | simonwiles 96218 | closed | 0 | 3 | 2020-08-28T22:22:57Z | 2020-08-30T07:28:23Z | 2020-08-28T22:30:14Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/142 | Closes #139. | sqlite-utils 140912432 | pull | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/142/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
688659182 | MDU6SXNzdWU2ODg2NTkxODI= | 145 | Bug when first record contains fewer columns than subsequent records | simonwiles 96218 | closed | 0 | 2 | 2020-08-30T05:44:44Z | 2020-09-08T23:21:23Z | 2020-09-08T23:21:23Z | CONTRIBUTOR | `insert_all()` selects the maximum batch size based on the number of fields in the first record. If the first record has fewer fields than subsequent records (and `alter=True` is passed), this can result in SQL statements with more than the maximum permitted number of host parameters. This situation is perhaps unlikely to occur, but could happen if the first record had, say, 10 columns, such that `batch_size` (based on `SQLITE_MAX_VARIABLE_NUMBER = 999`) would be 99. If the next 98 rows had 11 columns, the resulting SQL statement for the first batch would have `10 * 1 + 11 * 98 = 1088` host parameters (and subsequent batches, if the data were consistent from thereon out, would have `99 * 11 = 1089`). I suspect that this bug is masked somewhat by the fact that while: > [`SQLITE_MAX_VARIABLE_NUMBER`](https://www.sqlite.org/limits.html#max_variable_number) ... defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0. it is common that it is increased at compile time. Debian-based systems, for example, seem to ship with a version of sqlite compiled with `SQLITE_MAX_VARIABLE_NUMBER` set to 250,000, and I believe this is the case for homebrew installations too. A test for this issue might look like this: ```python def test_columns_not_in_first_record_should_not_cause_batch_to_be_too_large(fresh_db): # sqlite on homebrew and Debian/Ubuntu etc. is typically compiled with # SQLITE_MAX_VARIABLE_NUMBER set to 250,000, so we need to exceed this value to # trigger the error on these systems. THRESHOLD = 250000 extra_columns = 1 + (THRESHOLD - 1) // 99 records = [ {"c0": "first record"}, # one column in first record -> batch_size = 100 # fill out the batch with 99 records with enough columns to exceed THRESHOLD *[ dict([("c{}".format(i), j) for i in range(extra_columns)]) for j in range(99) ] ] try: fresh_db["too_many_columns"].insert_all(records, a… | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/145/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
688668680 | MDExOlB1bGxSZXF1ZXN0NDc1ODc0NDkz | 146 | Handle case where subsequent records (after first batch) include extra columns | simonwiles 96218 | closed | 0 | 5 | 2020-08-30T07:13:58Z | 2020-09-08T23:20:37Z | 2020-09-08T23:20:37Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/146 | Addresses #145. I think this should do the job. If it meets with your approval I'll update this PR to include an update to the documentation -- I came across this bug while preparing a PR to update the documentation around `batch_size` in any event. | sqlite-utils 140912432 | pull | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/146/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
688670158 | MDU6SXNzdWU2ODg2NzAxNTg= | 147 | SQLITE_MAX_VARS maybe hard-coded too low | simonwiles 96218 | open | 0 | 7 | 2020-08-30T07:26:45Z | 2021-02-15T21:27:55Z | CONTRIBUTOR | I came across this while about to open an issue and PR against the documentation for `batch_size`, which is a bit incomplete. As mentioned in #145, while: > [`SQLITE_MAX_VARIABLE_NUMBER`](https://www.sqlite.org/limits.html#max_variable_number) ... defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0. it is common that it is increased at compile time. Debian-based systems, for example, seem to ship with a version of sqlite compiled with SQLITE_MAX_VARIABLE_NUMBER set to 250,000, and I believe this is the case for homebrew installations too. In working to understand what `batch_size` was actually doing and why, I realized that by setting `SQLITE_MAX_VARS` in `db.py` to match the value my sqlite was compiled with (I'm on Debian), I was able to decrease the time to `insert_all()` my test data set (~128k records across 7 tables) from ~26.5s to ~3.5s. Given that this about .05% of my total dataset, this is time I am keen to save... Unfortunately, it seems that `sqlite3` in the python standard library doesn't expose the `get_limit()` C API (even though `pysqlite` used to), so it's hard to know what value sqlite has been compiled with (note that this could mean, I suppose, that it's less than 999, and even hardcoding `SQLITE_MAX_VARS` to the conservative default might not be adequate. It can also be lowered -- but not raised -- at runtime). The best I could come up with is `echo "" | sqlite3 -cmd ".limits variable_number"` (only available in `sqlite >= 2015-05-07 (3.8.10)`). Obviously this couldn't be relied upon in `sqlite_utils`, but I wonder what your opinion would be about exposing `SQLITE_MAX_VARS` as a user-configurable parameter (with suitable "here be dragons" warnings)? I'm going to go ahead and monkey-patch it for my purposes in any event, but it seems like it might be worth considering. | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/147/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
697030843 | MDExOlB1bGxSZXF1ZXN0NDgzMDI3NTg3 | 156 | Typos in tests | simonwiles 96218 | closed | 0 | 1 | 2020-09-09T18:00:58Z | 2020-09-09T18:24:50Z | 2020-09-09T18:21:23Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/156 | One of these is my fault, and the other is one I just happened to come across. They're harmless, but might as well be fixed. | sqlite-utils 140912432 | pull | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/156/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
697203800 | MDExOlB1bGxSZXF1ZXN0NDgzMTc1NTA5 | 158 | Fix accidental mega long line in docs | tomviner 167319 | closed | 0 | 1 | 2020-09-09T22:31:23Z | 2020-09-16T06:21:43Z | 2020-09-16T06:21:43Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/158 | sqlite-utils 140912432 | pull | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/158/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||||
698791218 | MDU6SXNzdWU2OTg3OTEyMTg= | 50 | favorites --stop_after=N stops after min(N, 200) | mikepqr 370930 | open | 0 | 2 | 2020-09-11T03:38:14Z | 2020-09-13T05:11:14Z | CONTRIBUTOR | For any number greater than 200, `favorites --stop_after` stops after getting 200 tweets, e.g. ``` $ twitter-to-sqlite favorites tweets.db --stop_after=300 Importing favorites [####################################] 199 $ ``` I don't _think_ this is a limitation of the API (if you omit `--stop_after` you get some very large number, possibly all of them), so I _think_ this is a bug. | twitter-to-sqlite 206156866 | issue | {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
708185405 | MDU6SXNzdWU3MDgxODU0MDU= | 975 | Dependabot couldn't authenticate with https://pypi.python.org/simple/ | dependabot-preview[bot] 27856297 | closed | 0 | 0 | 2020-09-24T13:44:40Z | 2020-09-25T13:34:34Z | 2020-09-25T13:34:34Z | CONTRIBUTOR | Dependabot couldn't authenticate with https://pypi.python.org/simple/. You can provide authentication details in your [Dependabot dashboard](https://app.dependabot.com/accounts/simonw) by clicking into the account menu (in the top right) and selecting 'Config variables'. [View the update logs](https://app.dependabot.com/accounts/simonw/update-logs/48611311). | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/975/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
709043182 | MDExOlB1bGxSZXF1ZXN0NDkzMTYyNzY3 | 178 | Update README.md | shakeel 19921 | closed | 0 | 1 | 2020-09-25T15:52:11Z | 2020-10-01T14:18:30Z | 2020-09-30T20:29:28Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/178 | The `sqlite-utils insert releases.db releases - --pk` is missing the pk field name, added ` "id"` to fix it. | sqlite-utils 140912432 | pull | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/178/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
710269200 | MDExOlB1bGxSZXF1ZXN0NDk0MTQ2MDQz | 977 | Update pytest requirement from <6.1.0,>=5.2.2 to >=5.2.2,<6.2.0 | dependabot-preview[bot] 27856297 | closed | 0 | 1 | 2020-09-28T13:33:05Z | 2020-09-28T22:16:36Z | 2020-09-28T22:16:35Z | CONTRIBUTOR | simonw/datasette/pulls/977 | Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/releases">pytest's releases</a>.</em></p> <blockquote> <h2>6.1.0</h2> <h1>pytest 6.1.0 (2020-09-26)</h1> <h2>Breaking Changes</h2> <ul> <li> <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/5585">#5585</a>: As per our policy, the following features which have been deprecated in the 5.X series are now removed:</p> <ul> <li>The <code>funcargnames</code> read-only property of <code>FixtureRequest</code>, <code>Metafunc</code>, and <code>Function</code> classes. Use <code>fixturenames</code> attribute.</li> <li><code>@pytest.fixture</code> no longer supports positional arguments, pass all arguments by keyword instead.</li> <li>Direct construction of <code>Node</code> subclasses now raise an error, use <code>from_parent</code> instead.</li> <li>The default value for <code>junit_family</code> has changed to <code>xunit2</code>. If you require the old format, add <code>junit_family=xunit1</code> to your configuration file.</li> <li>The <code>TerminalReporter</code> no longer has a <code>writer</code> attribute. Plugin authors may use the public functions of the <code>TerminalReporter</code> instead of accessing the <code>TerminalWriter</code> object directly.</li> <li>The <code>--result-log</code> option has been removed. Users are recommended to use the <a href="https://github.com/pytest-dev/pytest-reportlog">pytest-reportlog</a> plugin instead.</li> </ul> <p>For more information consult <a href="https://docs.pytest.org/en/stable/deprecations.html">Deprecations and Removals</a> in the docs.</p> </li> </ul> <h2>Deprecations</h2> <ul> <li> <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6981">#6981</a>: The <code>pytest.collect</code> module is deprecated: all its names can be imported from <code>pytest</code> directly.</p> </li> <li> <p><a href="https://gi… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/977/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
715779909 | MDExOlB1bGxSZXF1ZXN0NDk4NjMwNzA4 | 995 | Document setting Google Cloud SDK properties | ghing 110420 | closed | 0 | Datasette 0.50 5971510 | 2 | 2020-10-06T15:18:01Z | 2020-10-08T23:55:30Z | 2020-10-06T16:25:38Z | CONTRIBUTOR | simonw/datasette/pulls/995 | Document setting Google Cloud SDK properties to avoid having to respond to interactive prompts when running `datasette publish cloudrun`. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/995/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||
719381863 | MDExOlB1bGxSZXF1ZXN0NTAxNTc5MDg4 | 1017 | Update janus requirement from <0.6,>=0.4 to >=0.4,<0.7 | dependabot-preview[bot] 27856297 | closed | 0 | 1 | 2020-10-12T13:29:46Z | 2020-10-14T21:52:08Z | 2020-10-14T21:52:07Z | CONTRIBUTOR | simonw/datasette/pulls/1017 | Updates the requirements on [janus](https://github.com/aio-libs/janus) to permit the latest version. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/aio-libs/janus/blob/master/CHANGES.rst">janus's changelog</a>.</em></p> <blockquote> <h1>Changes</h1> <h2>0.5.0 (2020-04-23)</h2> <ul> <li>Remove explicit loop arguments and forbid creating queues outside event loops <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/246">#246</a></li> </ul> <h2>0.4.0 (2018-07-28)</h2> <ul> <li>Add <code>py.typed</code> macro <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/89">#89</a></li> <li>Drop python 3.4 support and fix minimal version python3.5.3 <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/88">#88</a></li> <li>Add property with that indicates if queue is closed <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/86">#86</a></li> </ul> <h2>0.3.2 (2018-07-06)</h2> <ul> <li>Fixed python 3.7 support <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/97">#97</a></li> </ul> <h2>0.3.1 (2018-01-30)</h2> <ul> <li>Fixed bug with join() in case tasks are added by sync_q.put() <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/75">#75</a></li> </ul> <h2>0.3.0 (2017-02-21)</h2> <ul> <li>Expose unfinished_tasks property <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/34">#34</a></li> </ul> <h2>0.2.4 (2016-12-05)</h2> <ul> <li>Restore tarball deploying</li> </ul> <h2>0.2.3 (2016-07-12)</h2> <ul> <li>Fix exception type</li> </ul> <h2>0.2.2 (2016-07-11)</h2> <ul> <li>Update asyncio.async() to use asyncio.ensure_future() <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/6">#6</a></li> </ul> <h2>0.2.1 (2016-03-24)</h2> <ul> <li>Fix python setup.py test command <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/4">#4</a></li> </ul> <!-- raw HTML omitted --> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1017/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
719382156 | MDExOlB1bGxSZXF1ZXN0NTAxNTc5MzE1 | 1018 | Update asgiref requirement from ~=3.2.10 to >=3.2.10,<3.4.0 | dependabot-preview[bot] 27856297 | closed | 0 | 1 | 2020-10-12T13:30:09Z | 2020-10-14T21:51:36Z | 2020-10-14T21:51:35Z | CONTRIBUTOR | simonw/datasette/pulls/1018 | Updates the requirements on [asgiref](https://github.com/django/asgiref) to permit the latest version. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/django/asgiref/blob/master/CHANGELOG.txt">asgiref's changelog</a>.</em></p> <blockquote> <h2>3.3.0 (2020-10-09)</h2> <ul> <li>sync_to_async now defaults to thread-sensitive mode being on</li> <li>async_to_sync now works inside of forked processes</li> <li>WsgiToAsgi now correctly clamps its response body when Content-Length is set</li> </ul> <h2>3.2.10 (2020-08-18)</h2> <ul> <li>Fixed bugs due to bad WeakRef handling introduced in 3.2.8</li> </ul> <h2>3.2.9 (2020-06-16)</h2> <ul> <li>Fixed regression with exception handling in 3.2.8 related to the contextvars fix.</li> </ul> <h2>3.2.8 (2020-06-15)</h2> <ul> <li>Fixed small memory leak in local.Local</li> <li>contextvars are now persisted through AsyncToSync</li> </ul> <h2>3.2.7 (2020-03-24)</h2> <ul> <li>Bug fixed in local.Local where deleted Locals would occasionally inherit their storage into new Locals due to memory reuse.</li> </ul> <h2>3.2.6 (2020-03-23)</h2> <ul> <li>local.Local now works in all threading situations, no longer requires periodic garbage collection, and works with libraries that monkeypatch threading (like gevent)</li> </ul> <h2>3.2.5 (2020-03-11)</h2> <ul> <li><strong>self</strong> is now preserved on methods by async_to_sync</li> </ul> <h2>3.2.4 (2020-03-10)</h2> <!-- raw HTML omitted --> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/django/asgiref/commit/7dba5ff61313dfd1535de200590b9155abde20f0"><code>7dba5ff</code></a> Releasing 3.3.0</li> <li><a href="https://github.com/django/asgiref/commit/e1e0dd9a900966b29345e6069c0fdbc9a449e97f"><code>e1e0dd9</code></a> Added ZeroCopy extension</li> <li><a href="https://github.com/django/asgiref/commit/3834d13a8f5da921bb5f8ca16ee66d353fdbdadb"><code>3834d13</code></a> Added rpc.py to Implementations (<a href="https://github-redirect.dependabot.com/django/asgiref… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1018/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
719637258 | MDExOlB1bGxSZXF1ZXN0NTAxNzkxNjYz | 10 | Update utils.py to fix sqlite3.OperationalError | mattiaborsoi 29426418 | closed | 0 | 1 | 2020-10-12T20:17:53Z | 2020-10-12T20:25:10Z | 2020-10-12T20:25:09Z | CONTRIBUTOR | dogsheep/swarm-to-sqlite/pulls/10 | Fixes the errors: - sqlite3.OperationalError: table posts has no column named text - sqlite3.OperationalError: table photos has no column named hasSticker That will cause sqlite-utils to notice if there's a missing column and add it. As recommended by @simonw | swarm-to-sqlite 205429375 | pull | {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
721050815 | MDU6SXNzdWU3MjEwNTA4MTU= | 1019 | "Edit SQL" button on canned queries | jsfenfen 639012 | closed | 0 | 0.51 6026070 | 7 | 2020-10-14T00:51:39Z | 2020-10-23T19:44:06Z | 2020-10-14T03:44:23Z | CONTRIBUTOR | Feature request: Would it be possible to add an "edit this query" button on canned queries? Clicking it would open the canned query as an editable sql query. I think the intent is to have named parameters to allow this, but sometimes you just gotta rewrite it? | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1019/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | |||||
721830990 | MDExOlB1bGxSZXF1ZXN0NTAzNjg1MDc3 | 1022 | Fix table name in spatialite example command | jsfenfen 639012 | closed | 0 | 2 | 2020-10-14T22:19:34Z | 2020-10-14T23:46:46Z | 2020-10-14T23:46:46Z | CONTRIBUTOR | simonw/datasette/pulls/1022 | The example query for creating a new point geometry seems to be using a table called 'museums' but at one point it instead uses 'events'. I *believe* it is intended to be museums (the example makes more sense if so). | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1022/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
723837704 | MDExOlB1bGxSZXF1ZXN0NTA1MzM5NTE1 | 1029 | fix(docs): broken link | jthodge 17075617 | closed | 0 | 1 | 2020-10-17T20:03:20Z | 2020-10-17T20:05:04Z | 2020-10-17T20:05:04Z | CONTRIBUTOR | simonw/datasette/pulls/1029 | This PR fixes a broken markdown link in the `Publish` docs page. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1029/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
726154220 | MDExOlB1bGxSZXF1ZXN0NTA3MjY3MDg3 | 1038 | DOC: Fix syntax error | gerrymanoim 194147 | closed | 0 | 2 | 2020-10-21T05:45:38Z | 2020-10-21T22:57:21Z | 2020-10-21T22:44:17Z | CONTRIBUTOR | simonw/datasette/pulls/1038 | If I understand https://docs.datasette.io/en/stable/plugin_hooks.html#register-routes correctly, `register_routes` should return a `List[Tuple[str, Callable]]`. I believe the current code in documentation has a syntax error (extra `)`). | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1038/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
727915394 | MDExOlB1bGxSZXF1ZXN0NTA4NzE5NTY3 | 1043 | Include LICENSE in sdist | bollwyvl 45380 | closed | 0 | 4 | 2020-10-23T05:04:12Z | 2020-10-26T00:14:57Z | 2020-10-23T20:54:35Z | CONTRIBUTOR | simonw/datasette/pulls/1043 | Hi, thanks for `datasette`! This PR adds the `LICENSE` to source distributions, which seems the norm for Apache-2.0 stuff. I noticed the [0.50.2 sdist](https://files.pythonhosted.org/packages/f2/ba/1b5f182c3f1769c0863bcaa77406bdcb81c92e31bb579959c01b1d8951c0/datasette-0.50.2.tar.gz) doesn't ship `LICENSE`, but the 0.5.2 `whl` does, so I'm assuming the intent _is_ to ship... and it's a one-liner! Motivation: It might be a bit of a slog, but I'm looking to see about getting `datasette` (and friends!) available on conda-forge. There are a few missing upstreams (`asgi-csrf`, `python-basecov`, `mergedeep`) and some of the plugins don't even appear to _have_ tarballs (just `whl`!), but the little stuff like licenses are nice to get out handled upstream vs separately grabbing them. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1043/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
727916744 | MDExOlB1bGxSZXF1ZXN0NTA4NzIwNjYw | 1044 | Add minimum supported python | bollwyvl 45380 | closed | 0 | 2 | 2020-10-23T05:08:03Z | 2020-10-23T20:53:08Z | 2020-10-23T20:53:08Z | CONTRIBUTOR | simonw/datasette/pulls/1044 | Thanks for `datasette`! This PR adds `python_requires` to formally signal the [minimum supported python version](https://packaging.python.org/guides/dropping-older-python-versions/#specify-the-version-ranges-for-supported-python-distributions) (which is pointed out with classifiers, so seems pretty straightforward). | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1044/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
729017519 | MDExOlB1bGxSZXF1ZXN0NTA5NTkwMjA1 | 1049 | Add template block prior to extra URL loaders | psychemedia 82988 | closed | 0 | 4 | 2020-10-25T13:08:55Z | 2020-10-29T09:20:52Z | 2020-10-29T09:20:34Z | CONTRIBUTOR | simonw/datasette/pulls/1049 | To handle packages that require Javascript state setting prior to loading a package (eg [`thebelab`](https://thebelab.readthedocs.io/en/latest/examples/minimal_example.html), provide a template block before the URLs are loaded. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1049/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
729818242 | MDExOlB1bGxSZXF1ZXN0NTEwMjM1OTA5 | 189 | Allow iterables other than Lists in m2m records | adamwolf 35681 | closed | 0 | 3 | 2020-10-26T18:47:44Z | 2020-10-27T16:28:37Z | 2020-10-27T16:24:21Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/189 | I was playing around with sqlite-utils, creating a Roam Research dogsheep-style importer for Datasette, and ran into a slight snag. I wanted to use a generator to add an order column in an importer. It looked something like: ``` def order_generator(iterable, attr=None): if attr is None: attr = "order" order: int = 0 for i in iterable: i[attr] = order order += 1 yield i ``` When I used this with `insert_all` and other things, it worked fine--but it didn't work as the `records` argument to `m2m`. I dug into it, and sqlite-utils is explicitly checking if the records argument is a list or a tuple. I flipped the check upside down, and now it checks if the argument is a mapping. If it's a mapping, it wraps it in a list, otherwise it leaves it alone. (I get that it might not really make sense to put the order column on the second table. I changed my import schema a bit, and no longer have a real example, but maybe this change still makes sense.) The automated tests still pass, but I did not add any new ones. Let me know what you think! I'm really loving Datasette and its ecosystem; thanks for everything! | sqlite-utils 140912432 | pull | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/189/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
731445447 | MDExOlB1bGxSZXF1ZXN0NTExNTQ5Mzc0 | 1059 | Update aiofiles requirement from <0.6,>=0.4 to >=0.4,<0.7 | dependabot-preview[bot] 27856297 | closed | 0 | 2 | 2020-10-28T13:32:40Z | 2020-10-28T17:08:29Z | 2020-10-28T17:08:28Z | CONTRIBUTOR | simonw/datasette/pulls/1059 | Updates the requirements on [aiofiles](https://github.com/Tinche/aiofiles) to permit the latest version. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/Tinche/aiofiles/commit/b4482aeef5312d4bf0c1abda5892df80957fa01f"><code>b4482ae</code></a> 0.6.0</li> <li><a href="https://github.com/Tinche/aiofiles/commit/eaccd3e43013b843f606665a68d3db395437e537"><code>eaccd3e</code></a> Update Changelog</li> <li><a href="https://github.com/Tinche/aiofiles/commit/2de90cf95e48db9178ef5bb246b5866892e8ef8e"><code>2de90cf</code></a> Test on 3.9</li> <li><a href="https://github.com/Tinche/aiofiles/commit/62b4c27096a5437f0608886e98cbe2681992ca57"><code>62b4c27</code></a> Update .travis.yml</li> <li><a href="https://github.com/Tinche/aiofiles/commit/4083c7371a2e667a316efa55613cfe6b826304b2"><code>4083c73</code></a> Removed pypy3 for ppc64le</li> <li><a href="https://github.com/Tinche/aiofiles/commit/26e795b5dafe7d227147f1663ffeca6a76efe453"><code>26e795b</code></a> Update .travis.yml</li> <li><a href="https://github.com/Tinche/aiofiles/commit/bd154cf3bf60f03aa22b3b264578a987d136be85"><code>bd154cf</code></a> Update .travis.yml</li> <li><a href="https://github.com/Tinche/aiofiles/commit/7ae0e2ace9193e4e2417fc76ea51caf84ba0e656"><code>7ae0e2a</code></a> Update .travis.yml</li> <li><a href="https://github.com/Tinche/aiofiles/commit/84eeb6c686ca6ba0ac700e93a078547cbaac625d"><code>84eeb6c</code></a> Update .travis.yml</li> <li><a href="https://github.com/Tinche/aiofiles/commit/c48f5b0a825f8c68c2238f098585c1f1c6ba2c16"><code>c48f5b0</code></a> Proxy name and mode properties</li> <li>Additional commits viewable in <a href="https://github.com/Tinche/aiofiles/compare/v0.4.0...v0.6.0">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and opti… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1059/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
748370021 | MDExOlB1bGxSZXF1ZXN0NTI1MzcxMDI5 | 8 | fix import error if note has no "updated" element | mkorosec 4028322 | closed | 0 | 0 | 2020-11-22T22:51:05Z | 2021-02-11T22:34:06Z | 2021-02-11T22:34:06Z | CONTRIBUTOR | dogsheep/evernote-to-sqlite/pulls/8 | I got the following error when executing evernote-to-sqlite enex evernote.db evernote.enex ``` ... File "evernote_to_sqlite/cli.py", line 31, in enex save_note(db, note) File "evernote_to_sqlite/utils.py", line 28, in save_note updated = note.find("updated").text AttributeError: 'NoneType' object has no attribute 'text' ``` Seems that in some cases the updated element is not added to the note, this is a part of the problematic note: ``` <created>20201019T074518Z</created> <note-attributes> <source>web.clip7</source> <source-application>webclipper.evernote</source-application> </note-attributes> ``` | evernote-to-sqlite 303218369 | pull | {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
748372469 | MDU6SXNzdWU3NDgzNzI0Njk= | 9 | ParseError: undefined entity š | mkorosec 4028322 | closed | 0 | 1 | 2020-11-22T23:04:35Z | 2021-02-11T22:10:55Z | 2021-02-11T22:10:55Z | CONTRIBUTOR | I encountered a parse error if the enex file contained š or Run command: evernote-to-sqlite enex evernote.db evernote.enex ``` Traceback (most recent call last): ... File "evernote_to_sqlite/cli.py", line 31, in enex save_note(db, note) File "evernote_to_sqlite/utils.py", line 35, in save_note content = ET.tostring(ET.fromstring(content_xml)).decode("utf-8") File "/usr/lib/python3.8/xml/etree/ElementTree.py", line 1320, in XML parser.feed(text) xml.etree.ElementTree.ParseError: undefined entity š: line 3, column 35 ``` Workaround: ``` sed -i 's/š//g' evernote.enex sed -i 's/ //g' evernote.enex ``` | evernote-to-sqlite 303218369 | issue | {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
750141615 | MDExOlB1bGxSZXF1ZXN0NTI2ODQ3ODIz | 7 | Fixed conflicting CLI flags | tlockney 8944 | closed | 0 | 1 | 2020-11-24T23:25:12Z | 2022-08-21T21:11:56Z | 2022-08-21T21:11:56Z | CONTRIBUTOR | dogsheep/pocket-to-sqlite/pulls/7 | The `-a` used for the auth credentials and the shortened form of the `--all` flags were in conflict on the `fetch` command. To be consistent with other `-to-sqlite` libraries in the Dogsheep ecosystem, I removed the shortened form of the `--all` flag. | pocket-to-sqlite 213286752 | pull | {"url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
751195017 | MDU6SXNzdWU3NTExOTUwMTc= | 1111 | Accessing a database's `.json` is slow for very large SQLite files | asg017 15178711 | open | 0 | 3 | 2020-11-26T00:27:27Z | 2021-01-04T19:57:53Z | CONTRIBUTOR | I have a SQLite DB that's pretty large, 23GB and something like 300 million rows. I expect that most queries I run on it will be slow, which is fine, but there are some things that Datasette does that makes working with the DB very slow. Specifically, when I access the `.json` metadata for a table (which I believe it comes from `datasette/views/database.py`, it takes 43 seconds for the request to come in: ```bash $ time curl localhost:9999/out.json {"database": "out", "size": 24291454976, "tables": [{"name": "PageviewsHour", "columns": ["file", "code", "page", "pageviews"], "primary_keys": [], "count": null, "hidden": false, "fts_table": null, "foreign_keys": {"incoming": [], "outgoing": [{"other_table": "PageviewsHourFiles", "column": "file", "other_column": "file_id"}]}, "private": false}, {"name": "PageviewsHourFiles", "columns": ["file_id", "filename", "sha256", "size", "day", "hour"], "primary_keys": ["file_id"], "count": null, "hidden": false, "fts_table": null, "foreign_keys": {"incoming": [{"other_table": "PageviewsHour", "column": "file_id", "other_column": "file"}], "outgoing": []}, "private": false}, {"name": "sqlite_sequence", "columns": ["name", "seq"], "primary_keys": [], "count": 1, "hidden": false, "fts_table": null, "foreign_keys": {"incoming": [], "outgoing": []}, "private": false}], "hidden_count": 0, "views": [], "queries": [], "private": false, "allow_execute_sql": true, "query_ms": 43340.23213386536} real 0m43.417s user 0m0.006s sys 0m0.016s ``` I suspect this is because a `COUNT(*)` is happening under the hood, which, when I run it through sqlite directly, does take around the same time: ```bash $ time sqlite3 out.db < <(echo "select count(*) from PageviewsHour;") 362794272 real 0m44.523s user 0m2.497s sys 0m6.703s ``` I'm using the `.json` request in the [Observable Datasette Client](https://observablehq.com/@asg017/datasette-client) to 1) verify that a link passed in is a reachable Datasette instance, and 2) a quick way to look at metadata for a db. A few differe… | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1111/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
752749485 | MDExOlB1bGxSZXF1ZXN0NTI4OTk3NjE0 | 1112 | Fix --metadata doc usage | jefftriplett 50527 | closed | 0 | Datasette 0.52 6055094 | 3 | 2020-11-28T19:19:51Z | 2020-11-28T23:28:21Z | 2020-11-28T19:53:48Z | CONTRIBUTOR | simonw/datasette/pulls/1112 | I stumbled on this while trying to figure out how to configure datasette-ripgrep via https://github.com/simonw/datasette-ripgrep/issues/15 You may not want to update the changelog (those are annoying) so I added two commits in case that's easier. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1112/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||
752888228 | MDExOlB1bGxSZXF1ZXN0NTI5MDkwNTYw | 204 | use jsonify_if_need for sql updates | mfa 78035 | closed | 0 | 1 | 2020-11-29T10:49:00Z | 2020-12-08T17:49:42Z | 2020-12-08T17:49:42Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/204 | sqlite-utils 140912432 | pull | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/204/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||||
752966476 | MDU6SXNzdWU3NTI5NjY0NzY= | 1114 | --load-extension=spatialite not working with datasetteproject/datasette docker image | danp 2182 | closed | 0 | 4 | 2020-11-29T17:35:20Z | 2022-01-20T21:29:42Z | 2020-11-29T17:37:45Z | CONTRIBUTOR | https://github.com/simonw/datasette/commit/6aa5886379dd9017215904fb28567b80018902f9 added the `--load-extension=spatialite` shortcut looking for the extension in these places: https://github.com/simonw/datasette/blob/12877d7a48e2aa28bb5e780f929a218f7265d849/datasette/utils/__init__.py#L56-L60 However, in the datasetteproject/datasette docker image the file is at `/usr/local/lib/mod_spatialite.so`. This results in the example command [here](https://docs.datasette.io/en/stable/installation.html#loading-spatialite) failing: ``` % docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=spatialite Error: Could not find SpatiaLite extension ``` But it does work when given an explicit path: ``` % docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=/usr/local/lib/mod_spatialite.so INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ... ``` Perhaps `SPATIALITE_PATHS` should include `/usr/local/lib/mod_spatialite.so`? | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1114/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
754178780 | MDU6SXNzdWU3NTQxNzg3ODA= | 1121 | Table actions cog is misaligned | abdusco 3243482 | closed | 0 | 1 | 2020-12-01T08:41:25Z | 2020-12-03T01:03:19Z | 2020-12-03T00:33:36Z | CONTRIBUTOR | At the moment it looks like this https://datasette-graphql-demo.datasette.io/github/repos ![image](https://user-images.githubusercontent.com/3243482/100716533-e6e2d300-33c9-11eb-866e-1e83ba228bf5.png) Adding a few flex statements fixes the alignment and centers `h1` text and the cog icon vertically. ![image](https://user-images.githubusercontent.com/3243482/100716605-f8c47600-33c9-11eb-8d69-0e37499cf641.png) | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1121/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
754179035 | MDExOlB1bGxSZXF1ZXN0NTMwMTI1Njk1 | 1122 | Fix misaligned table actions cog | abdusco 3243482 | closed | 0 | 2 | 2020-12-01T08:41:46Z | 2020-12-03T10:56:40Z | 2020-12-03T00:33:37Z | CONTRIBUTOR | simonw/datasette/pulls/1122 | Fixes https://github.com/simonw/datasette/issues/1121 | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1122/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
756867924 | MDExOlB1bGxSZXF1ZXN0NTMyMzQyMDI1 | 1128 | Fix startup error on windows | abdusco 3243482 | closed | 0 | 2 | 2020-12-04T07:12:26Z | 2020-12-06T08:41:45Z | 2020-12-05T19:35:04Z | CONTRIBUTOR | simonw/datasette/pulls/1128 | Fixes https://github.com/simonw/datasette/issues/1094 This import isn't used at all, and causes error on startup on Windows. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1128/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
756875827 | MDU6SXNzdWU3NTY4NzU4Mjc= | 1129 | Fix footer to the bottom of the page | abdusco 3243482 | open | 0 | 0 | 2020-12-04T07:28:07Z | 2020-12-04T16:04:29Z | CONTRIBUTOR | Footer doesn't stick to the bottom if the body content isn't long enough to reach the end of viewport. ![before & after](https://user-images.githubusercontent.com/3243482/101134785-f6595a80-361b-11eb-81ce-b8b5cb9c5bc2.png) This can be fixed using flexbox. ```css body { min-height: 100vh; display: flex; flex-direction: column; } .content { flex-grow: 1; } ``` | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1129/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
756876238 | MDExOlB1bGxSZXF1ZXN0NTMyMzQ4OTE5 | 1130 | Fix footer not sticking to bottom in short pages | abdusco 3243482 | open | 0 | 4 | 2020-12-04T07:29:01Z | 2021-06-15T13:27:48Z | CONTRIBUTOR | simonw/datasette/pulls/1130 | Fixes https://github.com/simonw/datasette/issues/1129 | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1130/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||||
766156875 | MDU6SXNzdWU3NjYxNTY4NzU= | 209 | Test failure with sqlite 3.34 in test_cli.py::test_optimize | meatcar 191622 | closed | 0 | 1 | 2020-12-14T08:58:18Z | 2021-01-01T23:52:46Z | 2021-01-01T23:52:46Z | CONTRIBUTOR | pytest output: ``` ... ============================== short test summary info =============================== FAILED tests/test_cli.py::test_optimize[tables0] - assert 1662976 < 1662976 FAILED tests/test_cli.py::test_optimize[tables1] - assert 1667072 < 1662976 ===================== 2 failed, 538 passed, 3 skipped in 34.32s ====================== ``` Came across this while packaging `sqlite-utils` for NixOS, but it can be recreated it using the `alpine:edge` docker image as well as follows: ``` docker run --rm -it alpine:edge /bin/sh # apk update && apk add git sqlite python3 gcc python3-dev musl-dev && python3 -m ensurepip # git clone https://github.com/simonw/sqlite-utils.git # cd sqlite-utils/ # pip3 install -e .[test] # pytest ``` This definitely works on sqlite v3.33. | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/209/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
766494367 | MDExOlB1bGxSZXF1ZXN0NTM5NDg5NTI1 | 1145 | Update pytest requirement from <6.2.0,>=5.2.2 to >=5.2.2,<6.3.0 | dependabot-preview[bot] 27856297 | closed | 0 | Datasette 0.54 6346396 | 1 | 2020-12-14T14:22:16Z | 2021-01-24T21:20:29Z | 2020-12-16T21:44:39Z | CONTRIBUTOR | simonw/datasette/pulls/1145 | Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/releases">pytest's releases</a>.</em></p> <blockquote> <h2>6.2.0</h2> <h1>pytest 6.2.0 (2020-12-12)</h1> <h2>Breaking Changes</h2> <ul> <li><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7808">#7808</a>: pytest now supports python3.6+ only.</li> </ul> <h2>Deprecations</h2> <ul> <li> <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7469">#7469</a>: Directly constructing/calling the following classes/functions is now deprecated:</p> <ul> <li><code>_pytest.cacheprovider.Cache</code></li> <li><code>_pytest.cacheprovider.Cache.for_config()</code></li> <li><code>_pytest.cacheprovider.Cache.clear_cache()</code></li> <li><code>_pytest.cacheprovider.Cache.cache_dir_from_config()</code></li> <li><code>_pytest.capture.CaptureFixture</code></li> <li><code>_pytest.fixtures.FixtureRequest</code></li> <li><code>_pytest.fixtures.SubRequest</code></li> <li><code>_pytest.logging.LogCaptureFixture</code></li> <li><code>_pytest.pytester.Pytester</code></li> <li><code>_pytest.pytester.Testdir</code></li> <li><code>_pytest.recwarn.WarningsRecorder</code></li> <li><code>_pytest.recwarn.WarningsChecker</code></li> <li><code>_pytest.tmpdir.TempPathFactory</code></li> <li><code>_pytest.tmpdir.TempdirFactory</code></li> </ul> <p>These have always been considered private, but now issue a deprecation warning, which may become a hard error in pytest 7.0.0.</p> </li> <li> <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7530">#7530</a>: The <code>--strict</code> command-line option has been deprecated, use <code>--strict-markers</code> instead.</p> <p>We have plans to maybe in the future to reintroduce <code>--strict</code> and make it an encompassing flag for all strictness related options (<code>--strict-markers</code> and <code>--strict-config</code> a… | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1145/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||
771511344 | MDExOlB1bGxSZXF1ZXN0NTQzMDE1ODI1 | 31 | Update for Big Sur | RhetTbull 41546558 | open | 0 | 5 | 2020-12-20T04:36:45Z | 2023-04-14T22:41:31Z | CONTRIBUTOR | dogsheep/dogsheep-photos/pulls/31 | Refactored out the SQL for extracting aesthetic scores to use osxphotos -- adds compatbility for Big Sur via osxphotos which has been updated for new table names in Big Sur. Have not yet refactored the SQL for extracting labels which is still compatible with Big Sur. | dogsheep-photos 256834907 | pull | {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||||
771872303 | MDExOlB1bGxSZXF1ZXN0NTQzMjQ2NTM1 | 59 | Remove unneeded exists=True for -a/--auth flag. | frosencrantz 631242 | closed | 0 | 3 | 2020-12-21T06:03:55Z | 2021-05-22T14:06:19Z | 2021-05-19T16:08:12Z | CONTRIBUTOR | dogsheep/github-to-sqlite/pulls/59 | The file does not need to exist when using an environment variable. | github-to-sqlite 207052882 | pull | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
773913793 | MDExOlB1bGxSZXF1ZXN0NTQ0OTIzNDM3 | 1158 | Modernize code to Python 3.6+ | eumiro 6774676 | closed | 0 | Datasette 0.54 6346396 | 4 | 2020-12-23T16:21:38Z | 2021-01-24T21:20:50Z | 2020-12-23T17:04:32Z | CONTRIBUTOR | simonw/datasette/pulls/1158 | - compact dict and set building - remove redundant parentheses - simplify chained conditions - change method name to lowercase - use triple double quotes for docstrings please feel free to accept/reject any of these independent commits | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1158/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||
777677671 | MDU6SXNzdWU3Nzc2Nzc2NzE= | 1169 | Prettier package not actually being cached | benpickles 3637 | closed | 0 | 4 | 2021-01-03T17:04:41Z | 2021-01-04T19:52:34Z | 2021-01-04T19:52:33Z | CONTRIBUTOR | With the current configuration Prettier seems to be installed on every run - which can been [seen from the output](https://github.com/simonw/datasette/runs/1631686028?check_suite_focus=true#step:4:4): ``` npx: installed 1 in 5.166s ``` Prettier isn't explicitly being installed (it's surprising that actually installing the dependencies isn't included in the [actions/cache docs](https://github.com/actions/cache/blob/main/examples.md#macos-and-ubuntu)) but it turns out that `npx` will automatically install the package for the specified command (it actually _guesses_ the package name from the name of the command). I'm not sure where Prettier ends up being installed but it doesn't appear to be in `~/.npm` according to the [post-cache output](https://github.com/simonw/datasette/runs/1631686028#step:7:2) (or `./node_modules` when I tested locally): ``` Cache hit occurred on the primary key Linux-npm-565329898f77080e58b14d45cf816ab94877e6f2ece9d395c369c533548a7ee7, not saving cache. ``` I think there are a couple of approaches to tackling this, you could manually install/cache Prettier within the action, or add a `package.json` with Prettier. I would go with the latter because it's a more standard and maintainable approach and it will also ensure that, along with CI, anyone working on the project will run the same version of Prettier (you'll also get Dependabot JavaScript updates). I've tested the [`package.json` approach on a branch](https://github.com/simonw/datasette/compare/main...benpickles:cache-prettier) and am happy to turn it into a pull request if you fancy. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1169/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
778126516 | MDExOlB1bGxSZXF1ZXN0NTQ4MjcxNDcy | 1170 | Install Prettier via package.json | benpickles 3637 | closed | 0 | Datasette 0.54 6346396 | 3 | 2021-01-04T14:18:03Z | 2021-01-24T21:21:01Z | 2021-01-04T19:52:34Z | CONTRIBUTOR | simonw/datasette/pulls/1170 | This adds a package.json with Prettier and means that developers/CI will use the same version. It also ensures that NPM packages are cached on GitHub Actions which fixes #1169. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1170/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||
779088071 | MDU6SXNzdWU3NzkwODgwNzE= | 54 | Archive import appears to be broken on recent exports | jacobian 21148 | open | 0 | 5 | 2021-01-05T14:18:01Z | 2023-01-04T11:06:55Z | CONTRIBUTOR | I requested a Twitter export yesterday, and unfortunately they seem to have changed it such that `twitter-to-sqlite import` can't handle it anymore 😢 So far I've ran into two issues. The first was easy to work around, but the second will take more investigation. If I can find the time I'll keep working on it and update this issue accordingly. The issues (so far): ### 1. Data seems to have moved to a `data/` subdirectory Running `twitter-to-sqlite import` on the raw zip file reports a bunch of "not yet implemented" errors, and then exits without actually importing anything: ``` ❯ twitter-to-sqlite import tarchive.db twitter.zip ... data/manifest: not yet implemented data/account-creation-ip: not yet implemented data/account-suspension: not yet implemented ... (dozens of more lines like this, including critical stuff like data/tweets) ... ``` (`tarchive.db` now exists, but is empty) Workaround: unpack the zip file, and run `twitter-to-sqlite import tarchive.db path/to/archive/data` That gets further, but: ### 2. Some schema(s?) have changed At least, the `blocks` schema seems different now: ``` ❯ twitter-to-sqlite import tarchive.db archive/data direct-messages-group: not yet implemented branch-links: not yet implemented periscope-expired-broadcasts: not yet implemented direct-messages: not yet implemented mute: not yet implemented Traceback (most recent call last): File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/bin/twitter-to-sqlite", line 8, in <module> sys.exit(cli()) File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN… | twitter-to-sqlite 206156866 | issue | {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
779211940 | MDExOlB1bGxSZXF1ZXN0NTQ5MjA0MDYz | 55 | Fix archive imports | jacobian 21148 | closed | 0 | 2 | 2021-01-05T15:54:48Z | 2021-08-20T00:02:49Z | 2021-08-20T00:02:49Z | CONTRIBUTOR | dogsheep/twitter-to-sqlite/pulls/55 | This fixes the issues discussed in #54 | twitter-to-sqlite 206156866 | pull | {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
794554881 | MDU6SXNzdWU3OTQ1NTQ4ODE= | 1208 | A lot of open(file) functions are used without a context manager thus producing ResourceWarning: unclosed file <_io.TextIOWrapper | kbaikov 4488943 | closed | 0 | 2 | 2021-01-26T20:56:28Z | 2021-03-11T16:15:49Z | 2021-03-11T16:15:49Z | CONTRIBUTOR | Your code is full of open files that are never closed, especially when you deal with reading/writing json/yaml files. If you run python with warnings enabled this problem becomes evident. This probably contributes to some memory leaks in long running datasettes if the GC will not 'collect' those resources properly. This is easily fixed by using a context manager instead of just using open: ```python with open('some_file', 'w') as opened_file: opened_file.write('string') ``` In some newer parts of the code you use Path objects 'read_text' and 'write_text' functions which close the file properly and are prefered in some cases. If you want I can create a PR for all places i found this pattern in. Bellow is a fraction of places where i found a ResourceWarning: ```python update-docs-help.py: 20 actual = actual.replace("Usage: cli ", "Usage: datasette ") 21: open(docs_path / filename, "w").write(actual) 22 datasette\app.py: 210 ): 211: inspect_data = json.load((config_dir / "inspect-data.json").open()) 212 if immutables is None: 266 if config_dir and (config_dir / "settings.json").exists() and not config: 267: config = json.load((config_dir / "settings.json").open()) 268 self._settings = dict(DEFAULT_SETTINGS, **(config or {})) 445 self._app_css_hash = hashlib.sha1( 446: open(os.path.join(str(app_root), "datasette/static/app.css")) 447 .read() datasette\cli.py: 130 else: 131: out = open(inspect_file, "w") 132 loop = asyncio.get_event_loop() 459 if inspect_file: 460: inspect_data = json.load(open(inspect_file)) 461 ``` | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1208/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
797097140 | MDU6SXNzdWU3OTcwOTcxNDA= | 60 | Use Data from SQLite in other commands | daniel-butler 22578954 | open | 0 | 3 | 2021-01-29T18:35:52Z | 2021-02-12T18:29:43Z | CONTRIBUTOR | As a total beginner here how could you access data from the sqlite table to run other commands. What I am thinking is I want to get all the repos in an organization then using the repo list pull all the commit messages for each repo. I love this project by the way! | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
797108702 | MDExOlB1bGxSZXF1ZXN0NTY0MTcyMTQw | 61 | fixing typo in get cli help text | daniel-butler 22578954 | closed | 0 | 1 | 2021-01-29T18:57:04Z | 2021-05-19T16:07:09Z | 2021-05-19T16:07:09Z | CONTRIBUTOR | dogsheep/github-to-sqlite/pulls/61 | github-to-sqlite 207052882 | pull | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/61/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||||
797649915 | MDExOlB1bGxSZXF1ZXN0NTY0NjA4MjY0 | 1211 | Use context manager instead of plain open | kbaikov 4488943 | closed | 0 | 3 | 2021-01-31T07:58:10Z | 2021-03-11T16:15:50Z | 2021-03-11T16:15:50Z | CONTRIBUTOR | simonw/datasette/pulls/1211 | Context manager with open closes the files after usage. Fixes: https://github.com/simonw/datasette/issues/1208 When the object is already a pathlib.Path i used read_text write_text functions In some cases pathlib.Path.open were used in context manager, it is basically the same as builtin open. Tests are passing: 850 passed, 5 xfailed, 10 xpassed | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1211/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
797651831 | MDU6SXNzdWU3OTc2NTE4MzE= | 1212 | Tests are very slow. | kbaikov 4488943 | closed | 0 | 4 | 2021-01-31T08:06:16Z | 2021-02-19T22:54:13Z | 2021-02-19T22:54:13Z | CONTRIBUTOR | Working on my PR i noticed that tests are very slow. The plain pytest run took about 37 minutes for me. However i could shave of about 10 minutes from that if i used pytest-xdist to parallelize execution. `pytest -n 8` is run only in 28 minutes on my machine. I can create a PR to mention that in your documentation. This will be a simple change to add pytest-xdist to requirements and change a command to run pytest in documentation. Does that make sense to you? After a bit more investigation it looks like python-xdist is not an answer. It creates a race condition for tests that try to clead temp dir before run. Profiling shows that most time is spent on conn.executescript(TABLES) in make_app_client function. Which makes sense. Perhaps the better approach would be look at the app_client fixture which is already session scoped, but not used by all test cases. And/or use conn = sqlite3.connect(":memory:") which is much faster. And/or truncate tables after each TC instead of deleting the file and re-creating them. I can take a look which is the best approach if you give the go-ahead. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1212/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
797784080 | MDU6SXNzdWU3OTc3ODQwODA= | 62 | Stargazers and workflows commands always require an auth file when using GITHUB_TOKEN | frosencrantz 631242 | open | 0 | 0 | 2021-01-31T18:56:05Z | 2021-01-31T18:56:05Z | CONTRIBUTOR | Requested fix in https://github.com/dogsheep/github-to-sqlite/pull/59 The stargazers and workflows commands always require an auth file, even when using a `GITHUB_TOKEN`. Other commands don't require the auth file. | github-to-sqlite 207052882 | issue | {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/62/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
806918878 | MDExOlB1bGxSZXF1ZXN0NTcyMjU0MTAz | 1223 | Add compile option to Dockerfile to fix failing test (fixes #696) | bobwhitelock 7476523 | closed | 0 | 2 | 2021-02-12T03:38:05Z | 2021-03-07T12:01:12Z | 2021-03-07T07:41:17Z | CONTRIBUTOR | simonw/datasette/pulls/1223 | This test was failing when run inside the Docker container: `test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3]`, with this error: ``` def test_searchable(app_client, path, expected_rows): response = app_client.get(path) > assert expected_rows == response.json["rows"] E AssertionError: assert [[1, 'barry c...sel', 'puma']] == [] E Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther'] E Full diff: E + [] E - [[1, 'barry cat', 'terry dog', 'panther'], E - [2, 'terry dog', 'sara weasel', 'puma']] ``` The issue was that the version of sqlite3 built inside the Docker container was built with FTS3 and FTS4 enabled, but without the `SQLITE_ENABLE_FTS3_PARENTHESIS` compile option passed, which adds support for using `AND` and `NOT` within `match` expressions (see https://sqlite.org/fts3.html#compiling_and_enabling_fts3_and_fts4 and https://www.sqlite.org/compile.html). Without this, the `AND` used in the search in this test was being interpreted as a literal string, and so no matches were found. Adding this compile option fixes this. --- I actually ran into this issue because the same test was failing when I ran the test suite on my own machine, outside of Docker, and so I eventually tracked this down to my system sqlite3 also being compiled without this option. I wonder if this is a sign of a slightly deeper issue, that Datasette can silently behave differently based on the version and compilation of sqlite3 it is being used with. On my own system I fixed the test suite by running `pip install pysqlite3-binary`, so that this would be picked up instead of the `sqlite` package, as this seems to be compiled using this option, . Maybe using `pysqlite3-binary` could be installed/recommended by default so a more deterministic version of sqlite is used? Or there could be some feature detection done on the available sqlite version, to know what features are … | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1223/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
807433181 | MDU6SXNzdWU4MDc0MzMxODE= | 1224 | can't start immutable databases from configuration dir mode | camallen 295329 | closed | 0 | 0 | 2021-02-12T17:50:13Z | 2021-03-29T00:17:31Z | 2021-03-29T00:17:31Z | CONTRIBUTOR | Say I have a `/databases/` directory with multiple sqlite db files in that dir (`1.db` & `2.db`) and an `inspect-data.json` file. If I start datasette via `datasette -h 0.0.0.0 /databases/` then the resulting databases are set to `is_mutable: true` as inspected via http://127.0.0.1:8001/-/databases.json I don't want to have to list out the databases by name, e.g. `datasette -i /databases/1.db -i /databases/2.db` as i want the system to autodetect the sqlite dbs i have in the configuration directory According to the docs outlined in https://docs.datasette.io/en/latest/settings.html?highlight=immutable#configuration-directory-mode this should be possible > `inspect-data.json` the result of running datasette inspect - any database files listed here will be treated as immutable, so they should not be changed while Datasette is running I believe that if the `inspect-json.json` file present, then in theory the databases will be automatically set to immutable via this code https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/app.py#L211-L216 However it appears the Click Multiple Options will return a tuple via https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L311-L317 The resulting tuple is passed to the Datasette app via `kwargs` and overrides the behaviour to set the databases to immutable via this arg https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/app.py#L182 If you think this is a bug and needs fixing, I am willing to make a PR to check for the empty `immutable` tuple before calling the Datasette class initializer as I think leaving that class interface alone is the best path here. Thoughts? Also - i'm loving Datasette, it truly is a wonderful tool, thank you :) | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1224/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
810507413 | MDExOlB1bGxSZXF1ZXN0NTc1MTg3NDU3 | 1229 | ensure immutable databses when starting in configuration directory mode with | camallen 295329 | closed | 0 | 3 | 2021-02-17T20:18:26Z | 2022-04-22T13:16:36Z | 2021-03-29T00:17:32Z | CONTRIBUTOR | simonw/datasette/pulls/1229 | fixes #1224 This PR ensures all databases found in a configuration directory that match the files in `inspect-data.json` will be set to `immutable` as outlined in https://docs.datasette.io/en/latest/settings.html#configuration-directory-mode specifically on building the `datasette` instance it checks: - if `immutables` is an empty tuple - as passed by the cli code - if `immutables` is the default function value `None` - when it's not explicitly set And correctly builds the immutable database list from the `inspect-data[file]` keys. Note for this to work the `inspect-data.json` file must contain `file` paths which are relative to the configuration directory otherwise the file paths won't match and the dbs won't be set to immutable. I couldn't find an easy way to test this due to the way `make_app_client` works, happy to take directions on adding a test for this. I've updated the relevant docs as well, i.e. use the `inspect` cli cmd from the config directory path to create the relevant file ``` cd $config_dir datasette inspect *.db --inspect-file=inspect-data.json ``` https://docs.datasette.io/en/latest/performance.html#using-datasette-inspect | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1229/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
815955014 | MDExOlB1bGxSZXF1ZXN0NTc5Njk3ODMz | 1243 | fix small typo | UtahDave 306240 | closed | 0 | 2 | 2021-02-25T00:22:34Z | 2021-03-04T05:46:10Z | 2021-03-04T05:46:10Z | CONTRIBUTOR | simonw/datasette/pulls/1243 | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1243/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||||
817989436 | MDU6SXNzdWU4MTc5ODk0MzY= | 242 | Async support | eyeseast 25778 | open | 0 | 13 | 2021-02-27T18:29:38Z | 2021-10-28T14:37:56Z | CONTRIBUTOR | Following our conversation last week, want to note this here before I forget. I've had a couple situations where I'd like to do a bunch of updates in an async event loop, but I run into SQLite's issues with concurrent writes. This feels like something sqlite-utils could help with. PeeWee ORM has a [SQLite write queue](http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#sqliteq) that might be a good model. It's using threads or gevent, but I _think_ that approach would translate well enough to asyncio. Happy to help with this, too. | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
820468864 | MDExOlB1bGxSZXF1ZXN0NTgzNDA3OTg5 | 244 | Typo in upsert example | j-e-d 387669 | closed | 0 | 1 | 2021-03-02T23:14:14Z | 2021-05-19T02:58:21Z | 2021-05-19T02:58:21Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/244 | Remove extra `[` | sqlite-utils 140912432 | pull | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/244/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
825217564 | MDExOlB1bGxSZXF1ZXN0NTg3MzMyNDcz | 1252 | Add back styling to lists within table cells (fixes #1141) | bobwhitelock 7476523 | closed | 0 | 2 | 2021-03-09T03:00:57Z | 2021-03-29T00:14:04Z | 2021-03-29T00:14:04Z | CONTRIBUTOR | simonw/datasette/pulls/1252 | This overrides the Datasette reset - see https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/static/app.css#L35-L38 - to add back the default styling of list items displayed within Datasette table cells. Following this change, the same content as in the original issue looks like this: ![2021-03-09_02:57:32](https://user-images.githubusercontent.com/7476523/110411982-63e5ae80-8083-11eb-9b5c-e5dc825073e2.png) | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1252/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
827341657 | MDExOlB1bGxSZXF1ZXN0NTg5MjYzMjk3 | 1256 | Minor type in IP adress | JBPressac 6371750 | closed | 0 | 3 | 2021-03-10T08:28:22Z | 2021-03-10T18:26:46Z | 2021-03-10T18:26:40Z | CONTRIBUTOR | simonw/datasette/pulls/1256 | 127.0.01 replaced by 127.0.0.1 | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1256/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
830803173 | MDExOlB1bGxSZXF1ZXN0NTkyMjg5MzI0 | 245 | Correct some typos | dbready 1076745 | closed | 0 | 1 | 2021-03-13T04:26:56Z | 2021-05-19T02:58:04Z | 2021-05-19T02:58:04Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/245 | Noticed a typo in the docs and followed that up with a spellcheck. Had to bite my tongue at some of the British spellings. | sqlite-utils 140912432 | pull | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/245/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
831751367 | MDU6SXNzdWU4MzE3NTEzNjc= | 246 | Escaping FTS search strings | DeNeutoy 16001974 | closed | 0 | 4 | 2021-03-15T12:15:09Z | 2021-08-18T18:57:13Z | 2021-08-18T18:43:12Z | CONTRIBUTOR | Thanks for the excellent library, it's very nice to use! I've been building some in memory search functionality for a data annotation tool i'm making, and I got tripped up a little bit with escaping the full text search queries. First I tried using `db.quote(q)`, which doesn't work, because sqlite FTS has it's own (separate)[ query syntax](https://www2.sqlite.org/fts5.html#full_text_query_syntax). You can see this happening here also: http://search-24ways.herokuapp.com/24ways-f8f455f/articles?_search=acces%2A I got around this by aggressively escaping quotes inside the query string like this: ```python quoted = q.replace('"', '""') quoted = f'"{quoted}"' print(quoted) results = db["data"].search(quoted, columns=["id"]) return [x["id"] for x in results] ``` This works in the sense it doesn't crash, but it also removes access to the search query syntax. Given the well specified definition, it might be possible for sqlite-utils to provide a `db.quote_query(q)` which would intelligently escape a query whilst leaving the syntax intact. This would be very nice! | sqlite-utils 140912432 | issue | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/246/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
832687563 | MDExOlB1bGxSZXF1ZXN0NTkzODA1ODA0 | 247 | FTS quote functionality from datasette | DeNeutoy 16001974 | closed | 0 | 2 | 2021-03-16T11:17:34Z | 2021-08-18T18:43:12Z | 2021-08-18T18:43:12Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/247 | Addresses #246 - this is a bit of a kludge because it doesn't actually *validate* the FTS string, just makes sure that it will not crash when executed, but I figured that building a query parser is a bit out of the scope of sqlite-utils and if you actually want to use the query language, you probably need to parse that yourself. | sqlite-utils 140912432 | pull | {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/247/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
839008371 | MDU6SXNzdWU4MzkwMDgzNzE= | 1274 | Might there be some way to comment metadata.json? | mroswell 192568 | closed | 0 | 2 | 2021-03-23T18:33:00Z | 2021-03-23T20:14:54Z | 2021-03-23T20:14:54Z | CONTRIBUTOR | I don't know what license to use... Would be nice to be able to add a comment regarding that uncertainty in my metadata.json file I like laktak's little video comment in favor of Human json (Hjson) https://stackoverflow.com/questions/244777/can-comments-be-used-in-json Hmmm... one of the commenters there said comments are allowed in yaml... so that's a good argument for yaml. Anyhow, just came to mind, and thought I'd mention it here. Looks like https://hjson.github.io/ has the details. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1274/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | completed | ||||||
842556944 | MDExOlB1bGxSZXF1ZXN0NjAyMTA3OTM1 | 1279 | Minor Docs Update. Added `--app` to fly install command. | koaning 1019791 | closed | 0 | 2 | 2021-03-27T16:58:08Z | 2021-03-29T00:11:55Z | 2021-03-29T00:11:55Z | CONTRIBUTOR | simonw/datasette/pulls/1279 | Without this flag, there's an error locally. ``` > datasette publish fly bigmac.db Usage: datasette publish fly [OPTIONS] [FILES]... Try 'datasette publish fly --help' for help. Error: Missing option '-a' / '--app'. ``` I also got an error message which later turned out to be because I hadn't added my credit card information yet to `fly`. I wasn't sure if I should add that mention to the docs here, or to submit a bug-report over at https://github.com/simonw/datasette-publish-fly. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1279/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
843739658 | MDExOlB1bGxSZXF1ZXN0NjAzMDgyMjgw | 1282 | Fix little typo | mroswell 192568 | closed | 0 | 2 | 2021-03-29T19:45:28Z | 2021-03-29T19:57:34Z | 2021-03-29T19:57:34Z | CONTRIBUTOR | simonw/datasette/pulls/1282 | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1282/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | ||||||
843884745 | MDU6SXNzdWU4NDM4ODQ3NDU= | 1283 | advanced #export causes unexpected scrolling | mroswell 192568 | open | 0 | 0 | 2021-03-29T22:46:57Z | 2021-03-29T22:46:57Z | CONTRIBUTOR | 1. Visit a datasette table page 2. Click on the "(advanced)" link. This adds a fragment identifier "#export" to the URL, and scrolls down to the "Advanced export" div with the "export" id. 3. Manually scroll back up, and click on a suggested facet. The fragment identifier is still present, and the app scrolls back down to the "Advanced export" div. I think this is unwanted behavior. The user remedy seems to be to manually remove the "#export" from the URL. This behavior happens in my project, and in: https://covid-19.datasettes.com/covid/economist_excess_deaths (for instance) but not in this table: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1283/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
845794436 | MDU6SXNzdWU4NDU3OTQ0MzY= | 1284 | Feature or Documentation Request: Individual table as home page template | mroswell 192568 | open | 0 | 4 | 2021-03-31T03:56:17Z | 2021-11-04T03:15:01Z | CONTRIBUTOR | It would be great to have a sample showing how to move a single database that has a single table, to the index page. I'm trying it now, and find there is a real depth of Datasette and Python understanding that's required to be successful. I've got all the basic jinja concepts down... variables, template control structures, template inheritance, template overrides, css, html, the --template-dir and --static arguments, etc. But copying the table.html file to index.html doesn't work. There are undocumented functions and filters... I can figure some of them out (yay, url_builder.py and utils/__init__.py!) but it's a slog better handled by a much stronger Python developer. One sample would make a world of difference. The ideal form of this documentation would be a diff between the default table.html and how that would look if essentially moved to index.html. The use case is for everyone who wants to create a public-facing website to explore a single table at the root directory. (Maybe a second bit of documentation for people who have a single database with multiple tables.) (Hmm... might be cool to have a setting for that, where it happens automagically! If only one table, then home page is at the table level. if only one database, then home page is at the database level.... as an option.) I suppose I could ignore this, and somehow do this in the DNS settings once I hook up Vercel to a domain name, maybe.. and remove the breadcrumbs in table.html... but for now, a documentation request in the form of a diff... for viewing a single table (or a single database) at the root. (Actually, there's probably room for a whole expanded section on templates. Noticed some nice table metadata in one of the datasette examples, for instance... Hmm... maybe a whole library of solutions in one place... maybe a documentation hackathon! If that's of interest, of course it's a separate issue. ) | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1284/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
847700726 | MDU6SXNzdWU4NDc3MDA3MjY= | 1285 | Feature Request or Plugin Request: Numeric Range Facets | mroswell 192568 | open | 0 | 0 | 2021-04-01T01:50:20Z | 2021-04-01T02:28:19Z | CONTRIBUTOR | It would be great to offer facets for numeric data ranges. The ranges could pull from typical GIS methods of creating choropleth maps. https://gisgeography.com/choropleth-maps-data-classification/ Of the following, for mapping, I've always preferred a Jenks Natural Breaks, or a cross between Jenks and Pretty breaks. - Equal Intervals - Quantile (equal count) - Standard Deviation - Natural Breaks (Jenks) Classification - Pretty Breaks - Some sort of Aggregate Jenks Classification (this isn't standard, but it would be nice to be able to set classification ranges that work across tables.) Here are some links for Natural Breaks, in case this method is unfamiliar. - https://en.wikipedia.org/wiki/Jenks_natural_breaks_optimization - http://wiki.gis.com/wiki/index.php/Jenks_Natural_Breaks_Classification - https://medium.com/analytics-vidhya/jenks-natural-breaks-best-range-finder-algorithm-8d1907192051 Per that last link, there is a Jenks Python module... They also describe it as data-intensive for larger datasets. Maybe this is a good plugin idea. An example of equal Intervals would be 0 – < 10 10 – < 20 20 – < 30 30 – < 40 It's kind of confusing to have that less-than sign in there. it could also be displayed as: 0 – 10 10 – 20 20 – 30 30 – 40 But then it's not completely clear which category 10 is in, for instance. (Best to right-justify.. and use an "en dash" between numbers.) | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1285/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
849220154 | MDU6SXNzdWU4NDkyMjAxNTQ= | 1286 | Better default display of arrays of items | mroswell 192568 | open | 0 | 5 | 2021-04-02T13:31:40Z | 2021-06-12T12:36:15Z | CONTRIBUTOR | Would be great to have template filters that convert array fields to bullets and/or delimited lists upon table display: ``` |to_bullets |to_comma_delimited |to_semicolon_delimited ``` or maybe: ``` |join_array("bullet") |join_array("bullet","square") |join_array(";") |join_array(",") ``` Keeping in mind that bullets show up in html as \<li\> while other delimiting characters appear after the value. Of course, the fields themselves would remain as facetable arrays. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1286/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ||||||||
849582643 | MDExOlB1bGxSZXF1ZXN0NjA4MzM0MDk2 | 1291 | Update docs: explain allow_download setting | louispotok 5413548 | closed | 0 | 2 | 2021-04-03T05:28:33Z | 2021-06-05T19:48:51Z | 2021-06-05T19:48:51Z | CONTRIBUTOR | simonw/datasette/pulls/1291 | This fixes one possible source of confusion seen in #502 and clarifies when database downloads will be shown and allowed. | datasette 107914493 | pull | {"url": "https://api.github.com/repos/simonw/datasette/issues/1291/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 0 | |||||
853672224 | MDU6SXNzdWU4NTM2NzIyMjQ= | 1294 | "You can check out any time you like. But you can never leave!" | mroswell 192568 | open | 0 | 0 | 2021-04-08T17:02:15Z | 2021-04-08T18:35:50Z | CONTRIBUTOR | (Feel free to rename this one.) - The column gear lets you "Show not-blank rows." Then it places a parameter in the URL, which a web developer would notice, but a lot of users won't notice, or know to delete it. Would be good to toggle "Show not-blank rows" with "Show all rows." (Also would be quite helpful to have a "Show blank rows | Show all rows" option) - The column gear lets you "Sort ascending" and "Sort descending" but then you're stuck with some sort of sorted version thereafter, unless you know to sort the ID column, or to remove the full _sort parameter and its value in the URL. Would be good to offer a "Remove sort" option in the gear. - These requests are in the same camp as: https://github.com/simonw/datasette-vega/issues/36 - I suspect there are other url parameter instances where similar analysis would be helpful, but the three above are the use cases I've run across. UPDATE: - It would be helpful to have a "Previous page" available for all but the first table page. | datasette 107914493 | issue | {"url": "https://api.github.com/repos/simonw/datasette/issues/1294/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issues] ( [id] INTEGER PRIMARY KEY, [node_id] TEXT, [number] INTEGER, [title] TEXT, [user] INTEGER REFERENCES [users]([id]), [state] TEXT, [locked] INTEGER, [assignee] INTEGER REFERENCES [users]([id]), [milestone] INTEGER REFERENCES [milestones]([id]), [comments] INTEGER, [created_at] TEXT, [updated_at] TEXT, [closed_at] TEXT, [author_association] TEXT, [pull_request] TEXT, [body] TEXT, [repo] INTEGER REFERENCES [repos]([id]), [type] TEXT , [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT); CREATE INDEX [idx_issues_repo] ON [issues] ([repo]); CREATE INDEX [idx_issues_milestone] ON [issues] ([milestone]); CREATE INDEX [idx_issues_assignee] ON [issues] ([assignee]); CREATE INDEX [idx_issues_user] ON [issues] ([user]);