github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706834800 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | 706834800 | MDEyOklzc3VlQ29tbWVudDcwNjgzNDgwMA== | 9599 | 2020-10-12T03:24:57Z | 2020-10-16T20:16:28Z | MEMBER | Here's my first attempt at a plugin for this: ```python from datasette import hookimpl import jinja2 START = "<en-note" END = "</en-note>" TEMPLATE = """ <div style="max-width: 500px; white-space: normal; overflow-wrap: break-word;">{}</div> """.strip() EN_MEDIA_SCRIPT = """ Array.from(document.querySelectorAll('en-media')).forEach(el => { let hash = el.getAttribute('hash'); let type = el.getAttribute('type'); let path = `/evernote/resources_data/${hash}.json?_shape=array`; fetch(path).then(r => r.json()).then(rows => { let b64 = rows[0].data.encoded; let data = `data:${type};base64,${b64}`; el.innerHTML = `<img style="max-width: 300px" src="${data}">`; }); }); """ @hookimpl def render_cell(value, table): if not table: # Don't render content from arbitrary SQL queries, could be XSS hole return if not value or not isinstance(value, str): return value = value.strip() if value.startswith(START) and value.endswith(END): trimmed = value[len(START) : -len(END)] trimmed = trimmed.split(">", 1)[1] # Replace those horrible double newlines trimmed = trimmed.replace("<div><br /></div>", "<br>") return jinja2.Markup(TEMPLATE.format(trimmed)) @hookimpl def extra_body_script(): return EN_MEDIA_SCRIPT ``` It works! It does however demonstrate that Evernote's "clip this webpage" feature means there is a LOT of weird HTML that can get into a note. It looks like they've filtered out the scripts but I wouldn't bet on it - they certainly don't filter out many of the inline styles. So running Bleach is almost certainly a good idea. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 718938889 | |
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705624 | https://api.github.com/repos/simonw/sqlite-utils/issues/186 | 709705624 | MDEyOklzc3VlQ29tbWVudDcwOTcwNTYyNA== | 9599 | 2020-10-16T03:14:39Z | 2020-10-16T03:14:39Z | OWNER | How should this work with extractions covering multiple columns? If there's a single column then it makes sense that a `null` value would not be extracted into the lookup table, but would instead become stay as `null`. For a multiple column extraction, provided at least one of those columns is not null It should map to a record in the lookup table. Only if ALL of the extracted columns are null should the lookup value stay null. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 722816436 | |
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705885 | https://api.github.com/repos/simonw/sqlite-utils/issues/186 | 709705885 | MDEyOklzc3VlQ29tbWVudDcwOTcwNTg4NQ== | 9599 | 2020-10-16T03:15:39Z | 2020-10-16T03:15:39Z | OWNER | The alternative solution here would be that a single `null` value DOES get extracted. To implement this I would need to add some logic that uses `is null` instead of `=`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 722816436 | |
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706065 | https://api.github.com/repos/simonw/sqlite-utils/issues/186 | 709706065 | MDEyOklzc3VlQ29tbWVudDcwOTcwNjA2NQ== | 9599 | 2020-10-16T03:16:22Z | 2020-10-16T03:16:22Z | OWNER | Either way I think I'm going to need to add some SQL which uses `where a = b or (a is null and b is null)`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 722816436 | |
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706260 | https://api.github.com/repos/simonw/sqlite-utils/issues/186 | 709706260 | MDEyOklzc3VlQ29tbWVudDcwOTcwNjI2MA== | 9599 | 2020-10-16T03:17:02Z | 2020-10-16T03:17:17Z | OWNER | Actually I think this should be an option to `.extract()` which controls if nulls are extracted or left alone. Maybe called `extract_nulls=True/False`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 722816436 | |
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710178871 | https://api.github.com/repos/simonw/sqlite-utils/issues/182 | 710178871 | MDEyOklzc3VlQ29tbWVudDcxMDE3ODg3MQ== | 9599 | 2020-10-16T16:27:39Z | 2020-10-16T16:28:14Z | OWNER | The file is opened for me by `click.File()`, which also handles things like `-` for stdin. But i neee to be able to switch the encoding used to read from that based on the `--encoding` option. I think the way to do that is to open the file in binary mode and then wrap it in a codec reader: ```python fp = codecs.getreader(encoding)(fp) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 711649325 | |
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-710198162 | https://api.github.com/repos/simonw/sqlite-utils/issues/186 | 710198162 | MDEyOklzc3VlQ29tbWVudDcxMDE5ODE2Mg== | 9599 | 2020-10-16T16:41:00Z | 2020-10-16T16:41:00Z | OWNER | Failing test: ```python def test_extract_null_values(fresh_db): fresh_db["species"].insert({"id": 1, "species": "Wolf"}, pk="id") fresh_db["individuals"].insert_all( [ {"id": 10, "name": "Terriana", "species": "Fox"}, {"id": 11, "name": "Spenidorm", "species": None}, {"id": 12, "name": "Grantheim", "species": "Wolf"}, {"id": 13, "name": "Turnutopia", "species": None}, {"id": 14, "name": "Wargal", "species": "Wolf"}, ], pk="id", ) fresh_db["individuals"].extract("species") assert fresh_db["species"].schema == ( "CREATE TABLE [species] (\n" " [id] INTEGER PRIMARY KEY,\n" " [species] TEXT\n" ")" ) assert fresh_db["individuals"].schema == ( 'CREATE TABLE "individuals" (\n' " [id] INTEGER PRIMARY KEY,\n" " [name] TEXT,\n" " [species_id] INTEGER,\n" " FOREIGN KEY(species_id) REFERENCES species(id)\n" ")" ) assert list(fresh_db["species"].rows) == [ {"id": 1, "species": "Wolf"}, {"id": 2, "species": "Fox"}, ] assert list(fresh_db["individuals"].rows) == [ {"id": 10, "name": "Terriana", "species_id": 2}, {"id": 11, "name": "Spenidorm", "species_id": None}, {"id": 12, "name": "Grantheim", "species_id": 1}, {"id": 13, "name": "Turnutopia", "species_id": None}, {"id": 14, "name": "Wargal", "species_id": 1}, ] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 722816436 | |
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710258736 | https://api.github.com/repos/simonw/sqlite-utils/issues/182 | 710258736 | MDEyOklzc3VlQ29tbWVudDcxMDI1ODczNg== | 9599 | 2020-10-16T17:20:41Z | 2020-10-16T17:20:41Z | OWNER | Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#inserting-csv-or-tsv-data | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 711649325 | |
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710346830 | https://api.github.com/repos/simonw/sqlite-utils/issues/49 | 710346830 | MDEyOklzc3VlQ29tbWVudDcxMDM0NjgzMA== | 9599 | 2020-10-16T18:08:52Z | 2020-10-16T18:09:21Z | OWNER | The new `.extract()` method can handle multiple columns: https://github.com/simonw/sqlite-utils/blob/2c541fac352632e23e40b0d21e3f233f7a744a57/tests/test_extract.py#L70-L87 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 472115381 | |
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710359724 | https://api.github.com/repos/simonw/sqlite-utils/issues/49 | 710359724 | MDEyOklzc3VlQ29tbWVudDcxMDM1OTcyNA== | 9599 | 2020-10-16T18:15:31Z | 2020-10-16T18:15:31Z | OWNER | Using a tuple would work: ```python fresh_db.table("tree", extracts=[("common_name", "latin_name")]) ``` Or to define a custom name: ```python fresh_db.table("tree", extracts={("common_name", "latin_name"): "names"}) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 472115381 | |
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710363789 | https://api.github.com/repos/simonw/sqlite-utils/issues/49 | 710363789 | MDEyOklzc3VlQ29tbWVudDcxMDM2Mzc4OQ== | 9599 | 2020-10-16T18:18:05Z | 2020-10-16T18:18:05Z | OWNER | I wonder if there's value in extending the `extracts=` option at all given the existence of `table.extract()`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 472115381 | |
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710364942 | https://api.github.com/repos/simonw/sqlite-utils/issues/49 | 710364942 | MDEyOklzc3VlQ29tbWVudDcxMDM2NDk0Mg== | 9599 | 2020-10-16T18:18:48Z | 2020-10-16T18:18:48Z | OWNER | I think there is. It's a nice existing feature, and I don't think adding tuple support to it would be a huge lift. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 472115381 | |
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710390915 | https://api.github.com/repos/simonw/sqlite-utils/issues/49 | 710390915 | MDEyOklzc3VlQ29tbWVudDcxMDM5MDkxNQ== | 9599 | 2020-10-16T18:34:26Z | 2020-10-16T18:34:50Z | OWNER | Here's the most complex example of `.extracts()`: ```python db["Trees"].extract( ["CommonName", "LatinName"], table="Species", fk_column="species_id", rename={"CommonName": "name", "LatinName": "latin"} ) ``` Resulting in: ```sql CREATE TABLE [Species] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [latin] TEXT ) ``` From https://sqlite-utils.readthedocs.io/en/stable/python-api.html#extracting-columns-into-a-separate-table | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 472115381 | |
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550 | https://api.github.com/repos/simonw/sqlite-utils/issues/49 | 710393550 | MDEyOklzc3VlQ29tbWVudDcxMDM5MzU1MA== | 9599 | 2020-10-16T18:35:57Z | 2020-10-16T18:36:39Z | OWNER | If I want to support that most complicated example, I think the option to pass a `Extracts()` object to `extracts=` is the best way to do it: ```python fresh_db.table("tree", extracts=[Extract( columns=("CommonName", "LatinName"), table="Species", fk_column="species_id", rename={"CommonName": "name", "LatinName": "latin"} )]) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 472115381 | |
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710395444 | https://api.github.com/repos/simonw/sqlite-utils/issues/49 | 710395444 | MDEyOklzc3VlQ29tbWVudDcxMDM5NTQ0NA== | 9599 | 2020-10-16T18:37:10Z | 2020-10-16T18:37:10Z | OWNER | But this begins to feel too complicated, given that `table.extract()` can already be used to achieve the same thing. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 472115381 | |
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710397574 | https://api.github.com/repos/simonw/sqlite-utils/issues/49 | 710397574 | MDEyOklzc3VlQ29tbWVudDcxMDM5NzU3NA== | 9599 | 2020-10-16T18:38:21Z | 2020-10-16T18:38:21Z | OWNER | I'm not going to implement this. I'll leave `extract=...` as it is right now, suitable for quick simple single-column operations on input, but if users want to do something more complicated involving multiple columns they should use the `table.extract()` method after the initial insert instead. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 472115381 | |
https://github.com/simonw/sqlite-utils/issues/58#issuecomment-710399593 | https://api.github.com/repos/simonw/sqlite-utils/issues/58 | 710399593 | MDEyOklzc3VlQ29tbWVudDcxMDM5OTU5Mw== | 9599 | 2020-10-16T18:39:31Z | 2020-10-16T18:39:31Z | OWNER | I don't think this is valuable enough to justify adding to the library - especially since you can execute FTS search against views by joining to an FTS table built against an underlying table. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 488293926 | |
https://github.com/simonw/sqlite-utils/issues/48#issuecomment-710402331 | https://api.github.com/repos/simonw/sqlite-utils/issues/48 | 710402331 | MDEyOklzc3VlQ29tbWVudDcxMDQwMjMzMQ== | 9599 | 2020-10-16T18:41:06Z | 2020-10-16T18:41:06Z | OWNER | I could use this demo from JupyterCon 2020 https://gist.github.com/simonw/656c21b5800d5e4624dec9930f00e093 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 471818939 | |
https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710405658 | https://api.github.com/repos/simonw/sqlite-utils/issues/69 | 710405658 | MDEyOklzc3VlQ29tbWVudDcxMDQwNTY1OA== | 9599 | 2020-10-16T18:42:48Z | 2020-10-16T18:42:48Z | OWNER | Did some work on this for #134 in 7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d I still need to add `--load-extension` to other CLI methods, see #137. Closing this issue in favour of that one. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 534507142 | |
https://github.com/simonw/sqlite-utils/issues/136#issuecomment-710428802 | https://api.github.com/repos/simonw/sqlite-utils/issues/136 | 710428802 | MDEyOklzc3VlQ29tbWVudDcxMDQyODgwMg== | 9599 | 2020-10-16T18:56:47Z | 2020-10-16T18:56:47Z | OWNER | To keep the code cleaner, I'm tempted to support this instead: --load-extension=spatialite Where `spatialite` is a special shortcut value that triggers a search for that module in known locations. Users could still load a module in a file called `spatialite` in the current directory using: --load-extension=./spatialite In fact, `--load-extension=spatialite` could handle that case too by always checking for a file called `spatialite` before attempting to search for it in known locations. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 683812642 | |
https://github.com/simonw/sqlite-utils/issues/187#issuecomment-710440853 | https://api.github.com/repos/simonw/sqlite-utils/issues/187 | 710440853 | MDEyOklzc3VlQ29tbWVudDcxMDQ0MDg1Mw== | 9599 | 2020-10-16T19:04:19Z | 2020-10-16T19:04:19Z | OWNER | I split this off from #136. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 723460107 | |
https://github.com/simonw/sqlite-utils/issues/187#issuecomment-710456981 | https://api.github.com/repos/simonw/sqlite-utils/issues/187 | 710456981 | MDEyOklzc3VlQ29tbWVudDcxMDQ1Njk4MQ== | 9599 | 2020-10-16T19:15:13Z | 2020-10-16T19:15:13Z | OWNER | This is a duplicate of #79. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 723460107 | |
https://github.com/simonw/sqlite-utils/issues/89#issuecomment-710460242 | https://api.github.com/repos/simonw/sqlite-utils/issues/89 | 710460242 | MDEyOklzc3VlQ29tbWVudDcxMDQ2MDI0Mg== | 9599 | 2020-10-16T19:17:27Z | 2020-10-16T19:17:50Z | OWNER | I came up with potential syntax for that here: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550 - based on how `table.extract(...)` works: ```python fresh_db.table("tree", extracts=[Extract( columns=("CommonName", "LatinName"), table="Species", fk_column="species_id", rename={"CommonName": "name", "LatinName": "latin"} )]) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 573578548 | |
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710461468 | https://api.github.com/repos/simonw/sqlite-utils/issues/49 | 710461468 | MDEyOklzc3VlQ29tbWVudDcxMDQ2MTQ2OA== | 9599 | 2020-10-16T19:18:19Z | 2020-10-16T19:18:19Z | OWNER | Reconsidering: #89 was a feature request that relates to this, so maybe this is worth implementing after all. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 472115381 | |
https://github.com/simonw/datasette/issues/468#issuecomment-710693818 | https://api.github.com/repos/simonw/datasette/issues/468 | 710693818 | MDEyOklzc3VlQ29tbWVudDcxMDY5MzgxOA== | 9599 | 2020-10-16T23:20:22Z | 2020-10-16T23:20:22Z | OWNER | This is a duplicate of #461. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 444746021 | |
https://github.com/simonw/datasette/issues/461#issuecomment-710693927 | https://api.github.com/repos/simonw/datasette/issues/461 | 710693927 | MDEyOklzc3VlQ29tbWVudDcxMDY5MzkyNw== | 9599 | 2020-10-16T23:20:43Z | 2020-10-16T23:20:43Z | OWNER | Related: redesign homepage entirely #991. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 443021509 | |
https://github.com/simonw/datasette/issues/991#issuecomment-710694144 | https://api.github.com/repos/simonw/datasette/issues/991 | 710694144 | MDEyOklzc3VlQ29tbWVudDcxMDY5NDE0NA== | 9599 | 2020-10-16T23:21:41Z | 2020-10-16T23:21:41Z | OWNER | Relevant: https://github.com/simonw/datasette/discussions/1021 asks about controlling the order of databases on that page that have been loaded using configuration directory mode. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 714377268 | |
https://github.com/simonw/datasette/issues/991#issuecomment-710694607 | https://api.github.com/repos/simonw/datasette/issues/991 | 710694607 | MDEyOklzc3VlQ29tbWVudDcxMDY5NDYwNw== | 9599 | 2020-10-16T23:23:34Z | 2020-10-16T23:23:34Z | OWNER | Thinking more about pagination and search from #461: if the job of the homepage is to showcase the data that is available in the instance - data that is mostly in tables - maybe it's the tables themselves (and the ability to paginate and search through them) that becomes key to the design of the page. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 714377268 | |
https://github.com/simonw/datasette/issues/991#issuecomment-710694711 | https://api.github.com/repos/simonw/datasette/issues/991 | 710694711 | MDEyOklzc3VlQ29tbWVudDcxMDY5NDcxMQ== | 9599 | 2020-10-16T23:24:03Z | 2020-10-16T23:24:03Z | OWNER | I'm really interested in exploring how this page could work with hundreds of database files attached and thousands of total tables. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 714377268 |