html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/sqlite-utils/issues/186#issuecomment-897588624,https://api.github.com/repos/simonw/sqlite-utils/issues/186,897588624,IC_kwDOCGYnMM41gB2Q,9308268,rayvoelker,2021-08-12T12:13:25Z,2021-08-12T12:13:25Z,NONE,"I think I ran into an issue that's perhaps related with `extract()` I have a case where I want to create a lookup table for all the related title data where there are possibly multiple null values in the related columns .... ```python3 sql = """"""\ INSERT INTO ""circulation_info"" (""item_id"", ""bib_title"", ""bib_creator"", ""bib_format"", ""bib_pub_year"", ""checkout_date"") VALUES (1, ""title one"", ""creator one"", ""Book"", 2018, ""2021-08-12 00:01""), (2, ""title two"", ""creator one"", ""Book"", 2019, ""2021-08-12 00:02""), (3, ""title three"", NULL, ""DVD"", 2020, ""2021-08-12 00:03""), (4, ""title four"", NULL, ""DVD"", NULL, ""2021-08-12 00:04""), (5, ""title five"", NULL, ""DVD"", NULL, ""2021-08-12 00:05"") """""" with sqlite3.connect('test_bib.db') as con: con.execute(sql) ``` when I run the `extract()` method ... ```python3 db[""circulation_info""].extract( [ ""bib_title"", ""bib_creator"", ""bib_format"" ], table=""bib_info"", fk_column=""bib_info_id"" ) db = sqlite_utils.Database(""test_bib.db"") for row in db[""circulation_info""].rows: print(row) print(""\n---\n"") for row in db[""bib_info""].rows: print(row) ``` results in this .. ``` {'id': 1, 'item_id': 1, 'bib_info_id': 1, 'bib_pub_year': 2018, 'checkout_date': '2021-08-12 00:01'} {'id': 2, 'item_id': 2, 'bib_info_id': 2, 'bib_pub_year': 2019, 'checkout_date': '2021-08-12 00:02'} {'id': 3, 'item_id': 3, 'bib_info_id': None, 'bib_pub_year': 2020, 'checkout_date': '2021-08-12 00:03'} {'id': 4, 'item_id': 4, 'bib_info_id': None, 'bib_pub_year': None, 'checkout_date': '2021-08-12 00:04'} {'id': 5, 'item_id': 5, 'bib_info_id': None, 'bib_pub_year': None, 'checkout_date': '2021-08-12 00:05'} --- {'id': 1, 'bib_title': 'title one', 'bib_creator': 'creator one', 'bib_format': 'Book'} {'id': 2, 'bib_title': 'title two', 'bib_creator': 'creator one', 'bib_format': 'Book'} {'id': 3, 'bib_title': 'title three', 'bib_creator': None, 'bib_format': 'DVD'} {'id': 4, 'bib_title': 'title four', 'bib_creator': None, 'bib_format': 'DVD'} {'id': 5, 'bib_title': 'title five', 'bib_creator': None, 'bib_format': 'DVD'} ``` Seems like it's correctly generating the row data for those lookups, but it's not correctly updating the foreign key back to the primary table? Looks like it just results in a `NULL` value in that original table. Any ideas on why? Thanks again!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values, https://github.com/simonw/sqlite-utils/issues/186#issuecomment-897600677,https://api.github.com/repos/simonw/sqlite-utils/issues/186,897600677,IC_kwDOCGYnMM41gEyl,9308268,rayvoelker,2021-08-12T12:32:14Z,2021-08-12T12:32:14Z,NONE,"Actually, I forgot to include the `bib_pub_year` in the extract ... But also, I tried again with empty string values instead of `NULL` values and it seems to place the foreign key properly / correctly... ```python3 sql = """"""\ INSERT INTO ""circulation_info"" (""item_id"", ""bib_title"", ""bib_creator"", ""bib_format"", ""bib_pub_year"", ""checkout_date"") VALUES (1, ""title one"", ""creator one"", ""Book"", 2018, ""2021-08-12 00:01""), (2, ""title two"", ""creator one"", ""Book"", 2019, ""2021-08-12 00:02""), (3, ""title three"", """", ""DVD"", 2020, ""2021-08-12 00:03""), (4, ""title four"", """", ""DVD"", """", ""2021-08-12 00:04""), (5, ""title five"", """", ""DVD"", """", ""2021-08-12 00:05"") """""" with sqlite3.connect('test_bib_2.db') as con: con.execute(sql) ``` ```python3 db[""circulation_info""].extract( [ ""bib_title"", ""bib_creator"", ""bib_format"", ""bib_pub_year"" ], table=""bib_info"", fk_column=""bib_info_id"" ) ``` ``` {'id': 1, 'item_id': 1, 'bib_info_id': 1, 'bib_pub_year': 2018, 'checkout_date': '2021-08-12 00:01'} {'id': 2, 'item_id': 2, 'bib_info_id': 2, 'bib_pub_year': 2019, 'checkout_date': '2021-08-12 00:02'} {'id': 3, 'item_id': 3, 'bib_info_id': 3, 'bib_pub_year': 2020, 'checkout_date': '2021-08-12 00:03'} {'id': 4, 'item_id': 4, 'bib_info_id': 4, 'bib_pub_year': '', 'checkout_date': '2021-08-12 00:04'} {'id': 5, 'item_id': 5, 'bib_info_id': 5, 'bib_pub_year': '', 'checkout_date': '2021-08-12 00:05'} --- {'id': 1, 'bib_title': 'title one', 'bib_creator': 'creator one', 'bib_format': 'Book'} {'id': 2, 'bib_title': 'title two', 'bib_creator': 'creator one', 'bib_format': 'Book'} {'id': 3, 'bib_title': 'title three', 'bib_creator': '', 'bib_format': 'DVD'} {'id': 4, 'bib_title': 'title four', 'bib_creator': '', 'bib_format': 'DVD'} {'id': 5, 'bib_title': 'title five', 'bib_creator': '', 'bib_format': 'DVD'} ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values, https://github.com/simonw/datasette/issues/1429#issuecomment-897960049,https://api.github.com/repos/simonw/datasette/issues/1429,897960049,IC_kwDOBm6k_c41hchx,9599,simonw,2021-08-12T20:53:04Z,2021-08-12T20:53:04Z,OWNER,"Maybe something like this: > [Next page](#) - 100 per page ([show 1,000 per page](#))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969548935,UI for setting `?_size=max` on table page, https://github.com/simonw/datasette/issues/942#issuecomment-897996296,https://api.github.com/repos/simonw/datasette/issues/942,897996296,IC_kwDOBm6k_c41hlYI,9599,simonw,2021-08-12T22:01:36Z,2021-08-12T22:01:36Z,OWNER,"I'm going with `""columns"": {""name-of-column"": ""description-of-column""}`. If I decide to make `""col""` and `""nocol""` available in metadata I'll use those as the keys in the metadata, for consistency with the existing query string parameters. I'm OK with having both `""columns"": ...` and `""col"": ...` keys in the metadata, even though they could be a tiny bit confusing without the documentation.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898021895,https://api.github.com/repos/simonw/datasette/issues/942,898021895,IC_kwDOBm6k_c41hroH,9599,simonw,2021-08-12T22:51:36Z,2021-08-12T22:51:36Z,OWNER,"Prototype: ```diff diff --git a/datasette/static/app.css b/datasette/static/app.css index c6be1e9..5ca64cb 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -784,9 +784,14 @@ svg.dropdown-menu-icon { font-size: 0.7em; color: #666; margin: 0; - padding: 0; padding: 4px 8px 4px 8px; } +.dropdown-menu .dropdown-column-description { + margin: 0; + color: #666; + padding: 4px 8px 4px 8px; + max-width: 20em; +} .dropdown-menu li { border-bottom: 1px solid #ccc; } diff --git a/datasette/static/table.js b/datasette/static/table.js index 991346d..a903112 100644 --- a/datasette/static/table.js +++ b/datasette/static/table.js @@ -9,6 +9,7 @@ var DROPDOWN_HTML = `
  • Show not-blank rows

    • +

    `; var DROPDOWN_ICON_SVG = ` @@ -166,6 +167,14 @@ var DROPDOWN_ICON_SVG = `
    {% for column in display_columns %} - + {% if not column.sortable %} {{ column.name }} {% else %} diff --git a/datasette/views/table.py b/datasette/views/table.py index 456d806..486a613 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -125,6 +125,7 @@ class RowTableShared(DataView): """"""Returns columns, rows for specified table - including fancy foreign key treatment"""""" db = self.ds.databases[database] table_metadata = self.ds.table_metadata(database, table) + column_descriptions = table_metadata.get(""columns"") or {} column_details = {col.name: col for col in await db.table_column_details(table)} sortable_columns = await self.sortable_columns_for_table(database, table, True) pks = await db.primary_keys(table) @@ -147,6 +148,7 @@ class RowTableShared(DataView): ""is_pk"": r[0] in pks_for_display, ""type"": type_, ""notnull"": notnull, + ""description"": column_descriptions.get(r[0]), } ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898022235,https://api.github.com/repos/simonw/datasette/issues/942,898022235,IC_kwDOBm6k_c41hrtb,9599,simonw,2021-08-12T22:52:23Z,2021-08-12T22:52:23Z,OWNER,I like this. Need to solve for mobile though where the cog menu isn't visible - I think I'll do that with a definition list at the top of the page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898032118,https://api.github.com/repos/simonw/datasette/issues/942,898032118,IC_kwDOBm6k_c41huH2,596279,zaneselvans,2021-08-12T23:12:00Z,2021-08-12T23:12:00Z,NONE,"This looks awesome. We'll definitely make extensive use of this feature! On Thu, Aug 12, 2021 at 5:52 PM Simon Willison ***@***.***> wrote: > I like this. Need to solve for mobile though where the cog menu isn't > visible - I think I'll do that with a definition list at the top of the > page. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > , > or unsubscribe > > . > Triage notifications on the go with GitHub Mobile for iOS > > or Android > > . > -- Zane A. Selvans, PhD Chief Data Wrangler Catalyst Cooperative https://catalyst.coop ***@***.*** Signal/WhatsApp/SMS: +1 720 443 1363 Twitter: @ZaneSelvans PGP : 0x64F7B56F3A127B04 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898037456,https://api.github.com/repos/simonw/datasette/issues/942,898037456,IC_kwDOBm6k_c41hvbQ,9599,simonw,2021-08-12T23:23:34Z,2021-08-12T23:23:34Z,OWNER,"Prototype with a `
    `: ```diff diff --git a/datasette/static/app.css b/datasette/static/app.css index c6be1e9..bf068fd 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -836,6 +841,16 @@ svg.dropdown-menu-icon { background-repeat: no-repeat; } +dl.column-descriptions dt { + font-weight: bold; +} +dl.column-descriptions dd { + padding-left: 1.5em; + white-space: pre-wrap; + line-height: 1.1em; + color: #666; +} + .anim-scale-in { animation-name: scale-in; animation-duration: 0.15s; diff --git a/datasette/templates/table.html b/datasette/templates/table.html index 211352b..466e8a4 100644 --- a/datasette/templates/table.html +++ b/datasette/templates/table.html @@ -51,6 +51,14 @@ {% block description_source_license %}{% include ""_description_source_license.html"" %}{% endblock %} +{% if metadata.columns %} +
    + {% for column_name, column_description in metadata.columns.items() %} +
    {{ column_name }}
    {{ column_description }}
    + {% endfor %} +
    +{% endif %} + {% if filtered_table_rows_count or human_description_en %}

    {% if filtered_table_rows_count or filtered_table_rows_count == 0 %}{{ ""{:,}"".format(filtered_table_rows_count) }} row{% if filtered_table_rows_count == 1 %}{% else %}s{% endif %}{% endif %} {% if human_description_en %}{{ human_description_en }}{% endif %} ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898037650,https://api.github.com/repos/simonw/datasette/issues/942,898037650,IC_kwDOBm6k_c41hveS,9599,simonw,2021-08-12T23:23:54Z,2021-08-12T23:23:54Z,OWNER,I like this enough that I'm going to ship it as an alpha and try it out on a couple of live projects.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/pull/1430#issuecomment-898043575,https://api.github.com/repos/simonw/datasette/issues/1430,898043575,IC_kwDOBm6k_c41hw63,22429695,codecov[bot],2021-08-12T23:39:36Z,2021-08-12T23:49:51Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1430](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9419947) into [main](https://codecov.io/gh/simonw/datasette/commit/b1fed48a95516ae84c0f020582303ab50ab817e2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b1fed48) will **increase** coverage by `0.00%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1430/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1430 +/- ## ======================================= Coverage 91.71% 91.71% ======================================= Files 34 34 Lines 4417 4418 +1 ======================================= + Hits 4051 4052 +1 Misses 366 366 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1430/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <100.00%> (+<0.01%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b1fed48...9419947](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969758038,Column metadata, https://github.com/simonw/datasette/issues/942#issuecomment-898050457,https://api.github.com/repos/simonw/datasette/issues/942,898050457,IC_kwDOBm6k_c41hymZ,9599,simonw,2021-08-12T23:59:53Z,2021-08-12T23:59:53Z,OWNER,"Documentation: https://docs.datasette.io/en/latest/metadata.html#column-descriptions Live demo: https://latest.datasette.io/fixtures/roadside_attractions ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,