issue_comments
525 rows where author_association = "MEMBER" sorted by id descending
This data as json, CSV (advanced)
id ▲ | html_url | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
735484186 | https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-735484186 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51 | MDEyOklzc3VlQ29tbWVudDczNTQ4NDE4Ng== | simonw 9599 | 2020-11-30T00:29:31Z | 2020-11-30T00:29:31Z | MEMBER | This just caused a failure in deploying the demo: https://github.com/dogsheep/github-to-sqlite/runs/1471304407?check_suite_focus=true ``` File "/opt/hostedtoolcache/Python/3.8.6/x64/bin/github-to-sqlite", line 33, in <module> sys.exit(load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')()) File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 142, in issue_comments for comment in utils.fetch_issue_comments(repo, token, issue): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 380, in fetch_issue_comments for comments in paginate(url, headers): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 472, in paginate raise GitHubError.from_response(response) github_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403) Error: Process completed with exit code 1. ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite should handle rate limits better 703246031 | |
735483820 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483820 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDczNTQ4MzgyMA== | simonw 9599 | 2020-11-30T00:27:47Z | 2020-11-30T00:27:47Z | MEMBER | So it looks like anything that pulls reviews needs to pull each review, then for each one pull the comments. I'm going to consider this blocked on smarter rate limit handling in #51. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: pull request reviews and comments 664485022 | |
735483604 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483604 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDczNTQ4MzYwNA== | simonw 9599 | 2020-11-30T00:26:50Z | 2020-11-30T00:26:50Z | MEMBER | It seems like there's a lot missing from that - those aren't particularly interesting given the data that is returned. From the docs at https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#reviews it looks like each review consists of multiple comments, and the comments are where the useful material is - https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#list-comments-for-a-pull-request-review `github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews/503368921/comments --accept 'application/vnd.github.v3+json'` ```json [ { "id": 500603838, "node_id": "MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwMzgzOA==", "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500603838", "pull_request_review_id": 503368921, "diff_hunk": "@@ -0,0 +1,370 @@\n+[\n+ {\n+ \"url\": \"https://api.github.com/repos/simonw/datasette/pulls/571\",\n+ \"id\": 313384926,\n+ \"node_id\": \"MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2\",\n+ \"html_url\": \"https://github.com/simonw/datasette/pull/571\",\n+ \"diff_url\": \"https://github.com/simonw/datasette/pull/571.diff\",\n+ \"patch_url\": \"https://github.com/simonw/datasette/pull/571.patch\",\n+ \"issue_url\": \"https://api.github.com/repos/simonw/datasette/issues/571\",\n+ \"number\": 571,\n+ \"state\": \"closed\",\n+ \"locked\": false,\n+ \"title\": \"detect_fts now works with alternative table escaping\",\n+ \"user\": {\n+ \"login\": \"simonw\",\n+ \"id\": 9599,\n+ \"node_id\": \"MDQ6VXNlcjk1OTk=\",\n+ \"avatar_url\": \"https://avatars0.githubusercontent.com/u/9599?v=4\",\n+ \"gravatar_id\": \"\",\n+ \"url\": \"https://api.github.com/users/simonw\",\n+ \"html_url\": \"https://github.com/simonw\",\n+ \"followers_url\": \"https://api.github.com/users/simonw/followers\",\n+ \"following_ur… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: pull request reviews and comments 664485022 | |
735482546 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482546 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDczNTQ4MjU0Ng== | simonw 9599 | 2020-11-30T00:22:02Z | 2020-11-30T00:22:02Z | MEMBER | As for reviews... here's the output of `github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews --accept 'application/vnd.github.v3+json'` ```json [ { "id": 503368921, "node_id": "MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTAzMzY4OTIx", "user": { "login": "simonw", "id": 9599, "node_id": "MDQ6VXNlcjk1OTk=", "avatar_url": "https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4", "gravatar_id": "", "url": "https://api.github.com/users/simonw", "html_url": "https://github.com/simonw", "followers_url": "https://api.github.com/users/simonw/followers", "following_url": "https://api.github.com/users/simonw/following{/other_user}", "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}", "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/simonw/subscriptions", "organizations_url": "https://api.github.com/users/simonw/orgs", "repos_url": "https://api.github.com/users/simonw/repos", "events_url": "https://api.github.com/users/simonw/events{/privacy}", "received_events_url": "https://api.github.com/users/simonw/received_events", "type": "User", "site_admin": false }, "body": "", "state": "CHANGES_REQUESTED", "html_url": "https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-503368921", "pull_request_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48", "author_association": "MEMBER", "_links": { "html": { "href": "https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-503368921" }, "pull_request": { "href": "https://api.github.c… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: pull request reviews and comments 664485022 | |
735482187 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482187 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDczNTQ4MjE4Nw== | simonw 9599 | 2020-11-30T00:20:11Z | 2020-11-30T00:20:11Z | MEMBER | Pull request are now added, thanks to @adamjonas. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: pull request reviews and comments 664485022 | |
735465708 | https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735465708 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54 | MDEyOklzc3VlQ29tbWVudDczNTQ2NTcwOA== | simonw 9599 | 2020-11-29T22:08:46Z | 2020-11-29T22:08:46Z | MEMBER | Demo: - https://github-to-sqlite.dogsheep.net/github/steps?_facet=repo - https://github-to-sqlite.dogsheep.net/github/workflows - https://github-to-sqlite.dogsheep.net/github/jobs | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite workflows command 753026003 | |
735464493 | https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735464493 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54 | MDEyOklzc3VlQ29tbWVudDczNTQ2NDQ5Mw== | simonw 9599 | 2020-11-29T21:57:32Z | 2020-11-29T21:57:32Z | MEMBER | `$ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite workflows command 753026003 | |
735464438 | https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735464438 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54 | MDEyOklzc3VlQ29tbWVudDczNTQ2NDQzOA== | simonw 9599 | 2020-11-29T21:57:08Z | 2020-11-29T21:57:08Z | MEMBER | Inspired by this tweet from Michael Heap https://twitter.com/mheap/status/1333108608817631238 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | github-to-sqlite workflows command 753026003 | |
727692413 | https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-727692413 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcyNzY5MjQxMw== | simonw 9599 | 2020-11-16T02:15:22Z | 2020-11-16T02:15:22Z | MEMBER | Thanks, I'll look into this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Error thrown: sqlite3.OperationalError: table users has no column named lastName 743400216 | |
712266834 | https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-712266834 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29 | MDEyOklzc3VlQ29tbWVudDcxMjI2NjgzNA== | simonw 9599 | 2020-10-19T16:01:23Z | 2020-10-19T16:01:23Z | MEMBER | Might just be a documented pattern for how to configure this in YAML templates. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add search highlighting snippets 724759588 | |
711569063 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-711569063 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDcxMTU2OTA2Mw== | simonw 9599 | 2020-10-19T05:01:29Z | 2020-10-19T05:01:29Z | MEMBER | Demo of `--accept`: github-to-sqlite get /repos/simonw/datasette/readme --accept 'application/vnd.github.VERSION.html' | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
711089647 | https://github.com/dogsheep/dogsheep-beta/issues/28#issuecomment-711089647 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/28 | MDEyOklzc3VlQ29tbWVudDcxMTA4OTY0Nw== | simonw 9599 | 2020-10-17T22:43:13Z | 2020-10-17T22:43:13Z | MEMBER | Since my personal Dogsheep uses Datasette authentication, I'm going to need to pass through cookies. https://github.com/simonw/datasette/issues/1020 will solve that in the future but for now I need to solve it explicitly. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Switch to using datasette.client 723861683 | |
711081703 | https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711081703 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcxMTA4MTcwMw== | simonw 9599 | 2020-10-17T21:18:35Z | 2020-10-17T21:18:35Z | MEMBER | OK, if you upgrade to the just-released 1.0 this should work (it worked against my Spanish export). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | export.xml file name varies with different language settings 723838331 | |
711079760 | https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711079760 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcxMTA3OTc2MA== | simonw 9599 | 2020-10-17T21:00:05Z | 2020-10-17T21:00:05Z | MEMBER | Checking for either `<!DOCTYPE HealthData` or `<HealthData` in the first 1000 bytes should do it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | export.xml file name varies with different language settings 723838331 | |
711079056 | https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711079056 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcxMTA3OTA1Ng== | simonw 9599 | 2020-10-17T20:53:00Z | 2020-10-17T20:53:00Z | MEMBER | I think the safest thing is to sniff the first few lines of the file. Those should be the same no matter the language that was used: ```xml <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE HealthData [ ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | export.xml file name varies with different language settings 723838331 | |
711078917 | https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711078917 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcxMTA3ODkxNw== | simonw 9599 | 2020-10-17T20:51:55Z | 2020-10-17T20:52:03Z | MEMBER | I switched my phone to Spanish and ran an export - I got a file called `exportar.zip`. Unzipped I still got a `apple_ health_export` folder but the root contained: ``` electrocardiograms/ export_cda.xml exportar.xml workout-routes/ ``` It looks like `export_cda.xml` does not have a translated name, so maybe I can ignore it and look for the _other_ `.xml` file in that directory. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | export.xml file name varies with different language settings 723838331 | |
711074306 | https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711074306 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcxMTA3NDMwNg== | simonw 9599 | 2020-10-17T20:16:22Z | 2020-10-17T20:16:22Z | MEMBER | The "first XML file in the root" solution is probably easier though! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | export.xml file name varies with different language settings 723838331 | |
711074031 | https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711074031 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcxMTA3NDAzMQ== | simonw 9599 | 2020-10-17T20:14:01Z | 2020-10-17T20:14:01Z | MEMBER | I'd be happy to teach the tool to look for `export.xml` or `eksport.xml` - and then expand that list to other languages. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | export.xml file name varies with different language settings 723838331 | |
707332912 | https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-707332912 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDcwNzMzMjkxMg== | simonw 9599 | 2020-10-12T20:35:06Z | 2020-10-12T20:35:06Z | MEMBER | Shipped a fix for this in [swarm-to-sqlite 0.3.2](https://github.com/dogsheep/swarm-to-sqlite/releases/tag/0.3.2). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Error thrown: table photos has no column named hasSticker 648245071 | |
706834800 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706834800 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjgzNDgwMA== | simonw 9599 | 2020-10-12T03:24:57Z | 2020-10-16T20:16:28Z | MEMBER | Here's my first attempt at a plugin for this: ```python from datasette import hookimpl import jinja2 START = "<en-note" END = "</en-note>" TEMPLATE = """ <div style="max-width: 500px; white-space: normal; overflow-wrap: break-word;">{}</div> """.strip() EN_MEDIA_SCRIPT = """ Array.from(document.querySelectorAll('en-media')).forEach(el => { let hash = el.getAttribute('hash'); let type = el.getAttribute('type'); let path = `/evernote/resources_data/${hash}.json?_shape=array`; fetch(path).then(r => r.json()).then(rows => { let b64 = rows[0].data.encoded; let data = `data:${type};base64,${b64}`; el.innerHTML = `<img style="max-width: 300px" src="${data}">`; }); }); """ @hookimpl def render_cell(value, table): if not table: # Don't render content from arbitrary SQL queries, could be XSS hole return if not value or not isinstance(value, str): return value = value.strip() if value.startswith(START) and value.endswith(END): trimmed = value[len(START) : -len(END)] trimmed = trimmed.split(">", 1)[1] # Replace those horrible double newlines trimmed = trimmed.replace("<div><br /></div>", "<br>") return jinja2.Markup(TEMPLATE.format(trimmed)) @hookimpl def extra_body_script(): return EN_MEDIA_SCRIPT ``` It works! It does however demonstrate that Evernote's "clip this webpage" feature means there is a LOT of weird HTML that can get into a note. It looks like they've filtered out the scripts but I wouldn't bet on it - they certainly don't filter out many of the inline styles. So running Bleach is almost certainly a good idea. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
706786548 | https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706786548 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDcwNjc4NjU0OA== | simonw 9599 | 2020-10-11T23:39:46Z | 2020-10-11T23:39:46Z | MEMBER | Should have used porter stemming for this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Configure FTS + add an index on the date columns 718938508 | |
706785201 | https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785201 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDcwNjc4NTIwMQ== | simonw 9599 | 2020-10-11T23:29:39Z | 2020-10-11T23:29:39Z | MEMBER | It looks to me like each of those `<item>` blocks has a number of guesses in order of confidence: ```xml <item x="215" y="190" w="187" h="39"> <t w="57">wonders,</t> <t w="55">wanders,</t> <t w="52">wonders ?</t> <t w="45">wonders</t> <t w="42">wonders.</t> </item> ``` So maybe the best approach here is to just take the first `t` element within each `item`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Better handling of OCR data 718949182 | |
706785086 | https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785086 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDcwNjc4NTA4Ng== | simonw 9599 | 2020-10-11T23:28:50Z | 2020-10-11T23:28:50Z | MEMBER | The XML for the OCR stuff is a bit weird. Currently I'm doing this to it: https://github.com/dogsheep/evernote-to-sqlite/blob/c33d7b043a45eb3e88676e5fa3ce31755199d9f8/evernote_to_sqlite/utils.py#L70-L78 This can produce some odd results, for example: > Sure 'Sure, 'Sure. Sure, Sure. sure sure. sure ? If you If Yau [you live jive In m 1n an area devoid of natural wonders, wanders, wonders ? wonders wonders. your mind will be blown, blown' blown. blown ? -e i ? ,1 IL it ? at ? KY ? fl ft bat at Which came from this image: ![image](https://user-images.githubusercontent.com/9599/95692952-5dd7c880-0bde-11eb-939a-d10b800a4105.png) The XML for that is: ```xml <recoIndex docType="unknown" objType="image" objID="05ffb72b307bf495f064243c7099d94f" engineVersion="6.5.17.7" recoType="service" lang="en" objWidth="1000" objHeight="1504"> <item x="68" y="75" w="104" h="37"> <t w="60">Sure</t> <t w="52">'Sure,</t> <t w="47">'Sure.</t> <t w="33">Sure,</t> <t w="26">Sure.</t> </item> <item x="182" y="83" w="92" h="26"> <t w="62">sure</t> <t w="58">sure.</t> <t w="46">sure ?</t> </item> <item x="69" y="132" w="107" h="45"> <t w="81">If you</t> <t w="64">If Yau</t> <t w="31">[you</t> </item> <item x="186" y="132" w="67" h="35"> <t w="85">live</t> <t w="51">jive</t> </item> <item x="263" y="140" w="36" h="27"> <t w="82">In</t> <t w="56">m</t> <t w="53">1n</t> </item> <item x="309" y="140" w="53" h="27"> <t w="82">an</t> </item> <item x="372" y="141" w="90" h="26"> <t w="94">area</t> </item> <item x="472" y="132" w="138" h="35"> <t w="85">devoid</t> </item> <item x="620" y="132" w="43" h="35"> <t w="82">of</t> </item> <item x="68" y="190" w="137" h="35"> <t w="87">natural</t> </item> <item x="215" y="190" w="187" h="39"> <t w="57">wonders,</t> <t w="55">wanders,</t> <t w="52">wonders ?</t> <t w="45">wonders</t> <t w="42">won… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Better handling of OCR data 718949182 | |
706784028 | https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706784028 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDcwNjc4NDAyOA== | simonw 9599 | 2020-10-11T23:20:32Z | 2020-10-11T23:20:32Z | MEMBER | I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Configure FTS + add an index on the date columns 718938508 | |
706776808 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776808 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjc3NjgwOA== | simonw 9599 | 2020-10-11T22:23:14Z | 2020-10-11T22:23:14Z | MEMBER | ... but it's still important to be able to get to the rendered note directly from the browse notes `/evernote/notes` page. Maybe use a simple `render_cell()` hook that just knows how to generate the link to the rendered note page? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
706776680 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776680 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjc3NjY4MA== | simonw 9599 | 2020-10-11T22:22:16Z | 2020-10-11T22:22:16Z | MEMBER | Maybe the best way do this is with a custom route, `/-/evernote/note-id` - that way I can clean the HTML and resolve the other things in the `<en-note>` structure without using `render_cell()` and the like. My concern about using `render_cell()` is that it could lead to weird security problems when combined with `?sql=` queries. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
706776447 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776447 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjc3NjQ0Nw== | simonw 9599 | 2020-10-11T22:20:32Z | 2020-10-11T22:20:32Z | MEMBER | Or... I could do this client-side. JavaScript that looks for `<en-media>` tags and fetches the data using `fetch()` wouldn't be too hard to write. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
706776242 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776242 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjc3NjI0Mg== | simonw 9599 | 2020-10-11T22:18:30Z | 2020-10-11T22:19:48Z | MEMBER | Alternatively, rather than relying on `datasette-media` this could base64-embed the images. `evernote-to-sqlite` could register itself as a Datasette plugin that knows how to do this. Maybe rename the column to `evernote_content` and register a render cell hook that knows how to rewrite those note bodies so that they are visible? Might need to feed them through Bleach too, just in case any nasty code can get into them. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
706776180 | https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776180 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDcwNjc3NjE4MA== | simonw 9599 | 2020-10-11T22:17:55Z | 2020-10-11T22:17:55Z | MEMBER | We could even do server-side thumbnailing for some of these images, but I'm inclined to serve up the full size ones and set a width on the image element based on the `width` attribute on `<en-media>`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out how to display images from <en-media> tags inline in Datasette 718938889 | |
706775706 | https://github.com/dogsheep/evernote-to-sqlite/issues/1#issuecomment-706775706 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/1 | MDEyOklzc3VlQ29tbWVudDcwNjc3NTcwNg== | simonw 9599 | 2020-10-11T22:14:00Z | 2020-10-11T22:14:00Z | MEMBER | A live demo would be good too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Documentation on how to use this with Datasette 718934942 | |
704553385 | https://github.com/dogsheep/github-to-sqlite/pull/48#issuecomment-704553385 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48 | MDEyOklzc3VlQ29tbWVudDcwNDU1MzM4NQ== | simonw 9599 | 2020-10-06T21:07:44Z | 2020-10-06T21:07:44Z | MEMBER | Sorry for not looking at this sooner, trying it out now - pull request looks great! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add pull requests 681228542 | |
695879531 | https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879531 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | MDEyOklzc3VlQ29tbWVudDY5NTg3OTUzMQ== | simonw 9599 | 2020-09-21T02:55:28Z | 2020-09-21T02:55:54Z | MEMBER | Actually for the tie-breaker it should be something like https://latest.datasette.io/fixtures?sql=select+pk%2C+created%2C+planet_int%2C+on_earth%2C+state%2C+city_id%2C+neighborhood%2C+tags%2C+complex_array%2C+distinct_some_null+from+facetable+where+%28created+%3E+%3Ap1+or+%28created+%3D+%3Ap1+and+%28%28pk+%3E+%3Ap0%29%29%29%29+order+by+created%2C+pk+limit+11&p0=10&p1=2019-01-16+08%3A00%3A00 ```sql where ( created > :p1 or ( created = :p1 and ((pk > :p0)) ) ) ``` But with `rowid` and `timestamp` in place of `pk` and `created`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pagination 705215230 | |
695879237 | https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879237 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | MDEyOklzc3VlQ29tbWVudDY5NTg3OTIzNw== | simonw 9599 | 2020-09-21T02:53:29Z | 2020-09-21T02:53:29Z | MEMBER | If previous page ended at `2018-02-11T16:32:53+00:00`: ```sql select search_index.rowid, search_index.type, search_index.key, search_index.title, search_index.category, search_index.timestamp, search_index.search_1 from search_index where date("timestamp") = '2018-02-11' and timestamp < '2018-02-11T16:32:53+00:00' order by search_index.timestamp desc, rowid limit 41 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pagination 705215230 | |
695877627 | https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695877627 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16 | MDEyOklzc3VlQ29tbWVudDY5NTg3NzYyNw== | simonw 9599 | 2020-09-21T02:42:29Z | 2020-09-21T02:42:29Z | MEMBER | Fun twist: assuming `timestamp` is always stored as UTC, I need the interface to be timezone aware so I can see e.g. everything from 4th July 2020 in the San Francisco timezone definition of 4th July 2020. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Timeline view 694493566 | |
695875274 | https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695875274 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | MDEyOklzc3VlQ29tbWVudDY5NTg3NTI3NA== | simonw 9599 | 2020-09-21T02:28:58Z | 2020-09-21T02:28:58Z | MEMBER | Datasette's implementation is complex because it has to support compound primary keys: https://github.com/simonw/datasette/blob/a258339a935d8d29a95940ef1db01e98bb85ae63/datasette/utils/__init__.py#L88-L114 - but that's not something that's needed for dogsheep-beta. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pagination 705215230 | |
695856967 | https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856967 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | MDEyOklzc3VlQ29tbWVudDY5NTg1Njk2Nw== | simonw 9599 | 2020-09-21T00:26:59Z | 2020-09-21T00:26:59Z | MEMBER | It's a shame Datasette doesn't currently have an easy way to implement sorted-by-rank keyset-paginated using a TableView or QueryView. I'll have to do this using the custom SQL query constructed in the plugin: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L8-L43 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pagination 705215230 | |
695856398 | https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856398 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | MDEyOklzc3VlQ29tbWVudDY5NTg1NjM5OA== | simonw 9599 | 2020-09-21T00:22:20Z | 2020-09-21T00:22:20Z | MEMBER | I'm going to try for keyset pagination sorted by relevance just as a learning exercise. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pagination 705215230 | |
695855723 | https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855723 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | MDEyOklzc3VlQ29tbWVudDY5NTg1NTcyMw== | simonw 9599 | 2020-09-21T00:16:52Z | 2020-09-21T00:17:53Z | MEMBER | It feels a bit weird to implement keyset pagination against results sorted by `rank` because the ranks could change substantially if the search index gets updated while the user is paginating. I may just ignore that though. If you want reliable pagination you can get it by sorting by date. Maybe it doesn't even make sense to offer pagination if you sort by relevance? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pagination 705215230 | |
695855646 | https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855646 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | MDEyOklzc3VlQ29tbWVudDY5NTg1NTY0Ng== | simonw 9599 | 2020-09-21T00:16:11Z | 2020-09-21T00:16:11Z | MEMBER | Should I do this with offset/limit or should I do proper keyset pagination? I think keyset because then it will work well for the full search interface with no filters or search string. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Pagination 705215230 | |
695851036 | https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695851036 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16 | MDEyOklzc3VlQ29tbWVudDY5NTg1MTAzNg== | simonw 9599 | 2020-09-20T23:34:57Z | 2020-09-20T23:34:57Z | MEMBER | Really basic starting point is to add facet by date. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Timeline view 694493566 | |
695124698 | https://github.com/dogsheep/dogsheep-beta/issues/15#issuecomment-695124698 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/15 | MDEyOklzc3VlQ29tbWVudDY5NTEyNDY5OA== | simonw 9599 | 2020-09-18T23:17:38Z | 2020-09-18T23:17:38Z | MEMBER | This can be part of the demo instance in #6. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add a bunch of config examples 694136490 | |
695113871 | https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-695113871 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | MDEyOklzc3VlQ29tbWVudDY5NTExMzg3MQ== | simonw 9599 | 2020-09-18T22:30:17Z | 2020-09-18T22:30:17Z | MEMBER | I think I know what's going on here: https://github.com/dogsheep/dogsheep-beta/blob/0f1b951c5131d16f3c8559a8e4d79ed5c559e3cb/dogsheep_beta/__init__.py#L166-L171 This is a logic bug - the `compiled` variable could be the template from the previous loop! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | the JSON object must be str, bytes or bytearray, not 'Undefined' 703970814 | |
695109140 | https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695109140 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25 | MDEyOklzc3VlQ29tbWVudDY5NTEwOTE0MA== | simonw 9599 | 2020-09-18T22:12:20Z | 2020-09-18T22:12:20Z | MEMBER | Documented here: https://github.com/dogsheep/dogsheep-beta/blob/534fc9689227eba70e69a45da0cee5820bbda9e1/README.md#datasette-plugin | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | template_debug mechanism 704685890 | |
695108895 | https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695108895 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25 | MDEyOklzc3VlQ29tbWVudDY5NTEwODg5NQ== | simonw 9599 | 2020-09-18T22:11:32Z | 2020-09-18T22:11:32Z | MEMBER | I'm going to make this a new plugin configuration setting, `template_debug`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | template_debug mechanism 704685890 | |
694557425 | https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694557425 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | MDEyOklzc3VlQ29tbWVudDY5NDU1NzQyNQ== | simonw 9599 | 2020-09-17T23:41:01Z | 2020-09-17T23:41:01Z | MEMBER | I removed all of the `json.loads()` calls and I'm still getting that `Undefined` error. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | the JSON object must be str, bytes or bytearray, not 'Undefined' 703970814 | |
694554584 | https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | MDEyOklzc3VlQ29tbWVudDY5NDU1NDU4NA== | simonw 9599 | 2020-09-17T23:31:25Z | 2020-09-17T23:31:25Z | MEMBER | I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | the JSON object must be str, bytes or bytearray, not 'Undefined' 703970814 | |
694553579 | https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694553579 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | MDEyOklzc3VlQ29tbWVudDY5NDU1MzU3OQ== | simonw 9599 | 2020-09-17T23:28:37Z | 2020-09-17T23:28:37Z | MEMBER | More investigation in pdb: ``` (dogsheep-beta) dogsheep-beta % datasette . --get '/-/beta?q=pycon&sort=oldest' --pdb > /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads() -> raise TypeError(f'the JSON object must be str, bytes or bytearray, ' (Pdb) list 336 if s.startswith('\ufeff'): 337 raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)", 338 s, 0) 339 else: 340 if not isinstance(s, (bytes, bytearray)): 341 -> raise TypeError(f'the JSON object must be str, bytes or bytearray, ' 342 f'not {s.__class__.__name__}') 343 s = s.decode(detect_encoding(s), 'surrogatepass') 344 345 if "encoding" in kw: 346 import warnings (Pdb) bytes <class 'bytes'> (Pdb) locals()['s'] Undefined (Pdb) type(locals()['s']) <class 'jinja2.runtime.Undefined'> ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | the JSON object must be str, bytes or bytearray, not 'Undefined' 703970814 | |
694552681 | https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552681 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | MDEyOklzc3VlQ29tbWVudDY5NDU1MjY4MQ== | simonw 9599 | 2020-09-17T23:25:54Z | 2020-09-17T23:25:54Z | MEMBER | This is the template fragment it's rendering: ```html+jinja <div style="overflow: hidden;"> <p>Tweet by <a href="https://twitter.com/{{ display.screen_name }}">@{{ display.screen_name }}</a> ({{ display.user_name }}, {{ "{:,}".format(display.followers_count or 0) }} followers) on <a href="https://twitter.com/{{ display.screen_name }}/status/{{ display.tweet_id }}">{{ display.created_at }}</a></p> </p> <blockquote>{{ display.full_text }}</blockquote> {% if display.media_urls and json.loads(display.media_urls) %} {% for url in json.loads(display.media_urls) %} <img src="{{ url }}" style="height: 200px;"> {% endfor %} {% endif %} </div> ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | the JSON object must be str, bytes or bytearray, not 'Undefined' 703970814 | |
694552393 | https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552393 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | MDEyOklzc3VlQ29tbWVudDY5NDU1MjM5Mw== | simonw 9599 | 2020-09-17T23:25:01Z | 2020-09-17T23:25:17Z | MEMBER | Ran `locals()` In the debugger: `{'range': <class 'range'>, 'dict': <class 'dict'>, 'lipsum': <function generate_lorem_ipsum at 0x10aeff430>, 'cycler': <class 'jinja2.utils.Cycler'>, 'joiner': <class 'jinja2.utils.Joiner'>, 'namespace': <class 'jinja2.utils.Namespace'>, 'rank': -9.383801886431414, 'rowid': 14297, 'type': 'twitter.db/tweets', 'key': '312658917933076480', 'title': 'Tweet by @chrisstreeter', 'category': 2, 'timestamp': '2013-03-15T20:17:49+00:00', 'search_1': '@simonw are you at pycon? Would love to meet you.', 'display': {'avatar_url': 'https://pbs.twimg.com/profile_images/806275088597204993/38yLHfJi_normal.jpg', 'user_name': 'Chris Streeter', 'screen_name': 'chrisstreeter', 'followers_count': 280, 'tweet_id': 312658917933076480, 'created_at': '2013-03-15T20:17:49+00:00', 'full_text': '@simonw are you at pycon? Would love to meet you.', 'media_urls_2': '[]', 'media_urls': '[]'}, 'json': <module 'json' from '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py'>}` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | the JSON object must be str, bytes or bytearray, not 'Undefined' 703970814 | |
694551646 | https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551646 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | MDEyOklzc3VlQ29tbWVudDY5NDU1MTY0Ng== | simonw 9599 | 2020-09-17T23:22:48Z | 2020-09-17T23:22:48Z | MEMBER | Looks like its happening in a Jinja fragment template for one of the results: ``` /Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/__init__.py(169)process_results() -> output = compiled.render({**result, **{"json": json}}) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/asyncsupport.py(71)render() -> return original_render(self, *args, **kwargs) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(1090)render() -> self.environment.handle_exception() /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(832)handle_exception() -> reraise(*rewrite_traceback_stack(source=source)) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/_compat.py(28)reraise() -> raise value.with_traceback(tb) <template>(5)top-level template code() > /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads() -> raise TypeError(f'the JSON object must be str, bytes or bytearray, ' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | the JSON object must be str, bytes or bytearray, not 'Undefined' 703970814 | |
694551406 | https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551406 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | MDEyOklzc3VlQ29tbWVudDY5NDU1MTQwNg== | simonw 9599 | 2020-09-17T23:22:07Z | 2020-09-17T23:22:07Z | MEMBER | Neat, I can debug this with the new `--pdb` option: datasette . --get '/-/beta?q=pycon&sort=oldest' --pdb | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | the JSON object must be str, bytes or bytearray, not 'Undefined' 703970814 | |
694548909 | https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-694548909 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16 | MDEyOklzc3VlQ29tbWVudDY5NDU0ODkwOQ== | simonw 9599 | 2020-09-17T23:15:09Z | 2020-09-17T23:15:09Z | MEMBER | I have sort by date now, #21. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Timeline view 694493566 | |
693794700 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693794700 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDY5Mzc5NDcwMA== | simonw 9599 | 2020-09-17T04:02:39Z | 2020-09-17T04:02:39Z | MEMBER | It would be useful if you could pass an `--accept` option to this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
693789129 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693789129 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDY5Mzc4OTEyOQ== | simonw 9599 | 2020-09-17T03:40:01Z | 2020-09-17T03:40:01Z | MEMBER | Bug with endpoints that return dictionaries rather than arrays: ``` github-to-sqlite get /users/simonw [ "login", "id", "node_id", "avatar_url", "gravatar_id", "url", "html_url", "followers_url", "following_url", "gists_url", "starred_url", "subscriptions_url", "organizations_url", "repos_url", "events_url", "received_events_url", "type", "site_admin", "name", "company", "blog", "location", "email", "hireable", "bio", "twitter_username", "public_repos", "public_gists", "followers", "following", "created_at", "updated_at" ] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
693788387 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693788387 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDY5Mzc4ODM4Nw== | simonw 9599 | 2020-09-17T03:36:47Z | 2020-09-17T03:36:58Z | MEMBER | Fun demo of the `--nl` option: github-to-sqlite get /users/simonw/repos --paginate --nl | sqlite-utils insert simonw.db repos - --nl | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
693788032 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693788032 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDY5Mzc4ODAzMg== | simonw 9599 | 2020-09-17T03:35:22Z | 2020-09-17T03:35:22Z | MEMBER | Documentation: https://github.com/dogsheep/github-to-sqlite/blob/b02bf135485c0a7a3768868967f45a6b5e515289/README.md#making-authenticated-api-calls | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
693775622 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693775622 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDY5Mzc3NTYyMg== | simonw 9599 | 2020-09-17T02:48:34Z | 2020-09-17T02:48:34Z | MEMBER | I'd like a `--paginate` option that does the same thing as https://github.com/simonw/paginate-json | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
693773191 | https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693773191 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | MDEyOklzc3VlQ29tbWVudDY5Mzc3MzE5MQ== | simonw 9599 | 2020-09-17T02:39:26Z | 2020-09-17T02:39:26Z | MEMBER | I'm going to start with `github-to-sqlite get` and `github-to-sqlite post` - I may add `put` and suchlike later on. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Commands for making authenticated API calls 703218756 | |
689226390 | https://github.com/dogsheep/dogsheep-beta/issues/17#issuecomment-689226390 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17 | MDEyOklzc3VlQ29tbWVudDY4OTIyNjM5MA== | simonw 9599 | 2020-09-09T00:36:07Z | 2020-09-09T00:36:07Z | MEMBER | Alternative names: - type - record_type - doctype I think `type` is right. It matches what Elasticsearch used to call their equivalent of this (before they removed the feature!). https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rename "table" to "type" 694500679 | |
688626037 | https://github.com/dogsheep/dogsheep-beta/issues/19#issuecomment-688626037 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19 | MDEyOklzc3VlQ29tbWVudDY4ODYyNjAzNw== | simonw 9599 | 2020-09-08T05:27:07Z | 2020-09-08T05:27:07Z | MEMBER | A really clever way to do this would be with triggers. The indexer script would add triggers to each of the database tables that it is indexing - each in their own database. Those triggers would then maintain a `_index_queue_` table. This table would record the primary key of rows that are added, modified or deleted. The indexer could then work by reading through the `_index_queue_` table, re-indexing (or deleting) just the primary keys listed there, and then emptying the queue once it has finished. This would add a small amount of overhead to insert/update/delete queries run against the table. My hunch is that the overhead would be miniscule, but I could still allow people to opt-out for tables that are so high traffic that this would matter. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out incremental re-indexing 695556681 | |
688625430 | https://github.com/dogsheep/dogsheep-beta/issues/19#issuecomment-688625430 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19 | MDEyOklzc3VlQ29tbWVudDY4ODYyNTQzMA== | simonw 9599 | 2020-09-08T05:24:50Z | 2020-09-08T05:24:50Z | MEMBER | I thought about allowing tables to define a incremental indexing SQL query - maybe something that can return just records touched in the past hour, or records since a recorded "last indexed record" value. The problem with this is deletes - if you delete a record, how does the indexer know to remove it? See #18 - that's already caused problems. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Figure out incremental re-indexing 695556681 | |
688623097 | https://github.com/dogsheep/dogsheep-beta/issues/18#issuecomment-688623097 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18 | MDEyOklzc3VlQ29tbWVudDY4ODYyMzA5Nw== | simonw 9599 | 2020-09-08T05:15:51Z | 2020-09-08T05:15:51Z | MEMBER | I'm inclined to go with the first, simpler option. I have longer term plans for efficient incremental index updates based on clever trickery with triggers. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Deleted records stay in the search index 695553522 | |
688622995 | https://github.com/dogsheep/dogsheep-beta/issues/18#issuecomment-688622995 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18 | MDEyOklzc3VlQ29tbWVudDY4ODYyMjk5NQ== | simonw 9599 | 2020-09-08T05:15:21Z | 2020-09-08T05:15:21Z | MEMBER | Alternatively it could run as it does now but add a `DELETE FROM index1.search_index WHERE key not in (select key from ...)`. I'm not sure which would be more efficient. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Deleted records stay in the search index 695553522 | |
687880459 | https://github.com/dogsheep/dogsheep-beta/issues/17#issuecomment-687880459 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17 | MDEyOklzc3VlQ29tbWVudDY4Nzg4MDQ1OQ== | simonw 9599 | 2020-09-06T19:36:32Z | 2020-09-06T19:36:32Z | MEMBER | At some point I may even want to support search types which are indexed from (and inflated from) more than one database file. I'm going to ignore that for the moment though. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Rename "table" to "type" 694500679 | |
686774592 | https://github.com/dogsheep/dogsheep-beta/issues/13#issuecomment-686774592 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/13 | MDEyOklzc3VlQ29tbWVudDY4Njc3NDU5Mg== | simonw 9599 | 2020-09-03T21:30:21Z | 2020-09-03T21:30:21Z | MEMBER | This is partially supported: the custom search SQL we run doesn't escape them, but the `?_search` used to calculate facet counts does. So this is a bug. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support advanced FTS queries 692386625 | |
686767208 | https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686767208 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9 | MDEyOklzc3VlQ29tbWVudDY4Njc2NzIwOA== | simonw 9599 | 2020-09-03T21:12:14Z | 2020-09-03T21:12:14Z | MEMBER | Documentation: https://github.com/dogsheep/dogsheep-beta/blob/0.4/README.md#custom-results-display | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for defining custom display of results 691521965 | |
686689612 | https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-686689612 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3 | MDEyOklzc3VlQ29tbWVudDY4NjY4OTYxMg== | simonw 9599 | 2020-09-03T18:44:20Z | 2020-09-03T18:44:20Z | MEMBER | Facets are now displayed but selecting them doesn't work yet. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Datasette plugin to provide custom page for running faceted, ranked searches 689810340 | |
686689366 | https://github.com/dogsheep/dogsheep-beta/issues/5#issuecomment-686689366 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/5 | MDEyOklzc3VlQ29tbWVudDY4NjY4OTM2Ng== | simonw 9599 | 2020-09-03T18:43:50Z | 2020-09-03T18:43:50Z | MEMBER | No longer needed thanks to #9 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add a context column that's not searchable 689847361 | |
686689122 | https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686689122 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9 | MDEyOklzc3VlQ29tbWVudDY4NjY4OTEyMg== | simonw 9599 | 2020-09-03T18:43:20Z | 2020-09-03T18:43:20Z | MEMBER | Needs documentation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for defining custom display of results 691521965 | |
686688963 | https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686688963 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9 | MDEyOklzc3VlQ29tbWVudDY4NjY4ODk2Mw== | simonw 9599 | 2020-09-03T18:42:59Z | 2020-09-03T18:42:59Z | MEMBER | I'm pleased with how this works now. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for defining custom display of results 691521965 | |
686618669 | https://github.com/dogsheep/dogsheep-beta/issues/11#issuecomment-686618669 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/11 | MDEyOklzc3VlQ29tbWVudDY4NjYxODY2OQ== | simonw 9599 | 2020-09-03T16:47:34Z | 2020-09-03T16:53:25Z | MEMBER | I think a `is_public` integer column which defaults to 0 would be good here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Public / Private mechanism 692125110 | |
686238498 | https://github.com/dogsheep/dogsheep-beta/issues/10#issuecomment-686238498 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/10 | MDEyOklzc3VlQ29tbWVudDY4NjIzODQ5OA== | simonw 9599 | 2020-09-03T04:05:05Z | 2020-09-03T04:05:05Z | MEMBER | Since the first two categories are `created` and `saved` this one should be called `received`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Category 3: received 691557547 | |
686163754 | https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686163754 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9 | MDEyOklzc3VlQ29tbWVudDY4NjE2Mzc1NA== | simonw 9599 | 2020-09-03T00:46:21Z | 2020-09-03T00:46:21Z | MEMBER | Challenge: the `dogsheep-beta.yml` configuration file that is passed to the `dogsheep-beta index` command needs to also be made available to Datasette itself, so that it can read the configuration. Let's say it can either be duplicated in the `plugins` configuration block of the `metadata.yml` OR you can do this in `metadata.yml`: ```yaml plugins: dogsheep-beta: config_file: dogsheep-beta.yml ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for defining custom display of results 691521965 | |
686158454 | https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686158454 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9 | MDEyOklzc3VlQ29tbWVudDY4NjE1ODQ1NA== | simonw 9599 | 2020-09-03T00:32:42Z | 2020-09-03T00:32:42Z | MEMBER | If this turns out to be too inefficient I could add a `display` text column to the `search_index` table which is designed to be populated with arbitrary JSON by the indexing query, which can then be used to render the template fragment. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for defining custom display of results 691521965 | |
686154627 | https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686154627 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9 | MDEyOklzc3VlQ29tbWVudDY4NjE1NDYyNw== | simonw 9599 | 2020-09-03T00:19:22Z | 2020-09-03T00:19:22Z | MEMBER | If this performs well enough (100 displayed items will be 100 extra `display_sql` calls) then I'll go with this as the design for the feature. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for defining custom display of results 691521965 | |
686154486 | https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686154486 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9 | MDEyOklzc3VlQ29tbWVudDY4NjE1NDQ4Ng== | simonw 9599 | 2020-09-03T00:18:54Z | 2020-09-03T00:18:54Z | MEMBER | `display_sql` could be optional. If it's not defined, a `row` object is passed to the template which is the row that's stored in `search_index`. If `display_sql` IS defined then it's executed and the result is made available as a `display` object in addition to the `row` object. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for defining custom display of results 691521965 | |
686153967 | https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686153967 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9 | MDEyOklzc3VlQ29tbWVudDY4NjE1Mzk2Nw== | simonw 9599 | 2020-09-03T00:17:16Z | 2020-09-03T00:17:55Z | MEMBER | Maybe I can take advantage of https://sqlite.org/np1queryprob.html here - I could define a SQL query for fetching the "display" version of each item, and include a Jinja template fragment in the configuration as well. Maybe something like this: ```yaml photos.db: photos_with_apple_metadata: sql: |- select sha256 as key, 'Photo in ' || coalesce(place_city, 'unknown') as title, ( select group_concat(normalized_string, ' ') from labels where labels.uuid = photos_with_apple_metadata.uuid ) as search_1, date as timestamp, 1 as category from photos_with_apple_metadata display_sql: |- select sha256, place_city, date from photos_with_apple_metadata where sha256 = :key display: |- <img src="https://photos.simonwillison.net/i/{{ display.sha256 }}.jpeg?w=600"> <p>Taken in {{ display.place_city }} on {{ display.date }}</p> ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for defining custom display of results 691521965 | |
685970384 | https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685970384 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7 | MDEyOklzc3VlQ29tbWVudDY4NTk3MDM4NA== | simonw 9599 | 2020-09-02T20:11:41Z | 2020-09-02T20:11:59Z | MEMBER | Default categories: - 1 = created - 2 = saved | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for differentiating between "by me" and "liked by me" 691265198 | |
685966707 | https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966707 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7 | MDEyOklzc3VlQ29tbWVudDY4NTk2NjcwNw== | simonw 9599 | 2020-09-02T20:04:08Z | 2020-09-02T20:04:08Z | MEMBER | I'll make `category` a foreign key to a `categories` table so Datasette can automatically show the `name` column. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for differentiating between "by me" and "liked by me" 691265198 | |
685966361 | https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966361 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7 | MDEyOklzc3VlQ29tbWVudDY4NTk2NjM2MQ== | simonw 9599 | 2020-09-02T20:03:29Z | 2020-09-02T20:03:41Z | MEMBER | I'm going to implement the first version of this as an indexed integer `category` column which has 1 for "about me" and 2 for "liked by me" - and space for other category numerals in the future, albeit a row can only belong to one category. I'll think about a full tagging system separately. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for differentiating between "by me" and "liked by me" 691265198 | |
685965516 | https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685965516 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7 | MDEyOklzc3VlQ29tbWVudDY4NTk2NTUxNg== | simonw 9599 | 2020-09-02T20:01:54Z | 2020-09-02T20:01:54Z | MEMBER | Relevant post: https://sqlite.org/forum/forumpost/9f06fedaa5 - drh says: > Indexes are one-to-one. There is one entry in the index for each row in the table. > > You are asking for an index that is many-to-one - multiple index entries for each table row. > > A Full-Text Index is basically a many-to-one index. So if all of your array entries really are words, you could probably get this to work using a Full-Text Index. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for differentiating between "by me" and "liked by me" 691265198 | |
685962280 | https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685962280 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7 | MDEyOklzc3VlQ29tbWVudDY4NTk2MjI4MA== | simonw 9599 | 2020-09-02T19:55:26Z | 2020-09-02T19:59:58Z | MEMBER | Relevant: https://charlesleifer.com/blog/a-tour-of-tagging-schemas-many-to-many-bitmaps-and-more/ SQLite supports bitwise operators Binary AND (&) and Binary OR (|) - I could try those. Not sure how they interact with indexes though. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for differentiating between "by me" and "liked by me" 691265198 | |
685961809 | https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-685961809 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3 | MDEyOklzc3VlQ29tbWVudDY4NTk2MTgwOQ== | simonw 9599 | 2020-09-02T19:54:24Z | 2020-09-02T19:54:24Z | MEMBER | This should implement search highlighting too, as seen on https://til.simonwillison.net/til/search?q=cloud <img width="1029" alt="TIL_search__cloud" src="https://user-images.githubusercontent.com/9599/92029959-32c6a300-ed1b-11ea-8b5e-971950980c38.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Datasette plugin to provide custom page for running faceted, ranked searches 689810340 | |
685960072 | https://github.com/dogsheep/dogsheep-beta/issues/8#issuecomment-685960072 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/8 | MDEyOklzc3VlQ29tbWVudDY4NTk2MDA3Mg== | simonw 9599 | 2020-09-02T19:50:47Z | 2020-09-02T19:50:47Z | MEMBER | This doesn't actually help, because the Datasette table view page doesn't then support adding the `where search_index_fts match :query` bit. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create a view for running faceted searches 691369691 | |
685895540 | https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685895540 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7 | MDEyOklzc3VlQ29tbWVudDY4NTg5NTU0MA== | simonw 9599 | 2020-09-02T17:46:44Z | 2020-09-02T17:46:44Z | MEMBER | Some opet questions about this: - Should I restrict to two exclusive categories here, or should I have a generic category mechanism that can be expanded to more than two? - Should an item be able to exist in more than one category? Do I want to be able to mark an indexed item as both by-me and liked-by-me for example? This question is more interesting if the number of categories is greater than two. - How should this be modeled? Single column, multiple boolean columns, JSON array, m2m against separate table? - What's the best way to make this performant | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Mechanism for differentiating between "by me" and "liked by me" 691265198 | |
685121074 | https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685121074 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2 | MDEyOklzc3VlQ29tbWVudDY4NTEyMTA3NA== | simonw 9599 | 2020-09-01T20:42:00Z | 2020-09-01T20:42:00Z | MEMBER | Documentation at the bottom of the Usage section here: https://github.com/dogsheep/dogsheep-beta/blob/0.2/README.md#usage | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Apply porter stemming 689809225 | |
685115519 | https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685115519 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2 | MDEyOklzc3VlQ29tbWVudDY4NTExNTUxOQ== | simonw 9599 | 2020-09-01T20:31:57Z | 2020-09-01T20:31:57Z | MEMBER | Actually this doesn't work: you can't turn on stemming for specific tables, because all of the content goes into a single `search_index` table which is configured the same way. So stemming needs to be a global option. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Apply porter stemming 689809225 | |
684425714 | https://github.com/dogsheep/pocket-to-sqlite/issues/5#issuecomment-684425714 | https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDY4NDQyNTcxNA== | simonw 9599 | 2020-09-01T06:18:32Z | 2020-09-01T06:18:32Z | MEMBER | Good suggestion, I'll setup a demo somewhere. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Set up a demo 629473827 | |
684424396 | https://github.com/dogsheep/pocket-to-sqlite/issues/3#issuecomment-684424396 | https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDY4NDQyNDM5Ng== | simonw 9599 | 2020-09-01T06:17:45Z | 2020-09-01T06:17:45Z | MEMBER | It looks like I could ignore the `image` column and synthesize a unique key from the data in the `images` column using `$item_id/$image_id`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Extract images into separate tables 503243784 | |
684395444 | https://github.com/dogsheep/dogsheep-beta/issues/4#issuecomment-684395444 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/4 | MDEyOklzc3VlQ29tbWVudDY4NDM5NTQ0NA== | simonw 9599 | 2020-09-01T06:00:03Z | 2020-09-01T06:00:03Z | MEMBER | I ran `sqlite-utils optimize beta.db` against my test DB and the size reduced from 183M to 176M - and a 450ms search ran in 359ms. So not a huge improvement but still worthwhile. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Optimize the FTS table 689839399 | |
684250044 | https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-684250044 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3 | MDEyOklzc3VlQ29tbWVudDY4NDI1MDA0NA== | simonw 9599 | 2020-09-01T05:01:09Z | 2020-09-01T05:01:23Z | MEMBER | Maybe this starts out as a custom templated canned query. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Datasette plugin to provide custom page for running faceted, ranked searches 689810340 | |
675523053 | https://github.com/dogsheep/github-to-sqlite/issues/47#issuecomment-675523053 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/47 | MDEyOklzc3VlQ29tbWVudDY3NTUyMzA1Mw== | simonw 9599 | 2020-08-18T14:45:53Z | 2020-08-18T14:45:53Z | MEMBER | ``` % github-to-sqlite emojis emojis.db --fetch [########----------------------------] 397/1682 23% 00:03:43 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | emojis command 681086659 | |
675509550 | https://github.com/dogsheep/github-to-sqlite/issues/39#issuecomment-675509550 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDY3NTUwOTU1MA== | simonw 9599 | 2020-08-18T14:23:56Z | 2020-08-18T14:23:56Z | MEMBER | I think this is fixed: https://github-to-sqlite.dogsheep.net/github/issues?_facet=repo | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | issues foreign key to repo isn't working 613777056 | |
675259273 | https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-675259273 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | MDEyOklzc3VlQ29tbWVudDY3NTI1OTI3Mw== | simonw 9599 | 2020-08-18T05:28:32Z | 2020-08-18T05:28:32Z | MEMBER | Oh that's interesting - i didn't realize "reviews" were a separate concept. I'd definitely accept a pull request adding those! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature: pull request reviews and comments 664485022 | |
669241341 | https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-669241341 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDY2OTI0MTM0MQ== | simonw 9599 | 2020-08-05T14:55:14Z | 2020-08-05T14:55:14Z | MEMBER | Looks like there's a column that's not consistently there, so the table got created without it. Easiest fix is to add `alter=True` to this line: https://github.com/dogsheep/swarm-to-sqlite/blob/f4a82633da927cde672c9d9af92930bfca2e3ddf/swarm_to_sqlite/utils.py#L94 That will cause `sqlite-utils` to notice if there's a missing column and add it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Error thrown: table photos has no column named hasSticker 648245071 | |
663143160 | https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-663143160 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48 | MDEyOklzc3VlQ29tbWVudDY2MzE0MzE2MA== | simonw 9599 | 2020-07-23T17:46:07Z | 2020-07-23T17:46:07Z | MEMBER | Frustratingly, these links don't work on PyPI: https://pypi.org/project/twitter-to-sqlite/ There's an issue about that here: https://github.com/pypa/readme_renderer/issues/169 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add a table of contents to the README 663976976 | |
662630868 | https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662630868 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48 | MDEyOklzc3VlQ29tbWVudDY2MjYzMDg2OA== | simonw 9599 | 2020-07-22T19:03:02Z | 2020-07-22T19:03:02Z | MEMBER | Done! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add a table of contents to the README 663976976 | |
662626901 | https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662626901 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48 | MDEyOklzc3VlQ29tbWVudDY2MjYyNjkwMQ== | simonw 9599 | 2020-07-22T18:54:53Z | 2020-07-22T18:54:53Z | MEMBER | I'm going to use a GitHub Action to run `npx markdown-toc README.md -i` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add a table of contents to the README 663976976 | |
660554811 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554811 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1NDgxMQ== | simonw 9599 | 2020-07-18T23:03:13Z | 2020-07-18T23:03:13Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/tags now shows a `repo` column instead of a `repo_id` column. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 | |
660554299 | https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554299 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDY2MDU1NDI5OQ== | simonw 9599 | 2020-07-18T22:58:24Z | 2020-07-18T23:02:52Z | MEMBER | Deploying the fixed version like this: ``` $ gcloud config set run/region us-central1 $ gcloud config set project datasette-222320 $ datasette publish cloudrun /tmp/github.db \ -m demo-metadata.json \ --service github-to-sqlite \ --install=py-gfm \ --install='datasette-search-all>=0.3' \ --install='datasette-render-markdown>=1.1.2' \ --install=datasette-pretty-json \ --install=datasette-json-html \ --install=datasette-vega ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Fix the demo - it breaks because of the tags table change 660429601 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
updated_at (date) 97 ✖