html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461226402,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1,1461226402,IC_kwDOJHON9s5XGIui,9599,simonw,2023-03-09T03:47:11Z,2023-03-09T03:47:11Z,MEMBER,"Me:
> Write applescript to loop through all of my Apple Notes and output their contents

ChatGPT:

```applescript
tell application ""Notes""
   repeat with eachNote in every note
      set noteTitle to the name of eachNote
      set noteBody to the body of eachNote
      display dialog noteTitle & return & return & noteBody
   end repeat
end tell
```
Me:
> How can I turn that into a zsh script which will output the results to my terminal

ChatGPT:
```zsh
#!/usr/bin/zsh

osascript -e 'tell application ""Notes""
   repeat with eachNote in every note
      set noteTitle to the name of eachNote
      set noteBody to the body of eachNote
      set output to noteTitle & ""\n"" & noteBody & ""\n""
      display dialog output
      log output
   end repeat
end tell'
```
I had to change that first line to `#!/bin/zsh` - then when I ran it it spammed my screen with dialog boxes.

Me:

> I want to log the notes contents to the terminal, not display it in a dialog

ChatGPT:

```zsh
#!/bin/zsh

osascript -e 'tell application ""Notes""
   repeat with eachNote in every note
      set noteTitle to the name of eachNote
      set noteBody to the body of eachNote
      set output to noteTitle & ""\n"" & noteBody & ""\n""
      log output
   end repeat
end tell'
```
This really helps!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616347574,Initial proof of concept with ChatGPT,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461230197,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1,1461230197,IC_kwDOJHON9s5XGJp1,9599,simonw,2023-03-09T03:51:36Z,2023-03-09T03:51:36Z,MEMBER,"After a few more rounds I got to this script, which outputs them to a `/tmp/notes.txt` file:

```zsh
#!/bin/zsh

osascript -e '
set notesFile to ""/tmp/notes.txt""
set fileRef to open for access notesFile with write permission
tell application ""Notes""
   repeat with eachNote in every note
      set noteId to the id of eachNote
      set noteTitle to the name of eachNote
      set noteBody to the body of eachNote
      write ""------------------------"" & ""\n"" to fileRef
      write noteId & ""\n"" to fileRef
      write noteTitle & ""\n\n"" to fileRef
      write noteBody & ""\n"" to fileRef
   end repeat
end tell
close access fileRef'
```
Then I wrote this little Python script to load them into a database:
```python
import sqlite_utils
split = b""------------------------\n""
s = open(""/tmp/notes.txt"", ""rb"").read()
notes = [n.decode(""mac_roman"") for n in s.split(split) if n]

cleaned_notes = [{
    ""id"": n.split(""\n"")[0],
    ""title"": n.split(""\n"")[1],
    ""body"": ""\n"".join(n.split(""\n"")[2:]).strip()
} for n in notes]

db = sqlite_utils.Database(""/tmp/notes.db"")
db[""notes""].insert_all(cleaned_notes)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616347574,Initial proof of concept with ChatGPT,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461230436,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1,1461230436,IC_kwDOJHON9s5XGJtk,9599,simonw,2023-03-09T03:51:52Z,2023-03-09T03:51:52Z,MEMBER,This did the job! Next step is to turn that into a Python script.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616347574,Initial proof of concept with ChatGPT,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462962682,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11,1462962682,IC_kwDOJHON9s5XMwn6,9599,simonw,2023-03-09T23:20:35Z,2023-03-09T23:22:41Z,MEMBER,"Here's a query that returns all notes in folder 1, including notes in descendant folders:
```sql
with recursive nested_folders(folder_id, descendant_folder_id) as (
  -- base case: select all immediate children of the root folder
  select id, id from folders where parent is null
  union all
  -- recursive case: select all children of the previous level of nested folders
  select nf.folder_id, f.id from nested_folders nf
  join folders f on nf.descendant_folder_id = f.parent
)
-- Find notes within all descendants of folder 1
select *
from notes
where folder in (
  select descendant_folder_id from nested_folders where folder_id = 1
);
```
With assistance from ChatGPT. Prompts were:

```
SQLite schema:

CREATE TABLE [folders] (
   [id] INTEGER PRIMARY KEY,
   [long_id] TEXT,
   [name] TEXT,
   [parent] INTEGER,
   FOREIGN KEY([parent]) REFERENCES [folders]([id])
);

Write a recursive CTE that returns the following:

folder_id | descendant_folder_id

With a row for every nested child of every folder - so the top level folder has lots of rows
```
Then I tweaked it a bit, then ran this:
```
WITH RECURSIVE nested_folders(folder_id, descendant_folder_id) AS (
  -- base case: select all immediate children of the root folder
  SELECT id, id FROM folders WHERE parent IS NULL
  UNION ALL
  -- recursive case: select all children of the previous level of nested folders
  SELECT nf.folder_id, f.id FROM nested_folders nf
  JOIN folders f ON nf.descendant_folder_id = f.parent
)
-- select all rows from the recursive CTE
SELECT * from notes where folder in (select descendant_folder_id FROM nested_folders where folder_id = 1)

Convert all SQL keywords to lower case, and re-indent
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1618130434,Implement a SQL view to make it easier to query files in a nested folder,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462965256,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11,1462965256,IC_kwDOJHON9s5XMxQI,9599,simonw,2023-03-09T23:22:12Z,2023-03-09T23:22:12Z,MEMBER,"Here's what the CTE from that looks like:

<img width=""469"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/224182888-50f315eb-417f-4fb1-b624-151e1740ffed.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1618130434,Implement a SQL view to make it easier to query files in a nested folder,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462968053,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11,1462968053,IC_kwDOJHON9s5XMx71,9599,simonw,2023-03-09T23:24:01Z,2023-03-09T23:24:01Z,MEMBER,"I improved the readability by removing some unnecessary table aliases:
```sql
with recursive nested_folders(folder_id, descendant_folder_id) as (
  -- base case: select all immediate children of the root folder
  select id, id from folders where parent is null
  union all
  -- recursive case: select all children of the previous level of nested folders
  select nested_folders.folder_id, folders.id from nested_folders
  join folders on nested_folders.descendant_folder_id = folders.parent
)
-- Find notes within all descendants of folder 1
select *
from notes
where folder in (
  select descendant_folder_id from nested_folders where folder_id = 1
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1618130434,Implement a SQL view to make it easier to query files in a nested folder,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461232709,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461232709,IC_kwDOJHON9s5XGKRF,9599,simonw,2023-03-09T03:54:28Z,2023-03-09T03:54:28Z,MEMBER,"I think the AppleScript I want to pass to `osascript` looks like this:
```applescript
tell application ""Notes""
   repeat with eachNote in every note
      set noteId to the id of eachNote
      set noteTitle to the name of eachNote
      set noteBody to the body of eachNote
      log ""------------------------"" & ""\n""
      log noteId & ""\n""
      log noteTitle & ""\n\n""
      log noteBody & ""\n""
   end repeat
end tell
```
But there are a few more properties I'd like to get - created and updated date for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461234311,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461234311,IC_kwDOJHON9s5XGKqH,9599,simonw,2023-03-09T03:56:24Z,2023-03-09T03:56:24Z,MEMBER,"I opened the ""Script Editor"" app on my computer, used Window -> Library to open the Library panel, then clicked on the Notes app there. I got this:

<img width=""1240"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223912876-09e8660e-f59a-4f04-8992-668e09c8d7d4.png"">

So the notes object has these properties:

- name (text) : the name of the note (normally the first line of the body)
- id (text, r/o) : the unique identifier of the note
- container ([folder](applewebdata://621FA8D9-C995-4081-B3B3-149B0EA04C7F#Notes-Suite.folder), r/o) : the folder of the note
- body (text) : the HTML content of the note
- plaintext (text, r/o) : the plaintext content of the note
- creation date (date, r/o) : the creation date of the note
- modification date (date, r/o) : the modification date of the note
- password protected (boolean, r/o) : Is the note password protected?
- shared (boolean, r/o) : Is the note shared?

I'm going to ignore the concept of attachments for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461234591,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461234591,IC_kwDOJHON9s5XGKuf,9599,simonw,2023-03-09T03:56:45Z,2023-03-09T03:56:45Z,MEMBER,"My prototype showed that images embedded in notes come out in the HTML export as bas64 image URLs, which is neat.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461259490,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461259490,IC_kwDOJHON9s5XGQzi,9599,simonw,2023-03-09T04:24:27Z,2023-03-09T04:24:27Z,MEMBER,"Converting AppleScript date strings to ISO format is hard!

https://forum.latenightsw.com/t/formatting-dates/841 has a recipe I'll try:

    set todayISO to (todayDate as «class isot» as string)

Not clear to me how timezones work here. I'm going to ignore them for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461260978,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461260978,IC_kwDOJHON9s5XGRKy,9599,simonw,2023-03-09T04:27:18Z,2023-03-09T04:27:18Z,MEMBER,"Before that conversion:

    Monday, March 6, 2023 at 11:55:15 AM

After:

    2023-03-06T11:55:15","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461262577,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461262577,IC_kwDOJHON9s5XGRjx,9599,simonw,2023-03-09T04:30:00Z,2023-03-09T04:30:00Z,MEMBER,It doesn't have tests yet. I guess I'll need to mock `subprocess` to test this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461285545,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461285545,IC_kwDOJHON9s5XGXKp,9599,simonw,2023-03-09T05:06:24Z,2023-03-09T05:06:24Z,MEMBER,"OK, this works!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/4#issuecomment-1462554175,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4,1462554175,IC_kwDOJHON9s5XLM4_,9599,simonw,2023-03-09T18:19:34Z,2023-03-09T18:19:34Z,MEMBER,It looks like the iteration order is most-recently-modified-first - I tried editing a note a bit further back in my notes app and it was the first one output by `apple-notes-to-sqlite --dump`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616429236,Support incremental updates,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/4#issuecomment-1462556829,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4,1462556829,IC_kwDOJHON9s5XLNid,9599,simonw,2023-03-09T18:20:56Z,2023-03-09T18:20:56Z,MEMBER,"In terms of the UI: I'm tempted to say that the default behaviour is for it to run until it sees a note that it already knows about AND that has matching update/created dates, and then stop.

You can do a full import again ignoring that logic with `apple-notes-to-sqlite notes.db --full`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616429236,Support incremental updates,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/6#issuecomment-1493442956,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6,1493442956,IC_kwDOJHON9s5ZBCGM,14314871,amlestin,2023-04-02T21:20:43Z,2023-04-02T21:25:37Z,NONE,"I'm experiencing something similar. My apostrophes (') turn into (’) and the output is truncated. Hoping to debug next weekend 
<img width=""814"" alt=""Screenshot 2023-04-02 at 4 24 10 PM"" src=""https://user-images.githubusercontent.com/14314871/229379762-f622205a-7bba-4598-b0da-a6b86e5c0a44.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617602868,Character encoding problem,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/6#issuecomment-1508784533,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6,1508784533,IC_kwDOJHON9s5Z7jmV,579727,sirnacnud,2023-04-14T15:22:09Z,2023-04-14T15:22:09Z,NONE,"Just changing the encoding in `extract_notes` to `utf8` seems to fix it for my titles that were messed up.

![Screen Shot 2023-04-14 at 5 14 18 PM](https://user-images.githubusercontent.com/579727/232086062-e7edc4d1-0880-417a-925b-fd6c65b05155.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617602868,Character encoding problem,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462562735,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462562735,IC_kwDOJHON9s5XLO-v,9599,simonw,2023-03-09T18:23:56Z,2023-03-09T18:25:22Z,MEMBER,"From the Script Editor library docs:

A note has a:

> - `container` (folder), r/o) : the folder of the note

Here's what a folder looks like:

> folder n : a folder containing notes
> elements:
>
> - contains folders, notes; contained by application, accounts, folders.
>
> properties:
>
> - `name` (text) : the name of the folder
> - `id` (text, r/o) : the unique identifier of the folder
> - `shared` (boolean, r/o) : Is the folder shared?
> - `container` (account or folder, r/o) : the container of the folder
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462564717,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462564717,IC_kwDOJHON9s5XLPdt,9599,simonw,2023-03-09T18:25:39Z,2023-03-09T18:25:39Z,MEMBER,So it looks like folders can be hierarchical?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462570187,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462570187,IC_kwDOJHON9s5XLQzL,9599,simonw,2023-03-09T18:30:24Z,2023-03-09T18:30:24Z,MEMBER,"I used ChatGPT to write this:
```
osascript -e 'tell application ""Notes""
    set allFolders to folders
    repeat with aFolder in allFolders
        set folderId to id of aFolder
        set folderName to name of aFolder
        set folderContainer to container of aFolder
        set folderContainerName to name of folderContainer
        log ""Folder ID: "" & folderId
        log ""Folder Name: "" & folderName
        log ""Folder Container: "" & folderContainerName
        log "" ""
        --check for nested folders
        if count of folders of aFolder > 0 then
            set nestedFolders to folders of aFolder
            repeat with aNestedFolder in nestedFolders
                set nestedFolderId to id of aNestedFolder
                set nestedFolderName to name of aNestedFolder
                set nestedFolderContainer to container of aNestedFolder
                set nestedFolderContainerName to name of nestedFolderContainer
                log ""    Nested Folder ID: "" & nestedFolderId
                log ""    Nested Folder Name: "" & nestedFolderName
                log ""    Nested Folder Container: "" & nestedFolderContainerName
                log "" ""
            end repeat
        end if
    end repeat
end tell
'
```
Which for my account output this:
```
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113
Folder Name: Blog posts
Folder Container: iCloud
 
    Nested Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
    Nested Folder Name: Nested inside blog posts
    Nested Folder Container: Blog posts
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698
Folder Name: JSK
Folder Container: iCloud
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
Folder Name: Nested inside blog posts
Folder Container: Blog posts
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526
Folder Name: New Folder
Folder Container: iCloud
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839
Folder Name: New Folder 1
Folder Container: iCloud
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2
Folder Name: Notes
Folder Container: iCloud
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059
Folder Name: Quick Notes
Folder Container: iCloud
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283
Folder Name: UK Christmas 2022
Folder Container: iCloud
```
So I think the correct approach here is to run code at the start to list all of the folders (no need to do fancy recursion though, just a flat list with the parent containers is enough) and create a model of that hierarchy in SQLite.

Then when I import notes I can foreign key reference them back to their containing folder.

I'm tempted to use `rowid` for the foreign keys because the official IDs are pretty long.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462682795,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462682795,IC_kwDOJHON9s5XLsSr,9599,simonw,2023-03-09T19:52:20Z,2023-03-09T19:52:44Z,MEMBER,"Created through several rounds with ChatGPT (including hints like ""rewrite that using setdefault()""):
```python
def topological_sort(nodes):
    children = {}
    for node in nodes:
        parent_id = node[""parent""]
        if parent_id is not None:
            children.setdefault(parent_id, []).append(node)

    def traverse(node, result):
        result.append(node)
        if node[""id""] in children:
            for child in children[node[""id""]]:
                traverse(child, result)

    sorted_data = []

    for node in nodes:
        if node[""parent""] is None:
            traverse(node, sorted_data)

    return sorted_data
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462691466,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462691466,IC_kwDOJHON9s5XLuaK,9599,simonw,2023-03-09T19:59:52Z,2023-03-09T19:59:52Z,MEMBER,"Improved script:
```zsh
osascript -e 'tell application ""Notes""
    set allFolders to folders
    repeat with aFolder in allFolders
        set folderId to id of aFolder
        set folderName to name of aFolder
        set folderContainer to container of aFolder
        if class of folderContainer is folder then
            set folderContainerId to id of folderContainer
        else
            set folderContainerId to """"
        end if
        log ""ID: "" & folderId
        log ""Name: "" & folderName
        log ""Container: "" & folderContainerId
        log "" ""
    end repeat
end tell
'
```
```
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113
Name: Blog posts
Container: 
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698
Name: JSK
Container: 
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
Name: Nested inside blog posts
Container: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526
Name: New Folder
Container: 
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839
Name: New Folder 1
Container: 
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2
Name: Notes
Container: 
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059
Name: Quick Notes
Container: 
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283
Name: UK Christmas 2022
Container: 
```
I filtered out things where the parent was an account and not a folder using `if class of folderContainer is folder then`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462693867,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462693867,IC_kwDOJHON9s5XLu_r,9599,simonw,2023-03-09T20:01:39Z,2023-03-09T20:02:11Z,MEMBER,"My `folders` table will have:

- `id` - rowid
- `long_id` - that long unique string ID
- `name` - the name
- `parent` - foreign key to `id`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/8#issuecomment-1468898285,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/8,1468898285,IC_kwDOJHON9s5XjZvt,41546558,RhetTbull,2023-03-14T22:00:21Z,2023-03-14T22:00:21Z,NONE,"Well that's embarrassing. I made a fork using macnotesapp and it's actually slower. This is because the Scripting Bridge sometimes fails to return the folder and thus macnotesapp resorts to AppleScript in this situation. The repeated AppleScript calls on a large library are slower than your ""slurp it all in"" approach.  I've got some ideas about how to improve this--will make another attempt if I can fix the issues.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617823309,Increase performance using macnotesapp,
https://github.com/dogsheep/dogsheep-beta/issues/10#issuecomment-686238498,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/10,686238498,MDEyOklzc3VlQ29tbWVudDY4NjIzODQ5OA==,9599,simonw,2020-09-03T04:05:05Z,2020-09-03T04:05:05Z,MEMBER,Since the first two categories are `created` and `saved` this one should be called `received`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691557547,Category 3: received,
https://github.com/dogsheep/dogsheep-beta/issues/11#issuecomment-686618669,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/11,686618669,MDEyOklzc3VlQ29tbWVudDY4NjYxODY2OQ==,9599,simonw,2020-09-03T16:47:34Z,2020-09-03T16:53:25Z,MEMBER,I think a `is_public` integer column which defaults to 0 would be good here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",692125110,Public / Private mechanism,
https://github.com/dogsheep/dogsheep-beta/issues/13#issuecomment-686774592,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/13,686774592,MDEyOklzc3VlQ29tbWVudDY4Njc3NDU5Mg==,9599,simonw,2020-09-03T21:30:21Z,2020-09-03T21:30:21Z,MEMBER,"This is partially supported: the custom search SQL we run doesn't escape them, but the `?_search` used to calculate facet counts does. So this is a bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",692386625,Support advanced FTS queries,
https://github.com/dogsheep/dogsheep-beta/issues/15#issuecomment-695124698,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/15,695124698,MDEyOklzc3VlQ29tbWVudDY5NTEyNDY5OA==,9599,simonw,2020-09-18T23:17:38Z,2020-09-18T23:17:38Z,MEMBER,This can be part of the demo instance in #6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694136490,Add a bunch of config examples,
https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-694548909,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16,694548909,MDEyOklzc3VlQ29tbWVudDY5NDU0ODkwOQ==,9599,simonw,2020-09-17T23:15:09Z,2020-09-17T23:15:09Z,MEMBER,"I have sort by date now, #21.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694493566,Timeline view,
https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695851036,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16,695851036,MDEyOklzc3VlQ29tbWVudDY5NTg1MTAzNg==,9599,simonw,2020-09-20T23:34:57Z,2020-09-20T23:34:57Z,MEMBER,Really basic starting point is to add facet by date.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694493566,Timeline view,
https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695877627,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16,695877627,MDEyOklzc3VlQ29tbWVudDY5NTg3NzYyNw==,9599,simonw,2020-09-21T02:42:29Z,2020-09-21T02:42:29Z,MEMBER,"Fun twist: assuming `timestamp` is always stored as UTC, I need the interface to be timezone aware so I can see e.g. everything from 4th July 2020 in the San Francisco timezone definition of 4th July 2020.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694493566,Timeline view,
https://github.com/dogsheep/dogsheep-beta/issues/17#issuecomment-687880459,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17,687880459,MDEyOklzc3VlQ29tbWVudDY4Nzg4MDQ1OQ==,9599,simonw,2020-09-06T19:36:32Z,2020-09-06T19:36:32Z,MEMBER,At some point I may even want to support search types which are indexed from (and inflated from) more than one database file. I'm going to ignore that for the moment though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694500679,"Rename ""table"" to ""type""",
https://github.com/dogsheep/dogsheep-beta/issues/17#issuecomment-689226390,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17,689226390,MDEyOklzc3VlQ29tbWVudDY4OTIyNjM5MA==,9599,simonw,2020-09-09T00:36:07Z,2020-09-09T00:36:07Z,MEMBER,"Alternative names:

- type
- record_type
- doctype

I think `type` is right. It matches what Elasticsearch used to call their equivalent of this (before they removed the feature!). https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694500679,"Rename ""table"" to ""type""",
https://github.com/dogsheep/dogsheep-beta/issues/18#issuecomment-688622995,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18,688622995,MDEyOklzc3VlQ29tbWVudDY4ODYyMjk5NQ==,9599,simonw,2020-09-08T05:15:21Z,2020-09-08T05:15:21Z,MEMBER,"Alternatively it could run as it does now but add a `DELETE FROM index1.search_index WHERE key not in (select key from ...)`.

I'm not sure which would be more efficient.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695553522,Deleted records stay in the search index,
https://github.com/dogsheep/dogsheep-beta/issues/18#issuecomment-688623097,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18,688623097,MDEyOklzc3VlQ29tbWVudDY4ODYyMzA5Nw==,9599,simonw,2020-09-08T05:15:51Z,2020-09-08T05:15:51Z,MEMBER,"I'm inclined to go with the first, simpler option. I have longer term plans for efficient incremental index updates based on clever trickery with triggers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695553522,Deleted records stay in the search index,
https://github.com/dogsheep/dogsheep-beta/issues/19#issuecomment-688625430,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19,688625430,MDEyOklzc3VlQ29tbWVudDY4ODYyNTQzMA==,9599,simonw,2020-09-08T05:24:50Z,2020-09-08T05:24:50Z,MEMBER,"I thought about allowing tables to define a incremental indexing SQL query - maybe something that can return just records touched in the past hour, or records since a recorded ""last indexed record"" value.

The problem with this is deletes - if you delete a record, how does the indexer know to remove it? See #18  - that's already caused problems.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695556681,Figure out incremental re-indexing,
https://github.com/dogsheep/dogsheep-beta/issues/19#issuecomment-688626037,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19,688626037,MDEyOklzc3VlQ29tbWVudDY4ODYyNjAzNw==,9599,simonw,2020-09-08T05:27:07Z,2020-09-08T05:27:07Z,MEMBER,"A really clever way to do this would be with triggers. The indexer script would add triggers to each of the database tables that it is indexing - each in their own database.

Those triggers would then maintain a `_index_queue_` table. This table would record the primary key of rows that are added, modified or deleted. The indexer could then work by reading through the `_index_queue_` table, re-indexing (or deleting) just the primary keys listed there, and then emptying the queue once it has finished.

This would add a small amount of overhead to insert/update/delete queries run against the table. My hunch is that the overhead would be miniscule, but I could still allow people to opt-out for tables that are so high traffic that this would matter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695556681,Figure out incremental re-indexing,
https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685115519,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2,685115519,MDEyOklzc3VlQ29tbWVudDY4NTExNTUxOQ==,9599,simonw,2020-09-01T20:31:57Z,2020-09-01T20:31:57Z,MEMBER,"Actually this doesn't work: you can't turn on stemming for specific tables, because all of the content goes into a single `search_index` table which is configured the same way.

So stemming needs to be a global option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689809225,Apply porter stemming,
https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685121074,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2,685121074,MDEyOklzc3VlQ29tbWVudDY4NTEyMTA3NA==,9599,simonw,2020-09-01T20:42:00Z,2020-09-01T20:42:00Z,MEMBER,Documentation at the bottom of the Usage section here: https://github.com/dogsheep/dogsheep-beta/blob/0.2/README.md#usage,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689809225,Apply porter stemming,
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551406,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694551406,MDEyOklzc3VlQ29tbWVudDY5NDU1MTQwNg==,9599,simonw,2020-09-17T23:22:07Z,2020-09-17T23:22:07Z,MEMBER,"Neat, I can debug this with the new `--pdb` option:

    datasette . --get '/-/beta?q=pycon&sort=oldest' --pdb
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551646,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694551646,MDEyOklzc3VlQ29tbWVudDY5NDU1MTY0Ng==,9599,simonw,2020-09-17T23:22:48Z,2020-09-17T23:22:48Z,MEMBER,"Looks like its happening in a Jinja fragment template for one of the results:
```
  /Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/__init__.py(169)process_results()
-> output = compiled.render({**result, **{""json"": json}})
  /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/asyncsupport.py(71)render()
-> return original_render(self, *args, **kwargs)
  /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(1090)render()
-> self.environment.handle_exception()
  /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(832)handle_exception()
-> reraise(*rewrite_traceback_stack(source=source))
  /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/_compat.py(28)reraise()
-> raise value.with_traceback(tb)
  <template>(5)top-level template code()
> /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads()
-> raise TypeError(f'the JSON object must be str, bytes or bytearray, '
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552393,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694552393,MDEyOklzc3VlQ29tbWVudDY5NDU1MjM5Mw==,9599,simonw,2020-09-17T23:25:01Z,2020-09-17T23:25:17Z,MEMBER,"Ran `locals()` In the debugger:
`{'range': <class 'range'>, 'dict': <class 'dict'>, 'lipsum': <function generate_lorem_ipsum at 0x10aeff430>, 'cycler': <class 'jinja2.utils.Cycler'>, 'joiner': <class 'jinja2.utils.Joiner'>, 'namespace': <class 'jinja2.utils.Namespace'>, 'rank': -9.383801886431414, 'rowid': 14297, 'type': 'twitter.db/tweets', 'key': '312658917933076480', 'title': 'Tweet by @chrisstreeter', 'category': 2, 'timestamp': '2013-03-15T20:17:49+00:00', 'search_1': '@simonw are you at pycon? Would love to meet you.', 'display': {'avatar_url': 'https://pbs.twimg.com/profile_images/806275088597204993/38yLHfJi_normal.jpg', 'user_name': 'Chris Streeter', 'screen_name': 'chrisstreeter', 'followers_count': 280, 'tweet_id': 312658917933076480, 'created_at': '2013-03-15T20:17:49+00:00', 'full_text': '@simonw are you at pycon? Would love to meet you.', 'media_urls_2': '[]', 'media_urls': '[]'}, 'json': <module 'json' from '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py'>}`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552681,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694552681,MDEyOklzc3VlQ29tbWVudDY5NDU1MjY4MQ==,9599,simonw,2020-09-17T23:25:54Z,2020-09-17T23:25:54Z,MEMBER,"This is the template fragment it's rendering:
```html+jinja
            <div style=""overflow: hidden;"">
              <p>Tweet by <a href=""https://twitter.com/{{ display.screen_name }}"">@{{ display.screen_name }}</a> ({{ display.user_name }}, {{ ""{:,}"".format(display.followers_count or 0) }} followers)
                on <a href=""https://twitter.com/{{ display.screen_name }}/status/{{ display.tweet_id }}"">{{ display.created_at }}</a></p>
              </p>
              <blockquote>{{ display.full_text }}</blockquote>
              {% if display.media_urls and json.loads(display.media_urls) %}
                {% for url in json.loads(display.media_urls) %}
                  <img src=""{{ url }}"" style=""height: 200px;"">
                {% endfor %}
              {% endif %}
            </div>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694553579,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694553579,MDEyOklzc3VlQ29tbWVudDY5NDU1MzU3OQ==,9599,simonw,2020-09-17T23:28:37Z,2020-09-17T23:28:37Z,MEMBER,"More investigation in pdb:
```
(dogsheep-beta) dogsheep-beta % datasette . --get '/-/beta?q=pycon&sort=oldest' --pdb
> /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads()
-> raise TypeError(f'the JSON object must be str, bytes or bytearray, '
(Pdb) list
336  	        if s.startswith('\ufeff'):
337  	            raise JSONDecodeError(""Unexpected UTF-8 BOM (decode using utf-8-sig)"",
338  	                                  s, 0)
339  	    else:
340  	        if not isinstance(s, (bytes, bytearray)):
341  ->	            raise TypeError(f'the JSON object must be str, bytes or bytearray, '
342  	                            f'not {s.__class__.__name__}')
343  	        s = s.decode(detect_encoding(s), 'surrogatepass')
344  	
345  	    if ""encoding"" in kw:
346  	        import warnings
(Pdb) bytes
<class 'bytes'>
(Pdb) locals()['s']
Undefined
(Pdb) type(locals()['s'])
<class 'jinja2.runtime.Undefined'>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694554584,MDEyOklzc3VlQ29tbWVudDY5NDU1NDU4NA==,9599,simonw,2020-09-17T23:31:25Z,2020-09-17T23:31:25Z,MEMBER,"I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694557425,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694557425,MDEyOklzc3VlQ29tbWVudDY5NDU1NzQyNQ==,9599,simonw,2020-09-17T23:41:01Z,2020-09-17T23:41:01Z,MEMBER,I removed all of the `json.loads()` calls and I'm still getting that `Undefined` error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-695113871,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,695113871,MDEyOklzc3VlQ29tbWVudDY5NTExMzg3MQ==,9599,simonw,2020-09-18T22:30:17Z,2020-09-18T22:30:17Z,MEMBER,"I think I know what's going on here:

https://github.com/dogsheep/dogsheep-beta/blob/0f1b951c5131d16f3c8559a8e4d79ed5c559e3cb/dogsheep_beta/__init__.py#L166-L171

This is a logic bug - the `compiled` variable could be the template from the previous loop!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695108895,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25,695108895,MDEyOklzc3VlQ29tbWVudDY5NTEwODg5NQ==,9599,simonw,2020-09-18T22:11:32Z,2020-09-18T22:11:32Z,MEMBER,"I'm going to make this a new plugin configuration setting, `template_debug`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",704685890,template_debug mechanism,
https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695109140,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25,695109140,MDEyOklzc3VlQ29tbWVudDY5NTEwOTE0MA==,9599,simonw,2020-09-18T22:12:20Z,2020-09-18T22:12:20Z,MEMBER,Documented here: https://github.com/dogsheep/dogsheep-beta/blob/534fc9689227eba70e69a45da0cee5820bbda9e1/README.md#datasette-plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",704685890,template_debug mechanism,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855646,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695855646,MDEyOklzc3VlQ29tbWVudDY5NTg1NTY0Ng==,9599,simonw,2020-09-21T00:16:11Z,2020-09-21T00:16:11Z,MEMBER,"Should I do this with offset/limit or should I do proper keyset pagination?

I think keyset because then it will work well for the full search interface with no filters or search string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855723,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695855723,MDEyOklzc3VlQ29tbWVudDY5NTg1NTcyMw==,9599,simonw,2020-09-21T00:16:52Z,2020-09-21T00:17:53Z,MEMBER,"It feels a bit weird to implement keyset pagination against results sorted by `rank` because the ranks could change substantially if the search index gets updated while the user is paginating.

I may just ignore that though. If you want reliable pagination you can get it by sorting by date. Maybe it doesn't even make sense to offer pagination if you sort by relevance?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856398,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695856398,MDEyOklzc3VlQ29tbWVudDY5NTg1NjM5OA==,9599,simonw,2020-09-21T00:22:20Z,2020-09-21T00:22:20Z,MEMBER,I'm going to try for keyset pagination sorted by relevance just as a learning exercise.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856967,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695856967,MDEyOklzc3VlQ29tbWVudDY5NTg1Njk2Nw==,9599,simonw,2020-09-21T00:26:59Z,2020-09-21T00:26:59Z,MEMBER,It's a shame Datasette doesn't currently have an easy way to implement sorted-by-rank keyset-paginated using a TableView or QueryView. I'll have to do this using the custom SQL query constructed in the plugin: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L8-L43,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695875274,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695875274,MDEyOklzc3VlQ29tbWVudDY5NTg3NTI3NA==,9599,simonw,2020-09-21T02:28:58Z,2020-09-21T02:28:58Z,MEMBER,Datasette's implementation is complex because it has to support compound primary keys: https://github.com/simonw/datasette/blob/a258339a935d8d29a95940ef1db01e98bb85ae63/datasette/utils/__init__.py#L88-L114 - but that's not something that's needed for dogsheep-beta.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879237,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695879237,MDEyOklzc3VlQ29tbWVudDY5NTg3OTIzNw==,9599,simonw,2020-09-21T02:53:29Z,2020-09-21T02:53:29Z,MEMBER,"If previous page ended at `2018-02-11T16:32:53+00:00`:
```sql
select
  search_index.rowid,
  search_index.type,
  search_index.key,
  search_index.title,
  search_index.category,
  search_index.timestamp,
  search_index.search_1
from
  search_index
 where 
  date(""timestamp"") = '2018-02-11'
 and timestamp < '2018-02-11T16:32:53+00:00'
order by
  search_index.timestamp desc, rowid
limit 41
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879531,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695879531,MDEyOklzc3VlQ29tbWVudDY5NTg3OTUzMQ==,9599,simonw,2020-09-21T02:55:28Z,2020-09-21T02:55:54Z,MEMBER,"Actually for the tie-breaker it should be something like https://latest.datasette.io/fixtures?sql=select+pk%2C+created%2C+planet_int%2C+on_earth%2C+state%2C+city_id%2C+neighborhood%2C+tags%2C+complex_array%2C+distinct_some_null+from+facetable+where+%28created+%3E+%3Ap1+or+%28created+%3D+%3Ap1+and+%28%28pk+%3E+%3Ap0%29%29%29%29+order+by+created%2C+pk+limit+11&p0=10&p1=2019-01-16+08%3A00%3A00
```sql
where
  (
    created > :p1
    or (
      created = :p1
      and ((pk > :p0))
    )
  )
```
But with `rowid` and `timestamp` in place of `pk` and `created`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/28#issuecomment-711089647,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/28,711089647,MDEyOklzc3VlQ29tbWVudDcxMTA4OTY0Nw==,9599,simonw,2020-10-17T22:43:13Z,2020-10-17T22:43:13Z,MEMBER,"Since my personal Dogsheep uses Datasette authentication, I'm going to need to pass through cookies. https://github.com/simonw/datasette/issues/1020 will solve that in the future but for now I need to solve it explicitly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723861683,Switch to using datasette.client,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-712266834,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,712266834,MDEyOklzc3VlQ29tbWVudDcxMjI2NjgzNA==,9599,simonw,2020-10-19T16:01:23Z,2020-10-19T16:01:23Z,MEMBER,Might just be a documented pattern for how to configure this in YAML templates.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747029636,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,747029636,MDEyOklzc3VlQ29tbWVudDc0NzAyOTYzNg==,9599,simonw,2020-12-16T21:14:03Z,2020-12-16T21:14:03Z,MEMBER,"I think I can do this as a cunning trick in `display_sql`. Consider this example query: https://til.simonwillison.net/tils?sql=select%0D%0A++path%2C%0D%0A++snippet%28til_fts%2C+-1%2C+%27b4de2a49c8%27%2C+%278c94a2ed4b%27%2C+%27...%27%2C+60%29+as+snippet%0D%0Afrom%0D%0A++til%0D%0A++join+til_fts+on+til.rowid+%3D+til_fts.rowid%0D%0Awhere%0D%0A++til_fts+match+escape_fts%28%3Aq%29%0D%0A++and+path+%3D+%27asgi_lifespan-test-httpx.md%27%0D%0A&q=pytest

```sql
select
  path,
  snippet(til_fts, -1, 'b4de2a49c8', '8c94a2ed4b', '...', 60) as snippet
from
  til
  join til_fts on til.rowid = til_fts.rowid
where
  til_fts match escape_fts(:q)
  and path = 'asgi_lifespan-test-httpx.md'
```
The `and path = 'asgi_lifespan-test-httpx.md'` bit means we only get back a specific document - but the snippet highlighting is applied to it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747030964,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,747030964,MDEyOklzc3VlQ29tbWVudDc0NzAzMDk2NA==,9599,simonw,2020-12-16T21:14:54Z,2020-12-16T21:14:54Z,MEMBER,"To do this I'll need the search term to be passed to the `display_sql` SQL query:
https://github.com/dogsheep/dogsheep-beta/blob/4890ec87b5e2ec48940f32c9ad1f5aae25c75a4d/dogsheep_beta/__init__.py#L164-L171","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747031608,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,747031608,MDEyOklzc3VlQ29tbWVudDc0NzAzMTYwOA==,9599,simonw,2020-12-16T21:15:18Z,2020-12-16T21:15:18Z,MEMBER,Should I pass any other details to the `display_sql` here as well?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747034481,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,747034481,MDEyOklzc3VlQ29tbWVudDc0NzAzNDQ4MQ==,9599,simonw,2020-12-16T21:17:05Z,2020-12-16T21:17:05Z,MEMBER,I'm just going to add `q` for the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-684250044,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3,684250044,MDEyOklzc3VlQ29tbWVudDY4NDI1MDA0NA==,9599,simonw,2020-09-01T05:01:09Z,2020-09-01T05:01:23Z,MEMBER,Maybe this starts out as a custom templated canned query.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689810340,"Datasette plugin to provide custom page for running faceted, ranked searches",
https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-685961809,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3,685961809,MDEyOklzc3VlQ29tbWVudDY4NTk2MTgwOQ==,9599,simonw,2020-09-02T19:54:24Z,2020-09-02T19:54:24Z,MEMBER,"This should implement search highlighting too, as seen on https://til.simonwillison.net/til/search?q=cloud

<img width=""1029"" alt=""TIL_search__cloud"" src=""https://user-images.githubusercontent.com/9599/92029959-32c6a300-ed1b-11ea-8b5e-971950980c38.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689810340,"Datasette plugin to provide custom page for running faceted, ranked searches",
https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-686689612,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3,686689612,MDEyOklzc3VlQ29tbWVudDY4NjY4OTYxMg==,9599,simonw,2020-09-03T18:44:20Z,2020-09-03T18:44:20Z,MEMBER,Facets are now displayed but selecting them doesn't work yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689810340,"Datasette plugin to provide custom page for running faceted, ranked searches",
https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426501,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426501,MDEyOklzc3VlQ29tbWVudDc0ODQyNjUwMQ==,9599,simonw,2020-12-19T06:12:22Z,2020-12-19T06:12:22Z,MEMBER,I deliberately added support for advanced FTS in https://github.com/dogsheep/dogsheep-beta/commit/cbb2491b85d7ff416d6d429b60109e6c2d6d50b9 for #13 but that's the cause of this bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error",
https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426581,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426581,MDEyOklzc3VlQ29tbWVudDc0ODQyNjU4MQ==,9599,simonw,2020-12-19T06:13:17Z,2020-12-19T06:13:17Z,MEMBER,"One fix for this could be to try running the raw query, but if it throws an error run it again with the query escaped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error",
https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426663,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426663,MDEyOklzc3VlQ29tbWVudDc0ODQyNjY2Mw==,9599,simonw,2020-12-19T06:14:06Z,2020-12-19T06:14:06Z,MEMBER,Looks like I already do that here: https://github.com/dogsheep/dogsheep-beta/blob/9ba4401017ac24ffa3bc1db38e0910ea49de7616/dogsheep_beta/__init__.py#L141-L146,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error",
https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426877,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426877,MDEyOklzc3VlQ29tbWVudDc0ODQyNjg3Nw==,9599,simonw,2020-12-19T06:16:11Z,2020-12-19T06:16:11Z,MEMBER,"Here's why:

    if ""fts5"" in str(e):

But the error being raised here is:

    sqlite3.OperationalError: no such column: to

I'm going to attempt the escaped on on every error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error",
https://github.com/dogsheep/dogsheep-beta/issues/4#issuecomment-684395444,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/4,684395444,MDEyOklzc3VlQ29tbWVudDY4NDM5NTQ0NA==,9599,simonw,2020-09-01T06:00:03Z,2020-09-01T06:00:03Z,MEMBER,I ran `sqlite-utils optimize beta.db` against my test DB and the size reduced from 183M to 176M - and a 450ms search ran in 359ms. So not a huge improvement but still worthwhile.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689839399,Optimize the FTS table,
https://github.com/dogsheep/dogsheep-beta/issues/5#issuecomment-686689366,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/5,686689366,MDEyOklzc3VlQ29tbWVudDY4NjY4OTM2Ng==,9599,simonw,2020-09-03T18:43:50Z,2020-09-03T18:43:50Z,MEMBER,No longer needed thanks to #9,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689847361,Add a context column that's not searchable,
https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685895540,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685895540,MDEyOklzc3VlQ29tbWVudDY4NTg5NTU0MA==,9599,simonw,2020-09-02T17:46:44Z,2020-09-02T17:46:44Z,MEMBER,"Some opet questions about this:

- Should I restrict to two exclusive categories here, or should I have a generic category mechanism that can be expanded to more than two?
- Should an item be able to exist in more than one category? Do I want to be able to mark an indexed item as both by-me and liked-by-me for example? This question is more interesting if the number of categories is greater than two.
- How should this be modeled? Single column, multiple boolean columns, JSON array, m2m against separate table?
- What's the best way to make this performant","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""",
https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685962280,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685962280,MDEyOklzc3VlQ29tbWVudDY4NTk2MjI4MA==,9599,simonw,2020-09-02T19:55:26Z,2020-09-02T19:59:58Z,MEMBER,"Relevant: https://charlesleifer.com/blog/a-tour-of-tagging-schemas-many-to-many-bitmaps-and-more/

SQLite supports bitwise operators Binary AND (&) and Binary OR (|) - I could try those. Not sure how they interact with indexes though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""",
https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685965516,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685965516,MDEyOklzc3VlQ29tbWVudDY4NTk2NTUxNg==,9599,simonw,2020-09-02T20:01:54Z,2020-09-02T20:01:54Z,MEMBER,"Relevant post: https://sqlite.org/forum/forumpost/9f06fedaa5 - drh says:

> Indexes are one-to-one. There is one entry in the index for each row in the table.
>
> You are asking for an index that is many-to-one - multiple index entries for each table row.
>
> A Full-Text Index is basically a many-to-one index. So if all of your array entries really are words, you could probably get this to work using a Full-Text Index.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""",
https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966361,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685966361,MDEyOklzc3VlQ29tbWVudDY4NTk2NjM2MQ==,9599,simonw,2020-09-02T20:03:29Z,2020-09-02T20:03:41Z,MEMBER,"I'm going to implement the first version of this as an indexed integer `category` column which has 1 for ""about me"" and 2 for ""liked by me"" - and space for other category numerals in the future, albeit a row can only belong to one category.

I'll think about a full tagging system separately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""",
https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966707,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685966707,MDEyOklzc3VlQ29tbWVudDY4NTk2NjcwNw==,9599,simonw,2020-09-02T20:04:08Z,2020-09-02T20:04:08Z,MEMBER,I'll make `category` a foreign key to a `categories` table so Datasette can automatically show the `name` column.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""",
https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685970384,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685970384,MDEyOklzc3VlQ29tbWVudDY4NTk3MDM4NA==,9599,simonw,2020-09-02T20:11:41Z,2020-09-02T20:11:59Z,MEMBER,"Default categories:

- 1 = created
- 2 = saved","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""",
https://github.com/dogsheep/dogsheep-beta/issues/8#issuecomment-685960072,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/8,685960072,MDEyOklzc3VlQ29tbWVudDY4NTk2MDA3Mg==,9599,simonw,2020-09-02T19:50:47Z,2020-09-02T19:50:47Z,MEMBER,"This doesn't actually help, because the Datasette table view page doesn't then support adding the `where search_index_fts match :query` bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691369691,Create a view for running faceted searches,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686153967,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686153967,MDEyOklzc3VlQ29tbWVudDY4NjE1Mzk2Nw==,9599,simonw,2020-09-03T00:17:16Z,2020-09-03T00:17:55Z,MEMBER,"Maybe I can take advantage of https://sqlite.org/np1queryprob.html here - I could define a SQL query for fetching the ""display"" version of each item, and include a Jinja template fragment in the configuration as well. Maybe something like this:
```yaml
photos.db:
    photos_with_apple_metadata:
        sql: |-
            select
              sha256 as key,
              'Photo in ' || coalesce(place_city, 'unknown') as title,
              (
                select
                  group_concat(normalized_string, ' ')
                from
                  labels
                where
                  labels.uuid = photos_with_apple_metadata.uuid
              ) as search_1,
              date as timestamp,
              1 as category
            from
              photos_with_apple_metadata
        display_sql: |-
            select
              sha256, place_city, date
            from photos_with_apple_metadata
            where sha256 = :key
        display: |-
          <img src=""https://photos.simonwillison.net/i/{{ display.sha256 }}.jpeg?w=600"">
          <p>Taken in {{ display.place_city }} on {{ display.date }}</p>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686154486,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686154486,MDEyOklzc3VlQ29tbWVudDY4NjE1NDQ4Ng==,9599,simonw,2020-09-03T00:18:54Z,2020-09-03T00:18:54Z,MEMBER,"`display_sql` could be optional. If it's not defined, a `row` object is passed to the template which is the row that's stored in `search_index`. If `display_sql` IS defined then it's executed and the result is made available as a `display` object in addition to the `row` object.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686154627,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686154627,MDEyOklzc3VlQ29tbWVudDY4NjE1NDYyNw==,9599,simonw,2020-09-03T00:19:22Z,2020-09-03T00:19:22Z,MEMBER,If this performs well enough (100 displayed items will be 100 extra `display_sql` calls) then I'll go with this as the design for the feature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686158454,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686158454,MDEyOklzc3VlQ29tbWVudDY4NjE1ODQ1NA==,9599,simonw,2020-09-03T00:32:42Z,2020-09-03T00:32:42Z,MEMBER,"If this turns out to be too inefficient I could add a `display` text column to the `search_index` table which is designed to be populated with arbitrary JSON by the indexing query, which can then be used to render the template fragment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686163754,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686163754,MDEyOklzc3VlQ29tbWVudDY4NjE2Mzc1NA==,9599,simonw,2020-09-03T00:46:21Z,2020-09-03T00:46:21Z,MEMBER,"Challenge: the `dogsheep-beta.yml` configuration file that is passed to the `dogsheep-beta index` command needs to also be made available to Datasette itself, so that it can read the configuration.

Let's say it can either be duplicated in the `plugins` configuration block of the `metadata.yml` OR you can do this in `metadata.yml`:

```yaml
plugins:
  dogsheep-beta:
    config_file: dogsheep-beta.yml
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686688963,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686688963,MDEyOklzc3VlQ29tbWVudDY4NjY4ODk2Mw==,9599,simonw,2020-09-03T18:42:59Z,2020-09-03T18:42:59Z,MEMBER,I'm pleased with how this works now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686689122,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686689122,MDEyOklzc3VlQ29tbWVudDY4NjY4OTEyMg==,9599,simonw,2020-09-03T18:43:20Z,2020-09-03T18:43:20Z,MEMBER,Needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686767208,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686767208,MDEyOklzc3VlQ29tbWVudDY4Njc2NzIwOA==,9599,simonw,2020-09-03T21:12:14Z,2020-09-03T21:12:14Z,MEMBER,Documentation: https://github.com/dogsheep/dogsheep-beta/blob/0.4/README.md#custom-results-display,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623193947,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623193947,MDEyOklzc3VlQ29tbWVudDYyMzE5Mzk0Nw==,9599,simonw,2020-05-03T22:36:17Z,2020-05-03T22:36:17Z,MEMBER,"I'm going to use [osxphotos](https://github.com/RhetTbull/osxphotos) for this.

Since I've already got code to upload photos and insert them into a table based on their `sha256` hash, my first go at this will be to import data using the tool and foreign-key it to the `sha256` hash in the existing table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623195197,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623195197,MDEyOklzc3VlQ29tbWVudDYyMzE5NTE5Nw==,9599,simonw,2020-05-03T22:44:33Z,2020-05-03T22:44:33Z,MEMBER,"Command will be this:

    $ photos-to-sqlite apple-photos photos.db

This will populate a `apple_photos` table with the data imported by the `osxphotos` library, plus the calculated sha256.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198653,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623198653,MDEyOklzc3VlQ29tbWVudDYyMzE5ODY1Mw==,9599,simonw,2020-05-03T23:09:57Z,2020-05-03T23:09:57Z,MEMBER,"For locations: I'll add `place_x` columns for all of these:
```
(Pdb) photo.place.address._asdict()
{'street': None, 'sub_locality': None, 'city': 'Loreto', 'sub_administrative_area': 'Loreto', 'state_province': 'BCS', 'postal_code': None, 'country': 'Mexico', 'iso_country_code': 'MX'}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198986,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623198986,MDEyOklzc3VlQ29tbWVudDYyMzE5ODk4Ng==,9599,simonw,2020-05-03T23:12:31Z,2020-05-03T23:12:46Z,MEMBER,"To get the taken date in UTC:
```
from datetime import timezone
(Pdb) photo.date.astimezone(timezone.utc).isoformat()
'2018-02-13T20:21:31.620000+00:00'
(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(""."")
['2018-02-13T20:21:31', '620000+00:00']
(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(""."")[0]
'2018-02-13T20:21:31'
(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(""."")[0] + ""+00:00""
'2018-02-13T20:21:31+00:00'

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199214,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623199214,MDEyOklzc3VlQ29tbWVudDYyMzE5OTIxNA==,9599,simonw,2020-05-03T23:14:08Z,2020-05-03T23:14:08Z,MEMBER,"Albums have UUIDs:
```
(Pdb) photo.album_info[0].__dict__
{'_uuid': '17816791-ABF3-447B-942C-9FA8065EEBBA', '_db': osxphotos.PhotosDB(dbfile='/Users/simon/Pictures/Photos Library.photoslibrary/database/photos.db'), '_title': 'Geotaggable Photos geotagged'}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199701,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623199701,MDEyOklzc3VlQ29tbWVudDYyMzE5OTcwMQ==,9599,simonw,2020-05-03T23:17:38Z,2020-05-03T23:17:38Z,MEMBER,"Record burst_uuid as a column:
```
(Pdb) with_bursts[0]._info[""burstUUID""]
'703FAA23-57BF-40B4-8A33-D9CEB143391B'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199750,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623199750,MDEyOklzc3VlQ29tbWVudDYyMzE5OTc1MA==,9599,simonw,2020-05-03T23:17:58Z,2020-05-03T23:17:58Z,MEMBER,Reading this source code is really useful for figuring out how to store a photo in a DB table: https://github.com/RhetTbull/osxphotos/blob/7444b6d173918a3ad2a07aefce5ecf054786c787/osxphotos/photoinfo.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623232984,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623232984,MDEyOklzc3VlQ29tbWVudDYyMzIzMjk4NA==,9599,simonw,2020-05-04T02:41:32Z,2020-05-04T02:41:32Z,MEMBER,Needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/12#issuecomment-618796564,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/12,618796564,MDEyOklzc3VlQ29tbWVudDYxODc5NjU2NA==,9599,simonw,2020-04-24T04:35:25Z,2020-04-24T04:35:25Z,MEMBER,Code: https://github.com/dogsheep/photos-to-sqlite/blob/a388cf1f1b6b67752d669466cda8b171b6582171/photos_to_sqlite/cli.py#L109-L114,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",606033104,"If less than 500MB, show size in MB not GB",
https://github.com/dogsheep/dogsheep-photos/issues/13#issuecomment-620273692,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13,620273692,MDEyOklzc3VlQ29tbWVudDYyMDI3MzY5Mg==,9599,simonw,2020-04-27T22:42:50Z,2020-04-27T22:42:50Z,MEMBER,"```
>>> def ext_counts(directory):
...     counts = {}
...     for path in pathlib.Path(directory).glob(""**/*""):
...         ext = path.suffix
...         counts[ext] = counts.get(ext, 0) + 1
...     return counts
... 
>>> 
>>> ext_counts(""/Users/simon/Pictures/Photos Library.photoslibrary/originals"")
{'': 16, '.heic': 15478, '.jpeg': 21691, '.mov': 946, '.png': 2262, '.gif': 38, '.mp4': 116, '.aae': 2}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607888367,Also upload movie files,
https://github.com/dogsheep/dogsheep-photos/issues/13#issuecomment-620309185,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13,620309185,MDEyOklzc3VlQ29tbWVudDYyMDMwOTE4NQ==,9599,simonw,2020-04-28T00:39:45Z,2020-04-28T00:39:45Z,MEMBER,I'm going to leave this until I have the mechanism for associating a live photo video with the photo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607888367,Also upload movie files,
https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620769348,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620769348,MDEyOklzc3VlQ29tbWVudDYyMDc2OTM0OA==,9599,simonw,2020-04-28T18:09:21Z,2020-04-28T18:09:21Z,MEMBER,"Pricing is pretty good: free for first 1,000 calls per month, then $1.50 per thousand after that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747,Annotate photos using the Google Cloud Vision API,
https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620771067,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620771067,MDEyOklzc3VlQ29tbWVudDYyMDc3MTA2Nw==,9599,simonw,2020-04-28T18:12:34Z,2020-04-28T18:15:38Z,MEMBER,"Python library docs: https://googleapis.dev/python/vision/latest/index.html

I'm creating a new project for this called simonwillison-photos: https://console.cloud.google.com/projectcreate

https://console.cloud.google.com/home/dashboard?project=simonwillison-photos

Then I enabled the Vision API. The direct link to https://console.cloud.google.com/flows/enableapi?apiid=vision-json.googleapis.com which they provided in the docs didn't work - it gave me a ""You don't have sufficient permissions to use the requested API"" error - but starting at the ""Enable APIs"" page and searching for it worked fine.

I created a new service account as an ""owner"" of that project: https://console.cloud.google.com/apis/credentials/serviceaccountkey (and complained about it on Twitter and through their feedback form)

`pip install google-cloud-vision`

```python
from google.cloud import vision
client = vision.ImageAnnotatorClient.from_service_account_file(""simonwillison-photos-18c570b301fe.json"")
# Photo of a lemur
response = client.annotate_image(
    {
        ""image"": {
            ""source"": {
                ""image_uri"": ""https://photos.simonwillison.net/i/1b3414ee9ade67ce04ade9042e6d4b433d1e523c9a16af17f490e2c0a619755b.jpeg""
            }
        },
        ""features"": [
            {""type"": vision.enums.Feature.Type.IMAGE_PROPERTIES},
            {""type"": vision.enums.Feature.Type.OBJECT_LOCALIZATION},
            {""type"": vision.enums.Feature.Type.LABEL_DETECTION},
        ],
    }
)
response
```
Output is:
```
label_annotations {
  mid: ""/m/09686""
  description: ""Vertebrate""
  score: 0.9851104021072388
  topicality: 0.9851104021072388
}
label_annotations {
  mid: ""/m/04rky""
  description: ""Mammal""
  score: 0.975814163684845
  topicality: 0.975814163684845
}
label_annotations {
  mid: ""/m/01280g""
  description: ""Wildlife""
  score: 0.8973650336265564
  topicality: 0.8973650336265564
}
label_annotations {
  mid: ""/m/02f9pk""
  description: ""Lemur""
  score: 0.8270352482795715
  topicality: 0.8270352482795715
}
label_annotations {
  mid: ""/m/0fbf1m""
  description: ""Terrestrial animal""
  score: 0.7443860769271851
  topicality: 0.7443860769271851
}
label_annotations {
  mid: ""/m/06z_nw""
  description: ""Tail""
  score: 0.6934166550636292
  topicality: 0.6934166550636292
}
label_annotations {
  mid: ""/m/0b5gs""
  description: ""Branch""
  score: 0.6203985214233398
  topicality: 0.6203985214233398
}
label_annotations {
  mid: ""/m/05s2s""
  description: ""Plant""
  score: 0.585474967956543
  topicality: 0.585474967956543
}
label_annotations {
  mid: ""/m/089v3""
  description: ""Zoo""
  score: 0.5488107800483704
  topicality: 0.5488107800483704
}
label_annotations {
  mid: ""/m/02tcwp""
  description: ""Trunk""
  score: 0.5200017690658569
  topicality: 0.5200017690658569
}
image_properties_annotation {
  dominant_colors {
    colors {
      color {
        red: 172.0
        green: 146.0
        blue: 116.0
      }
      score: 0.24523821473121643
      pixel_fraction: 0.027533333748579025
    }
    colors {
      color {
        red: 54.0
        green: 50.0
        blue: 42.0
      }
      score: 0.10449723154306412
      pixel_fraction: 0.12893334031105042
    }
    colors {
      color {
        red: 141.0
        green: 121.0
        blue: 97.0
      }
      score: 0.1391485631465912
      pixel_fraction: 0.039133332669734955
    }
    colors {
      color {
        red: 28.0
        green: 25.0
        blue: 20.0
      }
      score: 0.08589499443769455
      pixel_fraction: 0.11506666988134384
    }
    colors {
      color {
        red: 87.0
        green: 82.0
        blue: 74.0
      }
      score: 0.0845794677734375
      pixel_fraction: 0.16113333404064178
    }
    colors {
      color {
        red: 121.0
        green: 117.0
        blue: 108.0
      }
      score: 0.05901569500565529
      pixel_fraction: 0.13379999995231628
    }
    colors {
      color {
        red: 94.0
        green: 83.0
        blue: 66.0
      }
      score: 0.049011144787073135
      pixel_fraction: 0.03946666792035103
    }
    colors {
      color {
        red: 155.0
        green: 117.0
        blue: 90.0
      }
      score: 0.04164913296699524
      pixel_fraction: 0.0023333332501351833
    }
    colors {
      color {
        red: 178.0
        green: 143.0
        blue: 102.0
      }
      score: 0.02993861958384514
      pixel_fraction: 0.0012666666880249977
    }
    colors {
      color {
        red: 61.0
        green: 51.0
        blue: 35.0
      }
      score: 0.027391711249947548
      pixel_fraction: 0.01953333243727684
    }
  }
}
crop_hints_annotation {
  crop_hints {
    bounding_poly {
      vertices {
        x: 2073
      }
      vertices {
        x: 4008
      }
      vertices {
        x: 4008
        y: 3455
      }
      vertices {
        x: 2073
        y: 3455
      }
    }
    confidence: 0.65625
    importance_fraction: 0.746666669845581
  }
}
localized_object_annotations {
  mid: ""/m/0jbk""
  name: ""Animal""
  score: 0.7008256912231445
  bounding_poly {
    normalized_vertices {
      x: 0.0390297956764698
      y: 0.26235100626945496
    }
    normalized_vertices {
      x: 0.8466796875
      y: 0.26235100626945496
    }
    normalized_vertices {
      x: 0.8466796875
      y: 0.9386426210403442
    }
    normalized_vertices {
      x: 0.0390297956764698
      y: 0.9386426210403442
    }
  }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747,Annotate photos using the Google Cloud Vision API,
https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620771698,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620771698,MDEyOklzc3VlQ29tbWVudDYyMDc3MTY5OA==,9599,simonw,2020-04-28T18:13:48Z,2020-04-28T18:13:48Z,MEMBER,"For face detection:
```
    {""type"": vision.enums.Feature.Type.Type.FACE_DETECTION}
```
For OCR:
```
    {""type"": vision.enums.Feature.Type.DOCUMENT_TEXT_DETECTION}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747,Annotate photos using the Google Cloud Vision API,
https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620772190,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620772190,MDEyOklzc3VlQ29tbWVudDYyMDc3MjE5MA==,9599,simonw,2020-04-28T18:14:43Z,2020-04-28T18:14:43Z,MEMBER,"Database schema for this will require some thought. Just dumping the output into a JSON column isn't going to be flexible enough - I want to be able to FTS against labels and OCR text, and potentially query against other characteristics too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747,Annotate photos using the Google Cloud Vision API,
https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620774507,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620774507,MDEyOklzc3VlQ29tbWVudDYyMDc3NDUwNw==,9599,simonw,2020-04-28T18:19:06Z,2020-04-28T18:19:06Z,MEMBER,"The default timeout is a bit aggressive and sometimes failed for me if my resizing proxy took too long to fetch and resize the image.

`client.annotate_image(..., timeout=3.0)` may be worth trying.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747,Annotate photos using the Google Cloud Vision API,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723026,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,623723026,MDEyOklzc3VlQ29tbWVudDYyMzcyMzAyNg==,9599,simonw,2020-05-04T21:41:30Z,2020-05-04T21:41:30Z,MEMBER,"I'm going to put these in a table called `apple_photos_scores` - I'll also pull in the following columns from the `ZGENERICASSET` table:

* `ZOVERALLAESTHETICSCORE`
* `ZCURATIONSCORE`
* `ZHIGHLIGHTVISIBILITYSCORE`
* `ZPROMOTIONSCORE`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723687,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,623723687,MDEyOklzc3VlQ29tbWVudDYyMzcyMzY4Nw==,9599,simonw,2020-05-04T21:43:06Z,2020-05-04T21:43:06Z,MEMBER,It looks like I can map the photos I'm importing to these tables using the `ZUUID` column on `ZGENERICASSET` to get a `Z_PK` which then maps to the rows in `ZGENERICASSET`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623730934,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,623730934,MDEyOklzc3VlQ29tbWVudDYyMzczMDkzNA==,9599,simonw,2020-05-04T22:00:38Z,2020-05-04T22:00:48Z,MEMBER,"Here's the query to create the new table:
```sql
create table apple_photos_scores as select
    ZGENERICASSET.ZUUID,
    ZGENERICASSET.ZOVERALLAESTHETICSCORE,
    ZGENERICASSET.ZCURATIONSCORE,
    ZGENERICASSET.ZPROMOTIONSCORE,
    ZGENERICASSET.ZHIGHLIGHTVISIBILITYSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZBEHAVIORALSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZFAILURESCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZHARMONIOUSCOLORSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZIMMERSIVENESSSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZINTERACTIONSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZINTERESTINGSUBJECTSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZINTRUSIVEOBJECTPRESENCESCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZLIVELYCOLORSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZLOWLIGHT,
    ZCOMPUTEDASSETATTRIBUTES.ZNOISESCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCAMERATILTSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCOMPOSITIONSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTLIGHTINGSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPATTERNSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPERSPECTIVESCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPOSTPROCESSINGSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTREFLECTIONSSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTSYMMETRYSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZSHARPLYFOCUSEDSUBJECTSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZTASTEFULLYBLURREDSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZWELLCHOSENSUBJECTSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZWELLFRAMEDSUBJECTSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZWELLTIMEDSHOTSCORE
from
    attached.ZGENERICASSET
      join attached.ZCOMPUTEDASSETATTRIBUTES on
          attached.ZGENERICASSET.Z_PK = attached.ZCOMPUTEDASSETATTRIBUTES.Z_PK;
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623739934,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,623739934,MDEyOklzc3VlQ29tbWVudDYyMzczOTkzNA==,9599,simonw,2020-05-04T22:24:26Z,2020-05-04T22:24:26Z,MEMBER,Twitter thread with some examples of photos that are coming up from queries against these scores: https://twitter.com/simonw/status/1257434670750408705,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436115,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,748436115,MDEyOklzc3VlQ29tbWVudDc0ODQzNjExNQ==,8573886,nickvazz,2020-12-19T07:43:38Z,2020-12-19T07:47:36Z,NONE,"Hey Simon! I really enjoy datasette so far, just started trying it out today following your iPhone photos [example](https://simonwillison.net/2020/May/21/dogsheep-photos/). 

I am not sure if you had run into this or not, but it seems like they might have changed one of the column names from
`ZGENERICASSET` to `ZASSET`. Should I open a PR? 

Would change:
- [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L209-L213)
- [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L238)
- [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L240)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436779,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,748436779,MDEyOklzc3VlQ29tbWVudDc0ODQzNjc3OQ==,41546558,RhetTbull,2020-12-19T07:49:00Z,2020-12-19T07:49:00Z,CONTRIBUTOR,@nickvazz ZGENERICASSET changed to ZASSET in Big Sur. Here's a list of other changes to the schema in Big Sur: https://github.com/RhetTbull/osxphotos/wiki/Changes-in-Photos-6---Big-Sur,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748562288,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,748562288,MDEyOklzc3VlQ29tbWVudDc0ODU2MjI4OA==,41546558,RhetTbull,2020-12-20T04:44:22Z,2020-12-20T04:44:22Z,CONTRIBUTOR,"@nickvazz @simonw I opened a [PR](https://github.com/dogsheep/dogsheep-photos/pull/31) that replaces the SQL for `ZCOMPUTEDASSETATTRIBUTES` to use osxphotos which now exposes all this data and has been updated for Big Sur.  I did regression tests to confirm the extracted data is identical, with one exception which should not affect operation: the old code pulled data from `ZCOMPUTEDASSETATTRIBUTES` for missing photos while the main loop ignores missing photos and does not add them to `apple_photos`.  The new code does not add rows to the `apple_photos_scores` table for missing photos.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623805823,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623805823,MDEyOklzc3VlQ29tbWVudDYyMzgwNTgyMw==,9599,simonw,2020-05-05T02:45:56Z,2020-05-05T02:45:56Z,MEMBER,I filed an issue with `osxphotos` about this here: https://github.com/RhetTbull/osxphotos/issues/121,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806085,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623806085,MDEyOklzc3VlQ29tbWVudDYyMzgwNjA4NQ==,9599,simonw,2020-05-05T02:47:18Z,2020-05-05T02:47:18Z,MEMBER,"In https://github.com/RhetTbull/osxphotos/issues/121#issuecomment-623249263 Rhet Turnbull spotted a table called `ZSCENEIDENTIFIER` which looked like it might have the right data, but the columns in it aren't particularly helpful:
```
Z_PK,Z_ENT,Z_OPT,ZSCENEIDENTIFIER,ZASSETATTRIBUTES,ZCONFIDENCE
8,49,1,731,5,0.11834716796875
9,49,1,684,6,0.0233648251742125
10,49,1,1702,1,0.026153564453125
```
I love the look of those confidence scores, but what do the numbers mean?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806533,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623806533,MDEyOklzc3VlQ29tbWVudDYyMzgwNjUzMw==,9599,simonw,2020-05-05T02:50:16Z,2020-05-05T02:50:16Z,MEMBER,"I figured there must be a separate database that Photos uses to store the text of the identified labels.

I used ""Open Files and Ports"" in Activity Monitor against the Photos app to try and spot candidates... and found `/Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite` - a 53MB SQLite database file.

<img width=""1365"" alt=""Item-0_and_Item-0_and_Item-0_and_Item-0"" src=""https://user-images.githubusercontent.com/9599/81031213-61ea0800-8e40-11ea-8237-cfce4a5128e0.png"">

Here's the schema of that file:
```
$ sqlite3 psi.sqlite .schema
CREATE TABLE word_embedding(word TEXT, extended_word TEXT, score DOUBLE);
CREATE INDEX word_embedding_index ON word_embedding(word);
CREATE VIRTUAL TABLE word_embedding_prefix USING fts5(extended_word)
/* word_embedding_prefix(extended_word) */;
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_content'(id INTEGER PRIMARY KEY, c0);
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID;
CREATE TABLE groups(category INT2, owning_groupid INT, content_string TEXT, normalized_string TEXT, lookup_identifier TEXT, token_ranges_0 INT8, token_ranges_1 INT8, UNIQUE(category, owning_groupid, content_string, lookup_identifier, token_ranges_0, token_ranges_1));
CREATE TABLE assets(uuid_0 INT, uuid_1 INT, creationDate INT, UNIQUE(uuid_0, uuid_1));
CREATE TABLE ga(groupid INT, assetid INT, PRIMARY KEY(groupid, assetid));
CREATE TABLE collections(uuid_0 INT, uuid_1 INT, startDate INT, endDate INT, title TEXT, subtitle TEXT, keyAssetUUID_0 INT, keyAssetUUID_1 INT, typeAndNumberOfAssets INT32, sortDate DOUBLE, UNIQUE(uuid_0, uuid_1));
CREATE TABLE gc(groupid INT, collectionid INT, PRIMARY KEY(groupid, collectionid));
CREATE VIRTUAL TABLE prefix USING fts5(content='groups', normalized_string, category UNINDEXED, tokenize = 'PSITokenizer');
CREATE TABLE IF NOT EXISTS 'prefix_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE IF NOT EXISTS 'prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE IF NOT EXISTS 'prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID;
CREATE TABLE lookup(identifier TEXT PRIMARY KEY, category INT2);
CREATE TRIGGER trigger_groups_insert AFTER INSERT ON groups BEGIN INSERT INTO prefix(rowid, normalized_string, category) VALUES (new.rowid, new.normalized_string, new.category); END;
CREATE TRIGGER trigger_groups_delete AFTER DELETE ON groups BEGIN INSERT INTO prefix(prefix, rowid, normalized_string, category) VALUES('delete', old.rowid, old.normalized_string, old.category); END;
CREATE INDEX group_pk ON groups(category, content_string, normalized_string, lookup_identifier);
CREATE INDEX asset_pk ON assets(uuid_0, uuid_1);
CREATE INDEX ga_assetid ON ga(assetid, groupid);
CREATE INDEX collection_pk ON collections(uuid_0, uuid_1);
CREATE INDEX gc_collectionid ON gc(collectionid);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806687,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623806687,MDEyOklzc3VlQ29tbWVudDYyMzgwNjY4Nw==,9599,simonw,2020-05-05T02:51:16Z,2020-05-05T02:51:16Z,MEMBER,"Running datasette against it directly doesn't work:
```
simon@Simons-MacBook-Pro search % datasette psi.sqlite
Serve! files=('psi.sqlite',) (immutables=()) on port 8001
Usage: datasette serve [OPTIONS] [FILES]...

Error: Connection to psi.sqlite failed check: no such tokenizer: PSITokenizer
```
Instead, I created a new SQLite database with a copy of some of the key tables, like this:
```
sqlite-utils rows psi.sqlite groups | sqlite-utils insert /tmp/search.db groups -
sqlite-utils rows psi.sqlite assets | sqlite-utils insert /tmp/search.db assets -
sqlite-utils rows psi.sqlite ga | sqlite-utils insert /tmp/search.db ga -
sqlite-utils rows psi.sqlite collections | sqlite-utils insert /tmp/search.db collections -
sqlite-utils rows psi.sqlite gc | sqlite-utils insert /tmp/search.db gc -
sqlite-utils rows psi.sqlite lookup | sqlite-utils insert /tmp/search.db lookup -
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623807568,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623807568,MDEyOklzc3VlQ29tbWVudDYyMzgwNzU2OA==,9599,simonw,2020-05-05T02:56:06Z,2020-05-05T02:56:06Z,MEMBER,"I'm pretty sure this is what I'm after. The `groups` table has what looks like identified labels in the rows with category = 2025:

<img width=""1122"" alt=""words__groups__2_528_rows_where_where_category___2025"" src=""https://user-images.githubusercontent.com/9599/81031361-e0df4080-8e40-11ea-9060-6d850aa52140.png"">

Then there's a `ga` table that maps groups to assets:

<img width=""304"" alt=""words__ga__633_653_rows"" src=""https://user-images.githubusercontent.com/9599/81031387-f48aa700-8e40-11ea-9a3d-da23903be928.png"">

And an `assets` table which looks like it has one row for every one of my photos:

<img width=""645"" alt=""words__assets__40_419_rows"" src=""https://user-images.githubusercontent.com/9599/81031402-04a28680-8e41-11ea-8047-e9199d068563.png"">

One major challenge: these UUIDs are split into two integer numbers, `uuid_0` and `uuid_1` - but the main photos database uses regular UUIDs like this:

![image](https://user-images.githubusercontent.com/9599/81031481-39164280-8e41-11ea-983b-005ced641a18.png)

I need to figure out how to match up these two different UUID representations. I asked on Twitter if anyone has any ideas: https://twitter.com/simonw/status/1257500689019703296","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623811131,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623811131,MDEyOklzc3VlQ29tbWVudDYyMzgxMTEzMQ==,9599,simonw,2020-05-05T03:16:18Z,2020-05-05T03:16:18Z,MEMBER,"Here's how to convert two integers unto a UUID using Java. Not sure if it's the solution I need though (or how to do the same thing in Python):

https://repl.it/repls/EuphoricSomberClasslibrary

<img width=""1494"" alt=""Repl_it_-_EuphoricSomberClasslibrary"" src=""https://user-images.githubusercontent.com/9599/81032267-0d488c00-8e44-11ea-9be7-680eaccd1611.png"">

```java
import java.util.UUID;

class Main {
  public static void main(String[] args) {
    java.util.UUID uuid = new java.util.UUID(
      2544182952487526660L,
      -3640314103732024685L
    );
    System.out.println(
      uuid
    );
  }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623845014,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623845014,MDEyOklzc3VlQ29tbWVudDYyMzg0NTAxNA==,41546558,RhetTbull,2020-05-05T03:55:14Z,2020-05-05T03:56:24Z,CONTRIBUTOR,"I'm traveling w/o access to my Mac so can't help with any code right now.  I suspected ZSCENEIDENTIFIER was a foreign key into one of these psi.sqlite tables.  But looks like you're on to something connecting groups to assets.  As for the UUID, I think there's two ints because each is 64-bits but UUIDs are 128-bits.  Thus they need to be combined to get the 128 bit UUID.  You might be able to use Apple's [NSUUID](https://developer.apple.com/documentation/foundation/nsuuid?language=objc), for example, by wrapping with pyObjC.  Here's one [example](https://github.com/ronaldoussoren/pyobjc/blob/881c82a7ba90f193934b52b44143360c80dce5e5/pyobjc-framework-Cocoa/PyObjCTest/test_nsuuid.py) of using this in PyObjC's test suite. Interesting it's stored this way instead of a UUIDString as in Photos.sqlite.  Perhaps it for faster indexing.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623846880,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623846880,MDEyOklzc3VlQ29tbWVudDYyMzg0Njg4MA==,9599,simonw,2020-05-05T04:06:08Z,2020-05-05T04:06:08Z,MEMBER,"This function seems to convert them into UUIDs that match my photos:
```python
def to_uuid(uuid_0, uuid_1):
    b = uuid_0.to_bytes(8, 'little', signed=True) + uuid_1.to_bytes(8, 'little', signed=True)
    return str(uuid.UUID(bytes=b)).upper()
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855841,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623855841,MDEyOklzc3VlQ29tbWVudDYyMzg1NTg0MQ==,9599,simonw,2020-05-05T04:54:28Z,2020-05-05T04:54:28Z,MEMBER,"Things were not matching up for me correctly:

<img width=""1143"" alt=""search__select_json_object__img_src____https___photos_simonwillison_net_i______photos_sha256___________photos_ext______w_400___as_photo__groups_content_string__assets_uuid_0__assets_uuid_1__to_uuid_assets_uuid_0__assets_uuid_1__as_uuid__pho"" src=""https://user-images.githubusercontent.com/9599/81035923-ca8db080-8e51-11ea-95a7-6ee60bae7502.png"">

I think that's because my import script didn't correctly import the existing `rowid` values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855885,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623855885,MDEyOklzc3VlQ29tbWVudDYyMzg1NTg4NQ==,9599,simonw,2020-05-05T04:54:39Z,2020-05-05T04:54:53Z,MEMBER,"Trying this import mechanism instead:
`sqlite3 /Users/simon/Pictures/Photos\ Library.photoslibrary/database/search/psi.sqlite .dump | grep -v 'CREATE INDEX' | grep -v 'CREATE TRIGGER' | grep -v 'CREATE VIRTUAL TABLE' | sqlite3 search.db`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623857417,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623857417,MDEyOklzc3VlQ29tbWVudDYyMzg1NzQxNw==,9599,simonw,2020-05-05T05:01:47Z,2020-05-05T05:01:47Z,MEMBER,"Even that didn't work - it didn't copy across the rowid values. I'm pretty sure that's what's wrong here:
```
sqlite3 /Users/simon/Pictures/Photos\ Library.photoslibrary/database/search/psi.sqlite 'select rowid, uuid_0, uuid_1 from assets limit 10'                                              
1619605|-9205353363298198838|4814875488794983828
1641378|-9205348195631362269|390804289838822030
1634974|-9205331524553603243|-3834026796261633148
1619083|-9205326176986145401|7563404215614709654
22131|-9205315724827218763|8370531509591906734
1645633|-9205247376092758131|-1311540150497601346
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623863902,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623863902,MDEyOklzc3VlQ29tbWVudDYyMzg2MzkwMg==,9599,simonw,2020-05-05T05:31:53Z,2020-05-05T05:31:53Z,MEMBER,"Yes! Turning those `rowid` values into `id` with this script did the job:
```python
import sqlite3
import sqlite_utils

conn = sqlite3.connect(
    ""/Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite""
)


def all_rows(table):
    result = conn.execute(""select rowid as id, * from {}"".format(table))
    cols = [c[0] for c in result.description]
    for row in result.fetchall():
        yield dict(zip(cols, row))


if __name__ == ""__main__"":
    db = sqlite_utils.Database(""psi_copy.db"")
    for table in (""assets"", ""collections"", ""ga"", ""gc"", ""groups""):
        db[table].upsert_all(all_rows(table), pk=""id"", alter=True)
```
Then I ran this query:
```sql
select 
  json_object('img_src', 'https://photos.simonwillison.net/i/' || photos.sha256 || '.' || photos.ext || '?w=400') as photo,
   group_concat(strip_null_chars(groups.content_string), ' ') as words, assets.uuid_0, assets.uuid_1, to_uuid(assets.uuid_0, assets.uuid_1) as uuid
from assets join ga on assets.id = ga.assetid
join groups on ga.groupid = groups.id
join photos on photos.uuid = to_uuid(assets.uuid_0, assets.uuid_1)
where groups.category = 2024
group by assets.id
order by random() limit 10
```
And got these results!
<img width=""1054"" alt=""psi_copy__select_json_object__img_src____https___photos_simonwillison_net_i______photos_sha256___________photos_ext______w_400___as_photo__group_concat_strip_null_chars_groups_content_string________as_words__assets_uuid_0__assets_uuid_1__to"" src=""https://user-images.githubusercontent.com/9599/81037264-f1021a80-8e56-11ea-9924-6f9f55a0fb4b.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623865250,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623865250,MDEyOklzc3VlQ29tbWVudDYyMzg2NTI1MA==,9599,simonw,2020-05-05T05:38:16Z,2020-05-05T05:38:16Z,MEMBER,It looks like `groups.content_string` often has a null byte in it. I should clean this up as part of the import.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624278090,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17,624278090,MDEyOklzc3VlQ29tbWVudDYyNDI3ODA5MA==,9599,simonw,2020-05-05T20:06:01Z,2020-05-05T20:06:01Z,MEMBER,https://www.python.org/dev/peps/pep-0508/#environment-markers I think I want `sys_platform` of `darwin`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612860531,Only install osxphotos if running on macOS,
https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624278714,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17,624278714,MDEyOklzc3VlQ29tbWVudDYyNDI3ODcxNA==,9599,simonw,2020-05-05T20:07:19Z,2020-05-05T20:07:19Z,MEMBER,"From https://hynek.me/articles/conditional-python-dependencies/ I think this will look like:
```python
setup(
    # ...
    install_requires=[
        # ...
        ""osxphotos>=0.28.13 ; sys_platform=='darwin'"",
    ]
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612860531,Only install osxphotos if running on macOS,
https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624284539,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17,624284539,MDEyOklzc3VlQ29tbWVudDYyNDI4NDUzOQ==,41546558,RhetTbull,2020-05-05T20:20:05Z,2020-05-05T20:20:05Z,CONTRIBUTOR,"FYI, I've got an [issue](https://github.com/RhetTbull/osxphotos/issues/25) to make osxphotos cross-platform but it's low on my priority list.  About 90% of the functionality could be done cross-platform but right now the MacOS specific stuff is embedded throughout and would take some work.  Though I try to minimize it, there's sprinklings of ObjC & Applescript throughout osxphotos.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612860531,Only install osxphotos if running on macOS,
https://github.com/dogsheep/dogsheep-photos/issues/18#issuecomment-624364557,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18,624364557,MDEyOklzc3VlQ29tbWVudDYyNDM2NDU1Nw==,9599,simonw,2020-05-05T23:49:18Z,2020-05-05T23:49:18Z,MEMBER,Label is `macos-latest`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612860758,Switch CI solution to GitHub Actions with a macOS runner,
https://github.com/dogsheep/dogsheep-photos/issues/19#issuecomment-624406285,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/19,624406285,MDEyOklzc3VlQ29tbWVudDYyNDQwNjI4NQ==,9599,simonw,2020-05-06T02:10:03Z,2020-05-06T02:10:03Z,MEMBER,"Most annoying part of this is the difficulty of actually showing a photo.

Maybe I need to run a local proxy that I can link to? A custom Datasette plugin perhaps?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613002220,apple-photos command should work even if upload has not run,
https://github.com/dogsheep/dogsheep-photos/issues/2#issuecomment-615931488,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/2,615931488,MDEyOklzc3VlQ29tbWVudDYxNTkzMTQ4OA==,9599,simonw,2020-04-18T19:24:02Z,2020-04-18T19:24:02Z,MEMBER,I made a start on this last week with a https://github.com/simonw/heic-to-jpeg proxy.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533352,Ability to convert HEIC images to JPEG,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408220,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,624408220,MDEyOklzc3VlQ29tbWVudDYyNDQwODIyMA==,9599,simonw,2020-05-06T02:18:47Z,2020-05-06T02:18:47Z,MEMBER,"The `apple_photos` table has an indexed `uuid` column and a `path` column which stores the full path to that photo file on disk.

I can write a custom Datasette plugin which takes the `uuid` from the URL, looks up the path, then serves up a thumbnail of the jpeg or heic image file.

I'll prototype this is a one-off plugin first, then package it on PyPI for other people to install.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408370,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,624408370,MDEyOklzc3VlQ29tbWVudDYyNDQwODM3MA==,9599,simonw,2020-05-06T02:19:27Z,2020-05-06T02:19:27Z,MEMBER,"The plugin can be generalized: it can be configured to know how to take the URL path, look it up in ANY table (via a custom SQL query) to get a path on disk and then serve that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408738,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,624408738,MDEyOklzc3VlQ29tbWVudDYyNDQwODczOA==,9599,simonw,2020-05-06T02:21:05Z,2020-05-06T02:21:32Z,MEMBER,"Here's rendering code from my hacked-together not-yet-released S3 image proxy:
```python
from starlette.responses import Response
from PIL import Image, ExifTags
import pyheif

for ORIENTATION_TAG in ExifTags.TAGS.keys():
    if ExifTags.TAGS[ORIENTATION_TAG] == ""Orientation"":
        break
    ...
    # Load it into Pillow
    if ext == ""heic"":
        heic = pyheif.read_heif(image_response.content)
        image = Image.frombytes(mode=heic.mode, size=heic.size, data=heic.data)
    else:
        image = Image.open(io.BytesIO(image_response.content))

    # Does EXIF tell us to rotate it?
    try:
        exif = dict(image._getexif().items())
        if exif[ORIENTATION_TAG] == 3:
            image = image.rotate(180, expand=True)
        elif exif[ORIENTATION_TAG] == 6:
            image = image.rotate(270, expand=True)
        elif exif[ORIENTATION_TAG] == 8:
            image = image.rotate(90, expand=True)
    except (AttributeError, KeyError, IndexError):
        pass

    # Resize based on ?w= and ?h=, if set
    width, height = image.size
    w = request.query_params.get(""w"")
    h = request.query_params.get(""h"")
    if w is not None or h is not None:
        if h is None:
            # Set h based on w
            w = int(w)
            h = int((float(height) / width) * w)
        elif w is None:
            h = int(h)
            # Set w based on h
            w = int((float(width) / height) * h)
        w = int(w)
        h = int(h)
        image.thumbnail((w, h))

    # ?bw= converts to black and white
    if request.query_params.get(""bw""):
        image = image.convert(""L"")

    # ?q= sets the quality - defaults to 75
    quality = 75
    q = request.query_params.get(""q"")
    if q and q.isdigit() and 1 <= int(q) <= 100:
        quality = int(q)

    # Output as JPEG or PNG
    output_image = io.BytesIO()
    image_type = ""JPEG""
    kwargs = {""quality"": quality}
    if image.format == ""PNG"":
        image_type = ""PNG""
        kwargs = {}

    image.save(output_image, image_type, **kwargs)
    return Response(
        output_image.getvalue(),
        media_type=""image/jpeg"",
        headers={""cache-control"": ""s-maxage={}, public"".format(365 * 24 * 60 * 60)},
    )
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-625947133,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,625947133,MDEyOklzc3VlQ29tbWVudDYyNTk0NzEzMw==,9599,simonw,2020-05-08T18:13:06Z,2020-05-08T18:13:06Z,MEMBER,`datasette-media` will be able to handle this once I implement https://github.com/simonw/datasette-media/issues/3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633234781,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633234781,MDEyOklzc3VlQ29tbWVudDYzMzIzNDc4MQ==,41439,dmd,2020-05-24T13:56:13Z,2020-05-24T13:56:13Z,NONE,"As that seems to be closed, can you give a hint on how to make this work?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633626741,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633626741,MDEyOklzc3VlQ29tbWVudDYzMzYyNjc0MQ==,9599,simonw,2020-05-25T15:38:55Z,2020-05-25T15:38:55Z,MEMBER,"Sure, I should absolutely document this!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633629944,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633629944,MDEyOklzc3VlQ29tbWVudDYzMzYyOTk0NA==,9599,simonw,2020-05-25T15:47:42Z,2020-05-25T15:47:42Z,MEMBER,"I'll add a proper section to the README, but for the moment here's how I do this.

First, install `datasette` and the `datasette-media` plugin.

Create a `metadata.yaml` file with the following content:

```yaml
plugins:
  datasette-media:
    photo:
      sql: |-
        select path as filepath, 200 as resize_height from apple_photos where uuid = :key
    photo-big:
      sql: |-
        select path as filepath, 1024 as resize_height from apple_photos where uuid = :key
```
Now run `datasette -m metadata.yaml photos.db` - thumbnails will be served at http://127.0.0.1:8001/-/media/photo/F4469918-13F3-43D8-9EC1-734C0E6B60AD and larger sizes of the image at http://127.0.0.1:8001/-/media/photo-big/A8B02C7D-365E-448B-9510-69F80C26304D

I also made myself two custom pages, one showing recent images and one showing random images.

To do this, install the `datasette-template-sql` plugin and then create a `templates/pages` directory and add these files:

`recent-photos.html`
```html
<h1>Recent photos</h1>

<div>
{% for photo in sql(""select * from apple_photos order by date desc limit 100"") %}
    <img src=""/-/media/photo/{{ photo['uuid'] }}"">
{% endfor %}
</div>
```

`random-photos.html`
```html
<h1>Random photos</h1>

<div>
{% for photo in sql(""with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 100"") %}
    <img src=""/-/media/photo/{{ photo['uuid'] }}"">
{% endfor %}
</div>
```

Now run `datasette -m metadata.yaml photos.db --template-dir=templates/`

Visit http://127.0.0.1:8001/random-photos to see some random photos or http://127.0.0.1:8002/recent-photos for recent photos.

This is using this mechanism: https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633643921,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633643921,MDEyOklzc3VlQ29tbWVudDYzMzY0MzkyMQ==,9599,simonw,2020-05-25T16:29:44Z,2020-05-25T16:29:44Z,MEMBER,https://github.com/dogsheep/dogsheep-photos/blob/dc43fa8653cb9c7238a36f52239b91d1ec916d5c/README.md#serving-photos-locally-with-datasette-media,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633644225,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633644225,MDEyOklzc3VlQ29tbWVudDYzMzY0NDIyNQ==,9599,simonw,2020-05-25T16:30:44Z,2020-05-25T16:30:44Z,MEMBER,I'll add docs on using `datasette-json-html` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633704127,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633704127,MDEyOklzc3VlQ29tbWVudDYzMzcwNDEyNw==,9599,simonw,2020-05-25T20:14:22Z,2020-05-25T20:14:22Z,MEMBER,https://github.com/dogsheep/dogsheep-photos/blob/0.4.1/README.md#serving-photos-locally-with-datasette-media,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388764,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626388764,MDEyOklzc3VlQ29tbWVudDYyNjM4ODc2NA==,9599,simonw,2020-05-10T20:58:52Z,2020-05-10T20:58:52Z,MEMBER,"More from the debugger:
```
> /Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py(614)place()
-> self._place = PlaceInfo5(self._info[""reverse_geolocation""])
```
And:
```
> /Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/utils.py(91)osxphoto_to_row()
-> place = photo.place
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388837,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626388837,MDEyOklzc3VlQ29tbWVudDYyNjM4ODgzNw==,9599,simonw,2020-05-10T20:59:32Z,2020-05-10T20:59:32Z,MEMBER,So it appears it's possible for `photo.place` to raise that exception. A workaround could be to catch that and treat those photos as not having a place.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626390317,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626390317,MDEyOklzc3VlQ29tbWVudDYyNjM5MDMxNw==,41546558,RhetTbull,2020-05-10T21:11:24Z,2020-05-10T21:50:58Z,CONTRIBUTOR,"Ugh....Yeah, I think easiest is to catch the exception and return no place as you suggest.  This particular bit of code involves un-archiving a serialized NSKeyedArchiver which uses an object table and it is certainly possible to create a circular reference that way.  Because this is happening in the decode, the circular reference must be in the original data.  Does Photos show valid reverse geolocation info for the photo in question? If so, Photos may be doing something beyond a simple decode of the binary plist.  For now, I'll push a patch to catch the exception.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626394989,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626394989,MDEyOklzc3VlQ29tbWVudDYyNjM5NDk4OQ==,9599,simonw,2020-05-10T21:50:36Z,2020-05-10T21:50:36Z,MEMBER,https://github.com/Marketcircle/bpylist/pull/2 looks relevant here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395103,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395103,MDEyOklzc3VlQ29tbWVudDYyNjM5NTEwMw==,9599,simonw,2020-05-10T21:51:36Z,2020-05-10T21:51:36Z,MEMBER,"@RhetTbull I tried that workaround and it turns out I'm getting this error on ALL of my photos now!

It's weird: a few day ago this wasn't happening. Now it's happening to everything. I'm not sure what I might have changed. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395209,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395209,MDEyOklzc3VlQ29tbWVudDYyNjM5NTIwOQ==,9599,simonw,2020-05-10T21:52:42Z,2020-05-10T21:52:42Z,MEMBER,"Aha! It looks like I accidentally installed the old bplist into the same environment:
```
$ pip freeze | grep bpylist
bpylist==0.1.4
bpylist2==3.0.0
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395507,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395507,MDEyOklzc3VlQ29tbWVudDYyNjM5NTUwNw==,41546558,RhetTbull,2020-05-10T21:54:45Z,2020-05-10T21:54:45Z,CONTRIBUTOR,"@simonw does Photos show valid reverse geolocation info?  Are you sure you're using [bpylist2](https://github.com/xa4a/bpylist2) and not bpylist?  They're both unfortunately imported as ""bpylist"" so if you somehow got the wrong (original bpylist) version installed, it could be the issue. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395641,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395641,MDEyOklzc3VlQ29tbWVudDYyNjM5NTY0MQ==,41546558,RhetTbull,2020-05-10T21:55:54Z,2020-05-10T21:55:54Z,CONTRIBUTOR,Did removing old bpylist solve the original problem or do you still have a photo that throws circular reference?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395781,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395781,MDEyOklzc3VlQ29tbWVudDYyNjM5NTc4MQ==,9599,simonw,2020-05-10T21:57:09Z,2020-05-10T21:57:09Z,MEMBER,"Yes, I just recreated my virtual environment from scratch and the error went away.

The problem occurred when I ran `pip install datasette-bplist` in the same virtual environment - https://github.com/simonw/datasette-bplist/blob/master/setup.py depends on `bpylist` which is incompatible with `bpylist2`.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626396379,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626396379,MDEyOklzc3VlQ29tbWVudDYyNjM5NjM3OQ==,41546558,RhetTbull,2020-05-10T22:01:48Z,2020-05-10T22:01:48Z,CONTRIBUTOR,"Frustrates me when package authors create a ""drop in"" replacement with the same import name...this kind of thing has bitten me more than once! Would've been nicer I think for bpylist2 to do ""import bpylist2 as bpylist""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-748436195,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,748436195,MDEyOklzc3VlQ29tbWVudDc0ODQzNjE5NQ==,8573886,nickvazz,2020-12-19T07:44:32Z,2020-12-19T07:44:49Z,NONE,"I have also run into this a bit, would it be possible to post your `requirements.txt` so I can try and reproduce your [blog post](https://simonwillison.net/2020/May/21/dogsheep-photos/)?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-626667235,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,626667235,MDEyOklzc3VlQ29tbWVudDYyNjY2NzIzNQ==,41546558,RhetTbull,2020-05-11T12:20:34Z,2020-05-11T12:20:34Z,CONTRIBUTOR,"@simonw FYI, osxphotos includes a built in ExifTool class that uses [exiftool](https://exiftool.org/) to read and write exif data.  It's not exposed yet in the docs because I really only use it right now in the osphotos command line interface to write tags when exporting.  In v0.28.16 (just pushed) I added an ExifTool.as_dict() method which will give you a dict with all the exif tags in a file.  For example:

```python
import osxphotos
photos = osxphotos.PhotosDB().photos()
exiftool = osxphotos.exiftool.ExifTool(photos[0].path)
exifdata = exiftool.as_dict()
tags = exifdata[""IPTC:Keywords""]
```

Not as elegant perhaps as a python only implementation because ExifTool has to make subprocess calls to an external tool but exiftool is by far the best tool available for reading and writing EXIF data and it does support HEIC.

As for implementation, ExifTool uses a singleton pattern so the first time you instantiate it, it spawns an IPC to exiftool but then keeps it open and uses the same process for any subsequent calls (even on different files). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118,Try out ExifReader,
https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-626941278,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,626941278,MDEyOklzc3VlQ29tbWVudDYyNjk0MTI3OA==,9599,simonw,2020-05-11T20:25:58Z,2020-05-11T20:25:58Z,MEMBER,"Interesting - do you know if there's anything the `exiftool` process handles that `ExifReader` doesn't?

I'm actually just going to extract a subset of the EXIF data at first - since the original photo files will always be available I don't feel the need to get everything out for the first step.

My plan is to use EXIF to help support photo collections that aren't in Apple Photos - I'm going to build a database table keyed by the `sha256` of each photo that extracts the camera make, lens, a few settings (ISO, aperture etc) and the GPS lat/lon.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118,Try out ExifReader,
https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-627007458,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,627007458,MDEyOklzc3VlQ29tbWVudDYyNzAwNzQ1OA==,41546558,RhetTbull,2020-05-11T22:51:52Z,2020-05-11T22:52:26Z,CONTRIBUTOR,"I'm not familiar with `ExifReader`.  I wrote my own wrapper around `exiftool` because I wanted a simple way to write EXIF data when exporting photos (e.g. writing out to PersonInImage and keywords to IPTC:Keywords) and the existing python packages like [pyexiftool](https://github.com/smarnach/pyexiftool) didn't do quite what I wanted.  If all you're after is the camera and shot info, that's available in `ZEXTENDEDATTRIBUTES` table.  I've got an open issue [#11](https://github.com/RhetTbull/osxphotos/issues/11) to add this to osxphotos but it hasn't bubbled to the top of my backlog yet.  

osxphotos will give you the location info: `PhotoInfo.location` returns a tuple of (lat, lon) though this info is in ZEXTENDEDATTRIBUTES too (though it might not be correct as I believe Photos creates this table at import and the user might have changed the location of a photo, e.g. if camera didn't have GPS).

```sql
CREATE TABLE ZEXTENDEDATTRIBUTES (
  Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER, 
  Z_OPT INTEGER, ZFLASHFIRED INTEGER, 
  ZISO INTEGER, ZMETERINGMODE INTEGER, 
  ZSAMPLERATE INTEGER, ZTRACKFORMAT INTEGER, 
  ZWHITEBALANCE INTEGER, ZASSET INTEGER, 
  ZAPERTURE FLOAT, ZBITRATE FLOAT, ZDURATION FLOAT, 
  ZEXPOSUREBIAS FLOAT, ZFOCALLENGTH FLOAT, 
  ZFPS FLOAT, ZLATITUDE FLOAT, ZLONGITUDE FLOAT, 
  ZSHUTTERSPEED FLOAT, ZCAMERAMAKE VARCHAR, 
  ZCAMERAMODEL VARCHAR, ZCODEC VARCHAR, 
  ZLENSMODEL VARCHAR
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118,Try out ExifReader,
https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-628405453,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,628405453,MDEyOklzc3VlQ29tbWVudDYyODQwNTQ1Mw==,41546558,RhetTbull,2020-05-14T05:59:53Z,2020-05-14T05:59:53Z,CONTRIBUTOR,"I've added support for the above exif data to [v0.28.17](https://github.com/RhetTbull/osxphotos/releases/tag/v0.28.17) of osxphotos.  `PhotoInfo.exif_info` will return an `ExifInfo` [dataclass](https://docs.python.org/3/library/dataclasses.html) object with the following properties:

```python
    flash_fired: bool
    iso: int
    metering_mode: int
    sample_rate: int
    track_format: int
    white_balance: int
    aperture: float
    bit_rate: float
    duration: float
    exposure_bias: float
    focal_length: float
    fps: float
    latitude: float
    longitude: float
    shutter_speed: float
    camera_make: str
    camera_model: str
    codec: str
    lens_model: str
```

It's not all the EXIF data available in most files but is the data Photos deems important to save.  Of course, you can get all the exif_data

Note: this only works in Photos 5.  As best as I can tell, EXIF data is not stored in the database for earlier versions. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118,Try out ExifReader,
https://github.com/dogsheep/dogsheep-photos/issues/23#issuecomment-631120771,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/23,631120771,MDEyOklzc3VlQ29tbWVudDYzMTEyMDc3MQ==,9599,simonw,2020-05-19T22:32:48Z,2020-05-19T22:32:48Z,MEMBER,Documentation: https://github.com/dogsheep/photos-to-sqlite/blob/e2fab012551eed05278040b5d57e7373a1b9a0bf/README.md#creating-a-subset-database,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621280529,create-subset command for creating a publishable subset of a photos database,
https://github.com/dogsheep/dogsheep-photos/issues/24#issuecomment-631255206,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/24,631255206,MDEyOklzc3VlQ29tbWVudDYzMTI1NTIwNg==,9599,simonw,2020-05-20T06:00:25Z,2020-05-20T06:00:25Z,MEMBER,This needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621323348,Configurable URL for images,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631127454,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631127454,MDEyOklzc3VlQ29tbWVudDYzMTEyNzQ1NA==,9599,simonw,2020-05-19T22:48:00Z,2020-05-21T15:58:32Z,MEMBER,"I built #23 to help with this.

    $ dogsheep-photos create-subset photos.db public.db \
        ""select sha256 from apple_photos where albums like '%Public%'""

And publish with Vercel:

    $ datasette publish now public.db --project dogsheep-photos \
        --about=dogsheep/dogsheep-photos \
        --about_url=""https://github.com/dogsheep/dogsheep-photos"" \
        --install=datasette-json-html \
        --install=datasette-cluster-map","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631251707,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631251707,MDEyOklzc3VlQ29tbWVudDYzMTI1MTcwNw==,9599,simonw,2020-05-20T05:49:27Z,2020-05-21T15:58:42Z,MEMBER,Renaming this demo to `dogsheep-photos.dogsheep.net`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253136,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631253136,MDEyOklzc3VlQ29tbWVudDYzMTI1MzEzNg==,9599,simonw,2020-05-20T05:53:58Z,2020-05-20T05:53:58Z,MEMBER,"Updated deploy command:
```
datasette publish now public.db --project dogsheep-photos \
    --about=dogsheep/dogsheep-photos \
    --about_url=""https://github.com/dogsheep/dogsheep-photos"" \
    --install=datasette-json-html \
    --install=datasette-cluster-map \
    --title ""Dogsheep Photos demo""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253248,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631253248,MDEyOklzc3VlQ29tbWVudDYzMTI1MzI0OA==,9599,simonw,2020-05-20T05:54:18Z,2020-05-20T05:54:18Z,MEMBER,https://dogsheep-photos.dogsheep.net/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253852,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631253852,MDEyOklzc3VlQ29tbWVudDYzMTI1Mzg1Mg==,9599,simonw,2020-05-20T05:56:17Z,2020-05-21T22:26:16Z,MEMBER,"I have a `deploy-demo.sh` script now:
```bash
#!/bin/bash
if [ -f public.db ]; then
  rm public.db
fi
pipenv run dogsheep-photos create-subset photos.db public.db \
    ""select sha256 from apple_photos where albums like '%Public%'""
pipenv run sqlite-utils create-view public.db photos_on_a_map \
    ""select
      date,
      latitude,
      longitude,
      apple_photos.sha256,
      uploads.ext,
      json_object(
        'title',
        'Taken on ' || date,
        'image',
        'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=400',
        'link',
        'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=1200'
      ) as popup
    from
      apple_photos
      join uploads on apple_photos.sha256 = uploads.sha256
    where
      latitude is not null
    order by
      date desc"" \
    --replace
pipenv run datasette publish now public.db --project dogsheep-photos \
    --about=dogsheep/dogsheep-photos \
    --about_url=""https://github.com/dogsheep/dogsheep-photos"" \
    --install=datasette-json-html \
    --install=datasette-pretty-json \
    --install=datasette-cluster-map>=0.10 \
    --title ""Dogsheep Photos demo""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226481,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631226481,MDEyOklzc3VlQ29tbWVudDYzMTIyNjQ4MQ==,9599,simonw,2020-05-20T04:18:29Z,2020-05-20T04:18:29Z,MEMBER,I just renamed the repository.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226572,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631226572,MDEyOklzc3VlQ29tbWVudDYzMTIyNjU3Mg==,9599,simonw,2020-05-20T04:18:52Z,2020-05-20T04:18:52Z,MEMBER,Need to reconfigure Circle CI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226953,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631226953,MDEyOklzc3VlQ29tbWVudDYzMTIyNjk1Mw==,9599,simonw,2020-05-20T04:20:34Z,2020-05-20T04:20:34Z,MEMBER,"Huh, it looks like Circle CI picked up the name change automatically. https://app.circleci.com/pipelines/github/dogsheep/dogsheep-photos","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227020,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631227020,MDEyOklzc3VlQ29tbWVudDYzMTIyNzAyMA==,9599,simonw,2020-05-20T04:20:48Z,2020-05-20T04:21:16Z,MEMBER,Next time I push a release it will create `dogsheep-photos` on PyPI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227105,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631227105,MDEyOklzc3VlQ29tbWVudDYzMTIyNzEwNQ==,9599,simonw,2020-05-20T04:21:06Z,2020-05-20T04:21:06Z,MEMBER,Then I just need to push a final photos-to-sqlite release that updates the README to tell people about the name change.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227245,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631227245,MDEyOklzc3VlQ29tbWVudDYzMTIyNzI0NQ==,9599,simonw,2020-05-20T04:21:38Z,2020-05-20T04:21:38Z,MEMBER,I'm going to release 0.4 now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229409,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631229409,MDEyOklzc3VlQ29tbWVudDYzMTIyOTQwOQ==,9599,simonw,2020-05-20T04:30:40Z,2020-05-20T04:30:40Z,MEMBER,https://pypi.org/project/photos-to-sqlite/ now links to dogsheep-photos.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229485,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631229485,MDEyOklzc3VlQ29tbWVudDYzMTIyOTQ4NQ==,9599,simonw,2020-05-20T04:31:02Z,2020-05-20T04:31:02Z,MEMBER,https://pypi.org/project/dogsheep-photos/ is live.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/28#issuecomment-751125270,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/28,751125270,MDEyOklzc3VlQ29tbWVudDc1MTEyNTI3MA==,129786,jmelloy,2020-12-24T22:26:22Z,2020-12-24T22:26:22Z,NONE,This comes around if you’ve run the photo export without running an s3 upload. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",624490929,Invalid SQL no such table: main.uploads,
https://github.com/dogsheep/dogsheep-photos/issues/3#issuecomment-934207940,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3,934207940,IC_kwDOD079W843ruHE,1751612,jratike80,2021-10-05T08:57:41Z,2021-10-05T08:57:41Z,NONE,"Maybe the exif-loader from the SpatiaLite project could be useful as a reference even it is written in C and it also saves images as blobs https://www.gaia-gis.it/fossil/spatialite-tools/file?name=exif_loader.c&ci=tip. The tool is also integrated into the spatialite-gui application.

I found some user documentation from the web archive http://web.archive.org/web/20180629041238/https://www.gaia-gis.it/spatialite-2.3.1/spatialite-exif-2.3.1.html.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533481,"Import EXIF data into SQLite - lens used, ISO, aperture etc",
https://github.com/dogsheep/dogsheep-photos/issues/3#issuecomment-934372104,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3,934372104,IC_kwDOD079W843sWMI,41546558,RhetTbull,2021-10-05T12:38:24Z,2021-10-05T12:38:24Z,CONTRIBUTOR,"As dogsheep-photos already uses [osxphotos](https://github.com/RhetTbull/osxphotos) to load photos you can access the EXIF data via osxphotos. Apple Photos imports a small subset of EXIF data at the time the photo is imported and osxphotos provides this via the [exif_info](https://github.com/RhetTbull/osxphotos#exifinfo) property. If you want the full EXIF data, osxphotos also provides a wrapper around [exiftool](https://github.com/RhetTbull/osxphotos#exiftool).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533481,"Import EXIF data into SQLite - lens used, ISO, aperture etc",
https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-791053721,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32,791053721,MDEyOklzc3VlQ29tbWVudDc5MTA1MzcyMQ==,6213,dsisnero,2021-03-05T00:31:27Z,2021-03-05T00:31:27Z,NONE,I am getting the same thing for US West (N. California) us-west-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803333769,KeyError: 'Contents' on running upload,
https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-882091516,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32,882091516,IC_kwDOD079W840k6X8,10793464,aaronyih1,2021-07-18T17:29:39Z,2021-07-18T17:33:02Z,NONE,Same here for US West (N. California) us-west-1. Running on Catalina.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803333769,KeyError: 'Contents' on running upload,
https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-884688833,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32,884688833,IC_kwDOD079W840u0fB,10793464,aaronyih1,2021-07-22T06:40:25Z,2021-07-22T06:40:25Z,NONE,The solution here is to upload an image to the bucket first. It is caused because it does not properly handle the case when there are no images in the bucket.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803333769,KeyError: 'Contents' on running upload,
https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-777951854,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33,777951854,MDEyOklzc3VlQ29tbWVudDc3Nzk1MTg1NA==,675335,leafgarland,2021-02-12T03:54:39Z,2021-02-12T03:54:39Z,NONE,"I think that is a typo in the docs, you can use

    > dogsheep-photos apple-photos photos.db","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803338729,photo-to-sqlite: command not found,
https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778002092,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33,778002092,MDEyOklzc3VlQ29tbWVudDc3ODAwMjA5Mg==,11855322,robmarkcole,2021-02-12T06:19:32Z,2021-02-12T06:19:32Z,NONE,"hi @leafgarland that results in a new error:
```
(venv) (base) Robins-MacBook:datasette robin$ dogsheep-photos apple-photos photos.db
Traceback (most recent call last):
  File ""/Users/robin/datasette/venv/bin/dogsheep-photos"", line 8, in <module>
    sys.exit(cli())
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/cli.py"", line 206, in apple_photos
    db.conn.execute(
sqlite3.OperationalError: no such table: attached.ZGENERICASSET
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803338729,photo-to-sqlite: command not found,
https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778014990,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33,778014990,MDEyOklzc3VlQ29tbWVudDc3ODAxNDk5MA==,675335,leafgarland,2021-02-12T06:54:14Z,2021-02-12T06:54:14Z,NONE,"Ahh, that might be because macOS Big Sur has changed the structure of the photos db. Might need to wait for a later release, there is a PR which adds support for Big Sur. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803338729,photo-to-sqlite: command not found,
https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778246347,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33,778246347,MDEyOklzc3VlQ29tbWVudDc3ODI0NjM0Nw==,41546558,RhetTbull,2021-02-12T15:00:43Z,2021-02-12T15:00:43Z,CONTRIBUTOR,"Yes, Big Sur Photos database doesn't have `ZGENERICASSET` table.   PR #31 will fix this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803338729,photo-to-sqlite: command not found,
https://github.com/dogsheep/dogsheep-photos/issues/35#issuecomment-813249000,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/35,813249000,MDEyOklzc3VlQ29tbWVudDgxMzI0OTAwMA==,1151557,ligurio,2021-04-05T07:37:57Z,2021-04-05T07:37:57Z,NONE,"There are trained ML models used in Photoprism:
- https://dl.photoprism.org/tensorflow/nasnet.zip
- https://dl.photoprism.org/tensorflow/nsfw.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842695374,Support to annotate photos on other than  macOS OSes,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932007,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615932007,MDEyOklzc3VlQ29tbWVudDYxNTkzMjAwNw==,9599,simonw,2020-04-18T19:27:55Z,2020-04-18T19:27:55Z,MEMBER,"Research thread: https://twitter.com/simonw/status/1249049694984011776
> I want to build some software that lets people store their own data in their own S3 bucket, but if possible I'd like not to have to teach people the incantations needed to get their bucket setup and minimum-permission credentials figures out

https://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/ looks useful","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932204,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615932204,MDEyOklzc3VlQ29tbWVudDYxNTkzMjIwNA==,9599,simonw,2020-04-18T19:29:22Z,2020-04-18T19:34:44Z,MEMBER,I'm going to call my bucket `dogsheep-photos-simon`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615933273,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615933273,MDEyOklzc3VlQ29tbWVudDYxNTkzMzI3Mw==,9599,simonw,2020-04-18T19:37:33Z,2020-04-18T19:37:33Z,MEMBER,"https://console.aws.amazon.com/s3/bucket/create?region=us-west-1

![S3_Management_Console](https://user-images.githubusercontent.com/9599/79669552-33e2a380-8171-11ea-9ab5-5785d34f652a.png)

I created it with no public read-write access. I plan to use signed URLs via a transforming proxy to access images for display on the web.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615935577,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615935577,MDEyOklzc3VlQ29tbWVudDYxNTkzNTU3Nw==,9599,simonw,2020-04-18T19:54:59Z,2020-04-18T19:55:30Z,MEMBER,"Creating IAM groups called `dogsheep-photos-simon-read-write` and `dogsheep-photos-simon-read`: https://console.aws.amazon.com/iam/home#/groups - I created them with no attached policies.

Now I can attach an ""inline policy"" to each one. For the read-write group I go here:

https://console.aws.amazon.com/iam/home#/groups/dogsheep-photos-simon-read-write

![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669703-2d086080-8172-11ea-9597-83e0b155193e.png)

Example policies are here: https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html

For the read-write one I went with:
```json
{
    ""Version"": ""2012-10-17"",
    ""Statement"": [
        {
            ""Effect"": ""Allow"",
            ""Action"": ""s3:*"",
            ""Resource"": [
                ""arn:aws:s3:::dogsheep-photos-simon/*""
            ]
        }
    ]
}
```
For the read-only policy I'm going to guess that this is appropriate:

```json
{
    ""Version"": ""2012-10-17"",
    ""Statement"": [
        {
            ""Effect"": ""Allow"",
            ""Action"": [
                ""s3:GetObject*"",
                ""s3:ListBucket""
            ],
            ""Resource"": [
                ""arn:aws:s3:::dogsheep-photos-simon/*""
            ]
        }
    ]
}
```
I tried the policy simulator to test this out: https://policysim.aws.amazon.com/home/index.jsp?#groups/dogsheep-photos-simon-read - this worked:

![IAM_Policy_Simulator](https://user-images.githubusercontent.com/9599/79669893-cd12b980-8173-11ea-8dfb-5660ce3652da.png)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615936880,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615936880,MDEyOklzc3VlQ29tbWVudDYxNTkzNjg4MA==,9599,simonw,2020-04-18T20:04:31Z,2020-04-18T20:04:31Z,MEMBER,"Next step: create two IAM users, one for each of those groups.

https://console.aws.amazon.com/iam/home#/users$new?step=details

![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669931-1bc05380-8174-11ea-9657-0e0c6a692d42.png)

![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79669941-27137f00-8174-11ea-8ce7-249f0d4f96f6.png)

I copied the keys into a secure note in 1password.

Couldn't get into Transmit with them though! https://library.panic.com/transmit/transmit5/iam-roles/ may help.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615941746,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615941746,MDEyOklzc3VlQ29tbWVudDYxNTk0MTc0Ng==,9599,simonw,2020-04-18T20:29:36Z,2020-04-18T20:29:36Z,MEMBER,"I'm going to create another user just for Transmit, with full S3 access.

name: `dogsheep-photos-simon-s3-all-access`

Rather than creating a group for that user, I'm trying the ""Attach existing policies directly"" option:

![IAM_Management_Console](https://user-images.githubusercontent.com/9599/79670182-03513880-8176-11ea-811a-c80aefb4538a.png)

That user DID work with Transmit. I uploaded a test HEIC image. I used Transmit to copy a signed URL for it.

```
~ $ curl -i 'https://dogsheep-photos-simon.s3.us-west-1.amazonaws.com/IMG_7195.HEIC?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAWXFXAI...' | head -n 100
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/1.1 200 OK
x-amz-id-2: gBOCYqZfbNAnv0R/uJ++qm2NbW5SgD4TapgF9RQjzzeDIThcCz/BkKU+YoxlG4NJHlcmMgAHyh4=
x-amz-request-id: C2FE7FCC3BD53A84
Date: Sat, 18 Apr 2020 20:28:54 GMT
Last-Modified: Sat, 18 Apr 2020 20:13:49 GMT
ETag: ""fe3e081239a123ef745517878c53b854""
Accept-Ranges: bytes
Content-Type: image/heic
Content-Length: 1913097
Server: AmazonS3
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615942116,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615942116,MDEyOklzc3VlQ29tbWVudDYxNTk0MjExNg==,9599,simonw,2020-04-18T20:30:56Z,2020-04-18T20:30:56Z,MEMBER,"Next step: attempt a programmatic upload using the `dogsheep-photos-simon-read-write` credentials from a Jupyter notebook.

Also attempt a programmatic bucket listing and read using `dogsheep-photos-simon-read` credentials.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615944806,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615944806,MDEyOklzc3VlQ29tbWVudDYxNTk0NDgwNg==,9599,simonw,2020-04-18T20:41:39Z,2020-04-18T20:41:39Z,MEMBER,"This worked!

![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670712-d868e380-8179-11ea-82a5-5dfd17356113.png)

And this worked:

![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670777-50370e00-817a-11ea-83cd-18ebf5702878.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615945056,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615945056,MDEyOklzc3VlQ29tbWVudDYxNTk0NTA1Ng==,9599,simonw,2020-04-18T20:42:41Z,2020-04-18T20:42:41Z,MEMBER,"But... `list_objects` failed for both of my keys (read and write):

![Dogsheep_Photos_S3_access](https://user-images.githubusercontent.com/9599/79670798-75c41780-817a-11ea-9907-2cbc4a2e497c.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615946537,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615946537,MDEyOklzc3VlQ29tbWVudDYxNTk0NjUzNw==,9599,simonw,2020-04-18T20:48:13Z,2020-04-18T20:48:13Z,MEMBER,"How about generating a signed URL?
```python
read_client.generate_presigned_url(
    ""get_object"",
    Params={
        ""Bucket"": ""dogsheep-photos-simon"",
        ""Key"": ""this_is_fine.jpg"",
    },
    ExpiresIn=600
)
```
Gave me https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398

Which does this:

```
~ $ curl -i 'https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'
HTTP/1.1 307 Temporary Redirect
x-amz-bucket-region: us-west-1
x-amz-request-id: E78CD859AEE21D33
x-amz-id-2: 648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=
Location: https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Sat, 18 Apr 2020 20:47:21 GMT
Server: AmazonS3

<?xml version=""1.0"" encoding=""UTF-8""?>
<Error><Code>TemporaryRedirect</Code><Message>Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.</Message><Endpoint>dogsheep-photos-simon.s3-us-west-1.amazonaws.com</Endpoint><Bucket>dogsheep-photos-simon</Bucket><RequestId>E78CD859AEE21D33</RequestId><HostId>648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=</HostId></Error>~ $ 
```
So it redirects to another URL... which returns this:
```
~ $ curl -i 'https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'
HTTP/1.1 200 OK
x-amz-id-2: XafOl6mswj3yz0GJC9+Ptot1ll5sROVwqsMc10CUUfgpaUANTdIx2GhnONb5d1GVFJ6wlS2j3UY=
x-amz-request-id: 258387C180411AFE
Date: Sat, 18 Apr 2020 20:47:52 GMT
Last-Modified: Sat, 18 Apr 2020 20:37:35 GMT
ETag: ""ee04081c3182a44a1c6944e94012e977""
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 53072
Server: AmazonS3

????JFIF??C
```
So that worked! It did come back with `Content-Type: binary/octet-stream` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947229,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615947229,MDEyOklzc3VlQ29tbWVudDYxNTk0NzIyOQ==,9599,simonw,2020-04-18T20:51:26Z,2020-04-18T20:51:26Z,MEMBER,"Running the upload again like this resulted in the correct content-type:
```python
client.upload_file(
    ""/Users/simonw/Desktop/this_is_fine.jpg"",
    ""dogsheep-photos-simon"",
    ""this_is_fine.jpg"",
    ExtraArgs={
        ""ContentType"": ""image/jpeg""
    }
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947370,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615947370,MDEyOklzc3VlQ29tbWVudDYxNTk0NzM3MA==,9599,simonw,2020-04-18T20:52:13Z,2020-04-18T20:52:13Z,MEMBER,"This is great! I now have a key that can upload photos, and a separate key that can download photos OR generate signed URLs to access those photos.

Next step: a script that starts uploading my photos.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615948102,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615948102,MDEyOklzc3VlQ29tbWVudDYxNTk0ODEwMg==,9599,simonw,2020-04-18T20:56:59Z,2020-04-18T20:56:59Z,MEMBER,"I'm going to start with this:

`photos-to-sqlite upload photos.db ~/path/to/directory`

This will scan the provided directory (and all sub-directories) for image files. It will then:

* Calculate a sha256 of the contents of that file
* Upload the file to a key that's `sha256.jpg` or `.heic`
* Upload a `sha256.json` file with the original path to the image
* Add that image to a `uploads` table in `photos.db`

Stretch goal: grab the EXIF data and include that in the `.json` upload AND the `uploads` database table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615957385,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615957385,MDEyOklzc3VlQ29tbWVudDYxNTk1NzM4NQ==,9599,simonw,2020-04-18T21:56:16Z,2020-04-18T21:58:11Z,MEMBER,Got this working! I'll do EXIF in a separate ticket #3.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/5#issuecomment-615949574,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/5,615949574,MDEyOklzc3VlQ29tbWVudDYxNTk0OTU3NA==,9599,simonw,2020-04-18T21:06:07Z,2020-04-18T21:06:07Z,MEMBER,"```
$ photos-to-sqlite s3-auth
Create S3 credentials and paste them here:

Access key ID: xxx
Secret access key: yyy
$ cat auth.json 
{
    ""access_key_id"": ""xxx"",
    ""secret_access_key"": ""yyy""
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602551638,photos-to-sqlite s3-auth command,
https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615979923,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6,615979923,MDEyOklzc3VlQ29tbWVudDYxNTk3OTkyMw==,9599,simonw,2020-04-18T23:36:02Z,2020-04-18T23:36:02Z,MEMBER,"I'll use a Click progress bar. To do this I need to first calculate the sum number of bytes in the photos that are going to be uploaded, then run the upload.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602575575,Add progress bar to upload command,
https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615983393,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6,615983393,MDEyOklzc3VlQ29tbWVudDYxNTk4MzM5Mw==,9599,simonw,2020-04-18T23:53:10Z,2020-04-18T23:53:10Z,MEMBER,"```
$ photos-to-sqlite upload photos3.db ~/Pictures/Photos\ Library.photoslibrary/Masters/2020
Uploading 2.09 GB  [##----------------------------------]    6%  00:36:37
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602575575,Add progress bar to upload command,
https://github.com/dogsheep/dogsheep-photos/issues/7#issuecomment-615993178,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7,615993178,MDEyOklzc3VlQ29tbWVudDYxNTk5MzE3OA==,9599,simonw,2020-04-19T00:37:08Z,2020-04-19T00:37:08Z,MEMBER,https://pypi.org/project/ImageHash/ Is one option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602585497,Integrate image content hashing,
https://github.com/dogsheep/dogsheep-photos/issues/7#issuecomment-906015471,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7,906015471,IC_kwDOD079W842ALLv,18232,dkam,2021-08-26T02:01:01Z,2021-08-26T02:01:01Z,NONE,Perceptual hashes might be what you're after : http://phash.org,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602585497,Integrate image content hashing,
https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100434,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8,618100434,MDEyOklzc3VlQ29tbWVudDYxODEwMDQzNA==,9599,simonw,2020-04-23T00:02:53Z,2020-04-23T00:02:53Z,MEMBER,"I don't think it matters one way or the other - I'm storing the sha256 in the filename, so the fact that I could read the MD5 back from the list bucket operation doesn't give me any benefits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605147638,Should I have used MD5 instead of SHA256?,
https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100658,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8,618100658,MDEyOklzc3VlQ29tbWVudDYxODEwMDY1OA==,9599,simonw,2020-04-23T00:03:35Z,2020-04-23T00:03:35Z,MEMBER,"Also MD5 isn't guaranteed for the ETag:
> If an object is created by either the Multipart Upload or Part Copy operation, the ETag is not an MD5 digest, regardless of the method of encryption.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605147638,Should I have used MD5 instead of SHA256?,
https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618724149,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9,618724149,MDEyOklzc3VlQ29tbWVudDYxODcyNDE0OQ==,9599,simonw,2020-04-23T23:35:29Z,2020-04-23T23:35:29Z,MEMBER,"```
% photos-to-sqlite upload photos.db ~/Pictures/Photos\ Library.photoslibrary/originals
Fetching existing keys from S3...
Got 22,446 existing keys
Calculating hashes  [####--------------------------------]   13%  00:04:14
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605938063,"upload command should be resumable, should only upload photos not already uploaded",
https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618725155,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9,618725155,MDEyOklzc3VlQ29tbWVudDYxODcyNTE1NQ==,9599,simonw,2020-04-23T23:39:14Z,2020-04-23T23:39:14Z,MEMBER,"A few minutes later...
```
Fetching existing keys from S3...
Got 22,446 existing keys
Calculating hashes  [####################################]  100%          
22,441 hashed files, 610 are not yet in S3
Uploading 0.99 GB
Uploading 610 photos  [------------------------------------]  1/610  03:10:35
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605938063,"upload command should be resumable, should only upload photos not already uploaded",
https://github.com/dogsheep/dogsheep-photos/pull/29#issuecomment-739058820,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/29,739058820,MDEyOklzc3VlQ29tbWVudDczOTA1ODgyMA==,9599,simonw,2020-12-04T22:32:35Z,2020-12-04T22:32:35Z,MEMBER,Thanks for this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638375985,Fixed bug in SQL query for photo scores,
https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1035717429,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31,1035717429,IC_kwDOD079W849u8s1,18504,harperreed,2022-02-11T01:55:38Z,2022-02-11T01:55:38Z,NONE,I would love this merged! ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771511344,Update for Big Sur,
https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1190995982,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31,1190995982,IC_kwDOD079W85G_SgO,19231792,jakewilkins,2022-07-21T03:26:38Z,2023-04-14T22:41:31Z,NONE,"👋 Any update on getting this merged?

Alternatively, is there a work around for this issue to unblock myself?

edit to add: huge fan of both this project and `osxphotos`, thanks so much for your work here 🙏  If I had any experience with Python I would offer to help but somehow I've managed to not write any Python in 10+ years of programming 😅 

Edit again to add:

> Alternatively, is there a work around for this issue to unblock myself?

Yes, there is. I was able to apply the patch of this PR and it applies (mostly) cleanly and works.

- verified I have a high enough version of `osxphotos`
- downloaded the .patch of this (by appending `.patch` to the URL)
- edited the patch to remove the `setup.py` changes
- `cd` to the directory containing `dogsheep-photos` and `git apply 31.patch`
","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771511344,Update for Big Sur,
https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1382655354,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31,1382655354,IC_kwDOD079W85SaaV6,2704860,fidiego,2023-01-14T04:08:36Z,2023-01-14T04:08:36Z,NONE,"I just tried this branch and saw some errors. 

I installed this PR locally with:

```bash
pip install https://github.com/RhetTbull/dogsheep-photos/archive/update_for_bigsur.zip
```

<details>
<summary>System Details</summary>

**OS:** MacOS Monterey
**Python Version:** Python 3.10.8

</details>

<details>
<summary>Stacktrace</summary>

```python
Traceback (most recent call last):
  File ""/Users/df/.venvs/photo-experiments/bin/dogsheep-photos"", line 8, in <module>
    sys.exit(cli())
  File ""/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py"", line 1130, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py"", line 1055, in main
    rv = self.invoke(ctx)
  File ""/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py"", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py"", line 760, in invoke
    return __callback(*args, **kwargs)
  File ""/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/dogsheep_photos/cli.py"", line 254, in apple_photos
    sha256 = calculate_hash(pathlib.Path(photo.path))
  File ""/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py"", line 960, in __new__
    self = cls._from_parts(args)
  File ""/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py"", line 594, in _from_parts
    drv, root, parts = self._parse_args(args)
  File ""/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py"", line 578, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
```
</details>","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771511344,Update for Big Sur,
https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-748562330,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31,748562330,MDEyOklzc3VlQ29tbWVudDc0ODU2MjMzMA==,41546558,RhetTbull,2020-12-20T04:45:08Z,2020-12-20T04:45:08Z,CONTRIBUTOR,Fixes the issue mentioned here: https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436115,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771511344,Update for Big Sur,
https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-811362316,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31,811362316,MDEyOklzc3VlQ29tbWVudDgxMTM2MjMxNg==,871250,PabloLerma,2021-03-31T19:14:39Z,2021-03-31T19:14:39Z,NONE,👋  could I help somehow for this to be merged? As Big Sur is going to be more used as the time goes I think it would be nice to merge and publish a new version. Nice work!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771511344,Update for Big Sur,
https://github.com/dogsheep/dogsheep-photos/pull/36#issuecomment-1006708046,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/36,1006708046,IC_kwDOD079W848ASVO,71983,scoates,2022-01-06T16:04:46Z,2022-01-06T16:04:46Z,NONE,"This one got me, today, too. 👍","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988493790,Correct naming of tool in readme,
https://github.com/dogsheep/dogsheep.github.io/pull/6#issuecomment-1021264135,https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/6,1021264135,IC_kwDODMzF1s4830EH,1151557,ligurio,2022-01-25T14:52:40Z,2022-01-25T14:52:40Z,NONE,"@simonw, could you review?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842765105,Add testres-db tool,
https://github.com/dogsheep/evernote-to-sqlite/issues/1#issuecomment-706775706,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/1,706775706,MDEyOklzc3VlQ29tbWVudDcwNjc3NTcwNg==,9599,simonw,2020-10-11T22:14:00Z,2020-10-11T22:14:00Z,MEMBER,A live demo would be good too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718934942,Documentation on how to use this with Datasette,
https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777690332,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11,777690332,MDEyOklzc3VlQ29tbWVudDc3NzY5MDMzMg==,3613583,dskrad,2021-02-11T18:16:01Z,2021-02-11T18:16:01Z,NONE,"I solved this issue by modifying line 31 of utils.py

    content = ET.tostring(ET.fromstring(content_xml.strip())).decode(""utf-8"")","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792851444,XML parse error,
https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777798330,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11,777798330,MDEyOklzc3VlQ29tbWVudDc3Nzc5ODMzMA==,9599,simonw,2021-02-11T21:18:58Z,2021-02-11T21:18:58Z,MEMBER,Thanks for the fix!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792851444,XML parse error,
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905203570,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13,905203570,IC_kwDOEhK-wc419E9y,9599,simonw,2021-08-25T05:51:22Z,2021-08-25T05:53:27Z,MEMBER,"The debugger showed me that it broke on a string that looked like this:
```xml
<?xml version=""1.0"" encoding=""UTF-8"" standalone=""no""?>
<!DOCTYPE en-note SYSTEM ""http://xml.evernote.com/pub/enml2.dtd"">
<en-note>
  <h1 title=""Q3 2018 Reflection & Development"">
    <span title=Q3 2018 Reflection & Development"">
      Q3 2018 Reflection & Development
    </span>
  </h1>
  ...
```
Yeah that is not valid XML!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978743426,xml.etree.ElementTree.ParseError: not well-formed (invalid token),
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905206234,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13,905206234,IC_kwDOEhK-wc419Fna,9599,simonw,2021-08-25T05:58:42Z,2021-08-25T05:58:42Z,MEMBER,"https://github.com/dogsheep/evernote-to-sqlite/blob/36a466f142e5bad52719851c2fbda0c05cd35b99/evernote_to_sqlite/utils.py#L34-L42

Not sure why I was round-tripping the `content_xml` like that - I will try not doing that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978743426,xml.etree.ElementTree.ParseError: not well-formed (invalid token),
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906635938,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13,906635938,IC_kwDOEhK-wc42Ciqi,9599,simonw,2021-08-26T18:18:27Z,2021-08-26T18:18:27Z,MEMBER,"It looks like I was using the round-trip to dump the `<?xml version=""1.0"" encoding=""UTF-8"" standalone=""no""?>` and `<!DOCTYPE` prefixes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978743426,xml.etree.ElementTree.ParseError: not well-formed (invalid token),
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906646452,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13,906646452,IC_kwDOEhK-wc42ClO0,9599,simonw,2021-08-26T18:34:34Z,2021-08-26T18:35:20Z,MEMBER,"I tried this ampersand fix: https://regex101.com/r/ojU2H9/1
```python

# https://regex101.com/r/ojU2H9/1
_invalid_ampersand_re = re.compile(r'&(?![a-z0-9]+;)')

def fix_bad_xml(xml):
    # More fixes for things like '&' not as part of an entity
    return _invalid_ampersand_re.sub('&amp;', xml)
```

Even with that I'm still getting total garbage in the `<en-note>` content - it's just HTML, not even trying to be XML.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978743426,xml.etree.ElementTree.ParseError: not well-formed (invalid token),
https://github.com/dogsheep/evernote-to-sqlite/issues/14#issuecomment-911772943,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/14,911772943,IC_kwDOEhK-wc42WI0P,46968,step21,2021-09-02T14:53:11Z,2021-09-02T14:53:11Z,NONE,"Additionally, assuming the line numbers match up with the provided enenx file, the mentioned line plus one before and after is  as follows:
```
<![CDATA[>]]>
</span></div>
<div style=""padding: 0px; font-family: Arial, sans-serif; font-size: 12px; line-height: 16px; white-space: pre-wrap;""><br style="" padding: 0px;""/>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",986829194,xml.etree.ElementTree.Parse Error - mismatched tag,
https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706784028,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4,706784028,MDEyOklzc3VlQ29tbWVudDcwNjc4NDAyOA==,9599,simonw,2020-10-11T23:20:32Z,2020-10-11T23:20:32Z,MEMBER,I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938508,Configure FTS + add an index on the date columns,
https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706786548,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4,706786548,MDEyOklzc3VlQ29tbWVudDcwNjc4NjU0OA==,9599,simonw,2020-10-11T23:39:46Z,2020-10-11T23:39:46Z,MEMBER,Should have used porter stemming for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938508,Configure FTS + add an index on the date columns,
https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776180,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5,706776180,MDEyOklzc3VlQ29tbWVudDcwNjc3NjE4MA==,9599,simonw,2020-10-11T22:17:55Z,2020-10-11T22:17:55Z,MEMBER,"We could even do server-side thumbnailing for some of these images, but I'm inclined to serve up the full size ones and set a width on the image element based on the `width` attribute on `<en-media>`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from <en-media> tags inline in Datasette,
https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776242,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5,706776242,MDEyOklzc3VlQ29tbWVudDcwNjc3NjI0Mg==,9599,simonw,2020-10-11T22:18:30Z,2020-10-11T22:19:48Z,MEMBER,"Alternatively, rather than relying on `datasette-media` this could base64-embed the images. `evernote-to-sqlite` could register itself as a Datasette plugin that knows how to do this.

Maybe rename the column to `evernote_content` and register a render cell hook that knows how to rewrite those note bodies so that they are visible?

Might need to feed them through Bleach too, just in case any nasty code can get into them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from <en-media> tags inline in Datasette,
https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776447,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5,706776447,MDEyOklzc3VlQ29tbWVudDcwNjc3NjQ0Nw==,9599,simonw,2020-10-11T22:20:32Z,2020-10-11T22:20:32Z,MEMBER,Or... I could do this client-side. JavaScript that looks for `<en-media>` tags and fetches the data using `fetch()` wouldn't be too hard to write.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from <en-media> tags inline in Datasette,
https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776680,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5,706776680,MDEyOklzc3VlQ29tbWVudDcwNjc3NjY4MA==,9599,simonw,2020-10-11T22:22:16Z,2020-10-11T22:22:16Z,MEMBER,"Maybe the best way do this is with a custom route, `/-/evernote/note-id` - that way I can clean the HTML and resolve the other things in the `<en-note>` structure without using `render_cell()` and the like. My concern about using `render_cell()` is that it could lead to weird security problems when combined with `?sql=` queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from <en-media> tags inline in Datasette,
https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776808,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5,706776808,MDEyOklzc3VlQ29tbWVudDcwNjc3NjgwOA==,9599,simonw,2020-10-11T22:23:14Z,2020-10-11T22:23:14Z,MEMBER,... but it's still important to be able to get to the rendered note directly from the browse notes `/evernote/notes` page. Maybe use a simple `render_cell()` hook that just knows how to generate the link to the rendered note page?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from <en-media> tags inline in Datasette,
https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706834800,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5,706834800,MDEyOklzc3VlQ29tbWVudDcwNjgzNDgwMA==,9599,simonw,2020-10-12T03:24:57Z,2020-10-16T20:16:28Z,MEMBER,"Here's my first attempt at a plugin for this:
```python
from datasette import hookimpl
import jinja2

START = ""<en-note""
END = ""</en-note>""
TEMPLATE = """"""
<div style=""max-width: 500px; white-space: normal; overflow-wrap: break-word;"">{}</div>
"""""".strip()

EN_MEDIA_SCRIPT = """"""
Array.from(document.querySelectorAll('en-media')).forEach(el => {
    let hash = el.getAttribute('hash');
    let type = el.getAttribute('type');
    let path = `/evernote/resources_data/${hash}.json?_shape=array`;
    fetch(path).then(r => r.json()).then(rows => {
        let b64 = rows[0].data.encoded;
        let data = `data:${type};base64,${b64}`;
        el.innerHTML = `<img style=""max-width: 300px"" src=""${data}"">`;
    });
});
""""""


@hookimpl
def render_cell(value, table):
    if not table:
        # Don't render content from arbitrary SQL queries, could be XSS hole
        return
    if not value or not isinstance(value, str):
        return
    value = value.strip()
    if value.startswith(START) and value.endswith(END):
        trimmed = value[len(START) : -len(END)]
        trimmed = trimmed.split("">"", 1)[1]
        # Replace those horrible double newlines
        trimmed = trimmed.replace(""<div><br /></div>"", ""<br>"")
        return jinja2.Markup(TEMPLATE.format(trimmed))


@hookimpl
def extra_body_script():
    return EN_MEDIA_SCRIPT
```
It works!

It does however demonstrate that Evernote's ""clip this webpage"" feature means there is a LOT of weird HTML that can get into a note. It looks like they've filtered out the scripts but I wouldn't bet on it - they certainly don't filter out many of the inline styles. So running Bleach is almost certainly a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from <en-media> tags inline in Datasette,
https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785086,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6,706785086,MDEyOklzc3VlQ29tbWVudDcwNjc4NTA4Ng==,9599,simonw,2020-10-11T23:28:50Z,2020-10-11T23:28:50Z,MEMBER,"The XML for the OCR stuff is a bit weird. Currently I'm doing this to it:

https://github.com/dogsheep/evernote-to-sqlite/blob/c33d7b043a45eb3e88676e5fa3ce31755199d9f8/evernote_to_sqlite/utils.py#L70-L78

This can produce some odd results, for example:

> Sure 'Sure, 'Sure. Sure, Sure. sure sure. sure ? If you If Yau [you live jive In m 1n an area devoid of natural wonders, wanders, wonders ? wonders wonders. your mind will be blown, blown' blown. blown ? -e i ? ,1 IL it ? at ? KY ? fl ft bat at

Which came from this image:

![image](https://user-images.githubusercontent.com/9599/95692952-5dd7c880-0bde-11eb-939a-d10b800a4105.png)

The XML for that is:

```xml
<recoIndex docType=""unknown"" objType=""image"" objID=""05ffb72b307bf495f064243c7099d94f"" engineVersion=""6.5.17.7"" recoType=""service"" lang=""en"" objWidth=""1000"" objHeight=""1504"">
  <item x=""68"" y=""75"" w=""104"" h=""37"">
    <t w=""60"">Sure</t>
    <t w=""52"">'Sure,</t>
    <t w=""47"">'Sure.</t>
    <t w=""33"">Sure,</t>
    <t w=""26"">Sure.</t>
  </item>
  <item x=""182"" y=""83"" w=""92"" h=""26"">
    <t w=""62"">sure</t>
    <t w=""58"">sure.</t>
    <t w=""46"">sure ?</t>
  </item>
  <item x=""69"" y=""132"" w=""107"" h=""45"">
    <t w=""81"">If you</t>
    <t w=""64"">If Yau</t>
    <t w=""31"">[you</t>
  </item>
  <item x=""186"" y=""132"" w=""67"" h=""35"">
    <t w=""85"">live</t>
    <t w=""51"">jive</t>
  </item>
  <item x=""263"" y=""140"" w=""36"" h=""27"">
    <t w=""82"">In</t>
    <t w=""56"">m</t>
    <t w=""53"">1n</t>
  </item>
  <item x=""309"" y=""140"" w=""53"" h=""27"">
    <t w=""82"">an</t>
  </item>
  <item x=""372"" y=""141"" w=""90"" h=""26"">
    <t w=""94"">area</t>
  </item>
  <item x=""472"" y=""132"" w=""138"" h=""35"">
    <t w=""85"">devoid</t>
  </item>
  <item x=""620"" y=""132"" w=""43"" h=""35"">
    <t w=""82"">of</t>
  </item>
  <item x=""68"" y=""190"" w=""137"" h=""35"">
    <t w=""87"">natural</t>
  </item>
  <item x=""215"" y=""190"" w=""187"" h=""39"">
    <t w=""57"">wonders,</t>
    <t w=""55"">wanders,</t>
    <t w=""52"">wonders ?</t>
    <t w=""45"">wonders</t>
    <t w=""42"">wonders.</t>
  </item>
  <item x=""410"" y=""198"" w=""98"" h=""36"">
    <t w=""88"">your</t>
  </item>
  <item x=""518"" y=""190"" w=""102"" h=""35"">
    <t w=""86"">mind</t>
  </item>
  <item x=""630"" y=""190"" w=""69"" h=""34"">
    <t w=""87"">will</t>
  </item>
  <item x=""709"" y=""190"" w=""55"" h=""35"">
    <t w=""82"">be</t>
  </item>
  <item x=""774"" y=""190"" w=""137"" h=""34"">
    <t w=""56"">blown,</t>
    <t w=""55"">blown'</t>
    <t w=""48"">blown.</t>
    <t w=""48"">blown ?</t>
  </item>
  <item x=""166"" y=""736"" w=""8"" h=""6"">
    <t w=""66"">-e</t>
  </item>
  <item x=""273"" y=""966"" w=""29"" h=""21"">
    <t w=""11"">i ?</t>
  </item>
  <item x=""281"" y=""1004"" w=""28"" h=""11"">
    <t w=""11"">,1</t>
  </item>
  <item x=""512"" y=""1083"" w=""10"" h=""7"">
    <t w=""10"">IL</t>
  </item>
  <item x=""29"" y=""1447"" w=""7"" h=""23"">
    <t w=""17"">it ?</t>
    <t w=""15"">at ?</t>
    <t w=""13"">KY ?</t>
  </item>
  <item x=""414"" y=""841"" w=""8"" h=""16"">
    <t w=""22"">fl</t>
    <t w=""20"">ft</t>
    <t w=""20"">bat</t>
    <t w=""19"">at</t>
  </item>
</recoIndex>
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718949182,Better handling of OCR data,
https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785201,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6,706785201,MDEyOklzc3VlQ29tbWVudDcwNjc4NTIwMQ==,9599,simonw,2020-10-11T23:29:39Z,2020-10-11T23:29:39Z,MEMBER,"It looks to me like each of those `<item>` blocks has a number of guesses in order of confidence:
```xml
  <item x=""215"" y=""190"" w=""187"" h=""39"">
    <t w=""57"">wonders,</t>
    <t w=""55"">wanders,</t>
    <t w=""52"">wonders ?</t>
    <t w=""45"">wonders</t>
    <t w=""42"">wonders.</t>
  </item>
```
So maybe the best approach here is to just take the first `t` element within each `item`.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718949182,Better handling of OCR data,
https://github.com/dogsheep/evernote-to-sqlite/issues/7#issuecomment-777827396,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7,777827396,MDEyOklzc3VlQ29tbWVudDc3NzgyNzM5Ng==,9599,simonw,2021-02-11T22:13:14Z,2021-02-11T22:13:14Z,MEMBER,My best guess is that you have an older version of `sqlite-utils` installed here - the `replace=True` argument was added in version 2.0. I've bumped the dependency in `setup.py`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743297582,evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace',
https://github.com/dogsheep/evernote-to-sqlite/issues/9#issuecomment-777821383,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9,777821383,MDEyOklzc3VlQ29tbWVudDc3NzgyMTM4Mw==,9599,simonw,2021-02-11T22:01:28Z,2021-02-11T22:01:28Z,MEMBER,"Aha! I think I've figured out what's going on here.

The CData blocks containing the notes look like this:

`<![CDATA[<!DOCTYPE en-note SYSTEM ""http://xml.evernote.com/pub/enml2.dtd""><en-note><div>This note includes two images.</div><div><br /></div>...`

The DTD at http://xml.evernote.com/pub/enml2.dtd includes some entities:

```
<!--=========== External character mnemonic entities ===================-->

<!ENTITY % HTMLlat1 PUBLIC
   ""-//W3C//ENTITIES Latin 1 for XHTML//EN""
   ""http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent"">
%HTMLlat1;

<!ENTITY % HTMLsymbol PUBLIC
   ""-//W3C//ENTITIES Symbols for XHTML//EN""
   ""http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent"">
%HTMLsymbol;

<!ENTITY % HTMLspecial PUBLIC
   ""-//W3C//ENTITIES Special for XHTML//EN""
   ""http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent"">
%HTMLspecial;
```
So I need to be able to handle all of those different entities. I think I can do that using `html.entities.entitydefs` from the Python standard library, which looks a bit like this:

```python
{'Aacute': 'Á',
 'aacute': 'á',
 'Aacute;': 'Á',
 'aacute;': 'á',
 'Abreve;': 'Ă',
 'abreve;': 'ă',
 'ac;': '∾',
 'acd;': '∿',
# ...
}
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",748372469,ParseError: undefined entity &scaron;,
https://github.com/dogsheep/evernote-to-sqlite/pull/10#issuecomment-777839351,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10,777839351,MDEyOklzc3VlQ29tbWVudDc3NzgzOTM1MQ==,9599,simonw,2021-02-11T22:37:55Z,2021-02-11T22:37:55Z,MEMBER,"I've merged these changes by hand now, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770712149,BugFix for encoding and not update info.,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544646516,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,544646516,MDEyOklzc3VlQ29tbWVudDU0NDY0NjUxNg==,9599,simonw,2019-10-21T18:30:14Z,2019-10-21T18:30:14Z,MEMBER,"Thanks to help from Dr. Laura Cantino at Science Hack Day San Francisco I've been able to pull together this query:

```sql
select rsid, genotype, case genotype
  when 'AA' then 'brown eye color, 80% of the time'
  when 'AG' then 'brown eye color'
  when 'GG' then 'blue eye color, 99% of the time'
end as interpretation from genome where rsid = 'rs12913832'
```

See also https://www.snpedia.com/index.php/Rs12913832 - in particular this table:

<img width=""321"" alt=""rs12913832_-_SNPedia"" src=""https://user-images.githubusercontent.com/9599/67232392-216ff300-f3f6-11e9-8e14-b5f50c0c0d16.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544648863,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,544648863,MDEyOklzc3VlQ29tbWVudDU0NDY0ODg2Mw==,9599,simonw,2019-10-21T18:36:03Z,2019-10-21T18:36:03Z,MEMBER,"<img width=""1418"" alt=""natalie__select_rsid__genotype__case_genotype_when__AA__then__brown_eye_color__80__of_the_time__when__AG__then__brown_eye_color__when__GG__then__blue_eye_color__99__of_the_time__end_as_interpretation_from_genome_where_rsid____rs12913832__an"" src=""https://user-images.githubusercontent.com/9599/67232810-f4701000-f3f6-11e9-90e2-8fe2cca1d98d.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765495861,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765495861,MDEyOklzc3VlQ29tbWVudDc2NTQ5NTg2MQ==,25372415,cobiadigital,2021-01-22T15:44:00Z,2021-01-22T15:44:00Z,NONE,"Risk of autoimmune disorders: https://www.snpedia.com/index.php/Genotype
```
select rsid, genotype, case genotype
  when 'AA' then '2x risk of rheumatoid arthritis and other autoimmune diseases'
  when 'GG' then 'Normal risk for autoimmune disorders'
end as interpretation from genome where rsid = 'rs2476601'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765498984,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765498984,MDEyOklzc3VlQ29tbWVudDc2NTQ5ODk4NA==,25372415,cobiadigital,2021-01-22T15:48:25Z,2021-01-22T15:49:33Z,NONE,"The ""Warrior Gene"" https://www.snpedia.com/index.php/Rs4680

```
select rsid, genotype, case genotype
  when 'AA' then '(worrier) advantage in memory and attention tasks'
  when 'AG' then 'Intermediate dopamine levels, other effects'
  when 'GG' then '(warrior) multiple associations, see details'
end as interpretation from genome where rsid = 'rs4680'
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765502845,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765502845,MDEyOklzc3VlQ29tbWVudDc2NTUwMjg0NQ==,25372415,cobiadigital,2021-01-22T15:53:19Z,2021-01-22T15:53:19Z,NONE,"rs7903146 Influences risk of Type-2 diabetes
https://www.snpedia.com/index.php/Rs7903146
```
select rsid, genotype, case genotype
  when 'CC' then 'Normal (lower) risk of Type 2 Diabetes and Gestational Diabetes.'
  when 'CT' then '1.4x increased risk for diabetes (and perhaps colon cancer).'
  when 'TT' then '2x increased risk for Type-2 diabetes'
end as interpretation from genome where rsid = 'rs7903146'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765506901,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765506901,MDEyOklzc3VlQ29tbWVudDc2NTUwNjkwMQ==,25372415,cobiadigital,2021-01-22T15:58:41Z,2021-01-22T15:58:58Z,NONE,"Both rs10757274 and rs2383206 can both indicate higher risks of heart disease
https://www.snpedia.com/index.php/Rs2383206

```
select rsid, genotype, case genotype
  when 'AA' then 'Normal'
  when 'AG' then '~1.2x increased risk for heart disease'
  when 'GG' then '~1.3x increased risk for heart disease'
end as interpretation from genome where rsid = 'rs10757274'
```

```
select rsid, genotype, case genotype
  when 'AA' then 'Normal'
  when 'AG' then '1.4x increased risk for heart disease'
  when 'GG' then '1.7x increased risk for heart disease'
end as interpretation from genome where rsid = 'rs2383206'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765523517,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765523517,MDEyOklzc3VlQ29tbWVudDc2NTUyMzUxNw==,25372415,cobiadigital,2021-01-22T16:20:25Z,2021-01-22T16:20:25Z,NONE,"rs53576: the oxytocin receptor (OXTR) gene

```
select rsid, genotype, case genotype
  when 'AA' then 'Lack of empathy?'
  when 'AG' then 'Lack of empathy?'
  when 'GG' then 'Optimistic and empathetic; handle stress well'
end as interpretation from genome where rsid = 'rs53576'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765525338,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765525338,MDEyOklzc3VlQ29tbWVudDc2NTUyNTMzOA==,25372415,cobiadigital,2021-01-22T16:22:44Z,2021-01-22T16:22:44Z,NONE,"rs1333049 associated with coronary artery disease
https://www.snpedia.com/index.php/Rs1333049
```

select rsid, genotype, case genotype
  when 'CC' then '1.9x increased risk for coronary artery disease'
  when 'CG' then '1.5x increased risk for CAD'
  when 'GG' then 'normal'
end as interpretation from genome where rsid = 'rs1333049'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-831004775,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,831004775,MDEyOklzc3VlQ29tbWVudDgzMTAwNDc3NQ==,25372415,cobiadigital,2021-05-03T03:46:23Z,2021-05-03T03:46:23Z,NONE,"RS1800955 is related to novelty seeking and ADHD
https://www.snpedia.com/index.php/Rs1800955

`select rsid, genotype, case genotype
  when 'CC' then 'increased susceptibility to novelty seeking'
  when 'CT' then 'increased susceptibility to novelty seeking'
  when 'TT' then 'normal'
end as interpretation from genome where rsid = 'rs1800955'`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-549230337,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10,549230337,MDEyOklzc3VlQ29tbWVudDU0OTIzMDMzNw==,9599,simonw,2019-11-04T05:47:18Z,2019-11-04T05:47:18Z,MEMBER,This definition isn't quite right - it's not pulling the identity of the user who starred the repo (`users.login` ends up being the owner login instead).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516967682,Add this repos_starred view,
https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622461122,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10,622461122,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTEyMg==,9599,simonw,2020-05-01T16:34:39Z,2020-05-01T16:34:39Z,MEMBER,Blocked on #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516967682,Add this repos_starred view,
https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622980203,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10,622980203,MDEyOklzc3VlQ29tbWVudDYyMjk4MDIwMw==,9599,simonw,2020-05-02T16:34:29Z,2020-05-02T16:34:29Z,MEMBER,"Fixed definition:
```sql
select
  stars.starred_at,
  starring_user.login as starred_by,
  repos.*
from
  repos
  join stars on repos.id = stars.repo
  join users as starring_user on stars.user = starring_user.id
  join users on repos.owner = users.id
order by
  starred_at desc;
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516967682,Add this repos_starred view,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594151327,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,594151327,MDEyOklzc3VlQ29tbWVudDU5NDE1MTMyNw==,9599,simonw,2020-03-03T20:26:15Z,2020-03-03T20:32:23Z,MEMBER,"Better version (since this also includes JSON array of repository topics):
```sql
CREATE VIEW recent_releases AS select
  repos.rowid as rowid,
  json_object(""label"", repos.full_name, ""href"", repos.html_url) as repo,
  json_object(
    ""href"",
    releases.html_url,
    ""label"",
    releases.name
  ) as release,
  substr(releases.published_at, 0, 11) as date,
  releases.body as body_markdown,
  releases.published_at,
  coalesce(repos.topics, '[]') as topics
from
  releases
  join repos on repos.id = releases.repo
order by
  releases.published_at desc
```
That `repos.rowid as rowid` bit is necessary because otherwise clicking on a link in facet-by-topic doesn't return any results.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,Add this view for seeing new releases,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-594155249,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,594155249,MDEyOklzc3VlQ29tbWVudDU5NDE1NTI0OQ==,9599,simonw,2020-03-03T20:35:17Z,2020-03-03T20:35:17Z,MEMBER,"`swarm-to-sqlite` has an example of adding views here: https://github.com/dogsheep/swarm-to-sqlite/blob/f2c89dd613fb8a7f14e5267ccc2145463b996190/swarm_to_sqlite/utils.py#L141

I think that approach can be approved by first checking if the view exists, then dropping it, then recreating it. Could even try to see if the view exists and matches what we were going to set it to and do nothing if that is the case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,Add this view for seeing new releases,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,622461223,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw==,9599,simonw,2020-05-01T16:34:52Z,2020-05-01T16:34:52Z,MEMBER,Blocked on #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,Add this view for seeing new releases,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,622989874,MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA==,9599,simonw,2020-05-02T17:46:14Z,2020-05-02T17:46:14Z,MEMBER,Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,Add this view for seeing new releases,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,622998813,MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw==,9599,simonw,2020-05-02T18:58:17Z,2020-05-02T18:58:17Z,MEMBER,Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,Add this view for seeing new releases,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602861730,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602861730,MDEyOklzc3VlQ29tbWVudDYwMjg2MTczMA==,9599,simonw,2020-03-23T21:19:16Z,2020-03-23T21:19:42Z,MEMBER,I'm going to do this with a scheduled GitHub Action in this repo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862236,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602862236,MDEyOklzc3VlQ29tbWVudDYwMjg2MjIzNg==,9599,simonw,2020-03-23T21:20:26Z,2020-03-23T21:20:26Z,MEMBER,I'll run the `commits` and `issues` and `issue-comments` commands in addition to the `releases` command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602862967,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602862967,MDEyOklzc3VlQ29tbWVudDYwMjg2Mjk2Nw==,9599,simonw,2020-03-23T21:22:04Z,2020-03-23T21:22:04Z,MEMBER,Following these instructions: https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602895896,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602895896,MDEyOklzc3VlQ29tbWVudDYwMjg5NTg5Ng==,9599,simonw,2020-03-23T22:42:25Z,2020-03-23T22:42:25Z,MEMBER,"Urgh this is such a mess! I should have done this on a branch / pull request to avoid polluting my main master history, but never mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602916947,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602916947,MDEyOklzc3VlQ29tbWVudDYwMjkxNjk0Nw==,9599,simonw,2020-03-23T23:38:06Z,2020-03-23T23:38:06Z,MEMBER,Woohoo! https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602917713,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602917713,MDEyOklzc3VlQ29tbWVudDYwMjkxNzcxMw==,9599,simonw,2020-03-23T23:40:29Z,2020-03-23T23:40:29Z,MEMBER,"Most recently updated issues across all Dogsheep repos, with faceting: https://github-to-sqlite-j7hipcg4aq-uc.a.run.app/github/issues?_facet=repo&_facet=user&_facet=state&_facet=author_association&_facet=type&_sort_desc=updated_at","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602918689,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602918689,MDEyOklzc3VlQ29tbWVudDYwMjkxODY4OQ==,9599,simonw,2020-03-23T23:43:39Z,2020-03-23T23:47:50Z,MEMBER,I pointed https://github-to-sqlite.dogsheep.net/ at it. May take a few minutes for the new certificate to provision though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602919058,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602919058,MDEyOklzc3VlQ29tbWVudDYwMjkxOTA1OA==,9599,simonw,2020-03-23T23:44:48Z,2020-03-23T23:44:48Z,MEMBER,Next step: use a `metadata.json` file to add some extras. And add the `datasette-render-markdown` plugin as soon as I ship https://github.com/simonw/datasette-render-markdown/issues/2 (GFM support).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance,
https://github.com/dogsheep/github-to-sqlite/issues/13#issuecomment-602924714,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13,602924714,MDEyOklzc3VlQ29tbWVudDYwMjkyNDcxNA==,9599,simonw,2020-03-24T00:03:25Z,2020-03-24T00:03:25Z,MEMBER,This is good enough for the 1.0 release.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521275281,Set up a live demo Datasette instance,
https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559883311,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14,559883311,MDEyOklzc3VlQ29tbWVudDU1OTg4MzMxMQ==,9599,simonw,2019-11-29T21:30:37Z,2019-11-29T21:30:37Z,MEMBER,"I should build the command to persist ETags and obey their polling guidelines:

> Events are optimized for polling with the ""ETag"" header. If no new events have been triggered, you will see a ""304 Not Modified"" response, and your current rate limit will be untouched. There is also an ""X-Poll-Interval"" header that specifies how often (in seconds) you are allowed to poll. In times of high server load, the time may increase. Please obey the header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",530491074,Command for importing events,
https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559902818,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14,559902818,MDEyOklzc3VlQ29tbWVudDU1OTkwMjgxOA==,9599,simonw,2019-11-30T01:32:38Z,2019-11-30T01:32:38Z,MEMBER,"Prototype:
```
pip install sqlite-utils paginate-json
paginate-json ""https://api.github.com/users/simonw/events"" | sqlite-utils insert /tmp/events.db events - --pk=id
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",530491074,Command for importing events,
https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-613641947,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14,613641947,MDEyOklzc3VlQ29tbWVudDYxMzY0MTk0Nw==,9599,simonw,2020-04-14T19:38:24Z,2020-04-14T19:38:34Z,MEMBER,"Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",530491074,Command for importing events,
https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602309825,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15,602309825,MDEyOklzc3VlQ29tbWVudDYwMjMwOTgyNQ==,9599,simonw,2020-03-23T00:52:38Z,2020-03-23T00:52:38Z,MEMBER,Splitting assets out into a separate table totally makes sense to me. They can still be fetched as part of the `releases` command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",544571092,Assets table with downloads,
https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602311701,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15,602311701,MDEyOklzc3VlQ29tbWVudDYwMjMxMTcwMQ==,9599,simonw,2020-03-23T01:05:15Z,2020-03-23T01:06:14Z,MEMBER,"None of my own releases use assets (they are all pushed to PyPI instead) but I spotted that your project here uses assets, so I'll test against that: https://github.com/instrumenta/conftest/releases/tag/v0.18.0

```
github-to-sqlite releases releases.db instrumenta/conftest
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",544571092,Assets table with downloads,
https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-602312054,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15,602312054,MDEyOklzc3VlQ29tbWVudDYwMjMxMjA1NA==,9599,simonw,2020-03-23T01:07:10Z,2020-03-23T01:07:10Z,MEMBER,"Each asset looks like this:
```json
    {
        ""url"": ""https://api.github.com/repos/instrumenta/conftest/releases/assets/11811946"",
        ""id"": 11811946,
        ""node_id"": ""MDEyOlJlbGVhc2VBc3NldDExODExOTQ2"",
        ""name"": ""checksums.txt"",
        ""label"": """",
        ""uploader"": {
            ""login"": ""garethr"",
            ""id"": 2029,
            ""node_id"": ""MDQ6VXNlcjIwMjk="",
            ""avatar_url"": ""https://avatars2.githubusercontent.com/u/2029?v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/garethr"",
            ""html_url"": ""https://github.com/garethr"",
            ""followers_url"": ""https://api.github.com/users/garethr/followers"",
            ""following_url"": ""https://api.github.com/users/garethr/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/garethr/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/garethr/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/garethr/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/garethr/orgs"",
            ""repos_url"": ""https://api.github.com/users/garethr/repos"",
            ""events_url"": ""https://api.github.com/users/garethr/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/garethr/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""content_type"": ""text/plain; charset=utf-8"",
        ""state"": ""uploaded"",
        ""size"": 600,
        ""download_count"": 2,
        ""created_at"": ""2019-03-30T16:56:44Z"",
        ""updated_at"": ""2019-03-30T16:56:44Z"",
        ""browser_download_url"": ""https://github.com/instrumenta/conftest/releases/download/v0.1.0/checksums.txt""
    }
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",544571092,Assets table with downloads,
https://github.com/dogsheep/github-to-sqlite/issues/15#issuecomment-605439685,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15,605439685,MDEyOklzc3VlQ29tbWVudDYwNTQzOTY4NQ==,2029,garethr,2020-03-28T12:17:01Z,2020-03-28T12:17:01Z,NONE,"That looks great, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",544571092,Assets table with downloads,
https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-571412923,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16,571412923,MDEyOklzc3VlQ29tbWVudDU3MTQxMjkyMw==,15092,jayvdb,2020-01-07T03:06:46Z,2020-01-07T03:06:46Z,NONE,"I re-tried after doing `auth`, and I get the same result.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546051181,Exception running first command: IndexError: list index out of range,
https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-601979709,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16,601979709,MDEyOklzc3VlQ29tbWVudDYwMTk3OTcwOQ==,9599,simonw,2020-03-21T02:20:08Z,2020-03-21T02:20:08Z,MEMBER,I'm pretty sure this means I need to upgrade the underlying `sqlite-utils` dependency. I saw the same bug in `twitter-to-sqlite`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546051181,Exception running first command: IndexError: list index out of range,
https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-602136481,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16,602136481,MDEyOklzc3VlQ29tbWVudDYwMjEzNjQ4MQ==,15092,jayvdb,2020-03-22T02:08:57Z,2020-03-22T02:08:57Z,NONE,"I'd love to be using your library as a better cached gh layer for a new library I have built, replacing large parts of the very ugly https://github.com/jayvdb/pypidb/blob/master/pypidb/_github.py , and then probably being able to rebuild the setuppy chunk as a feature here at a later stage.

I would also need tokenless and netrc support, but I would be happy to add those bits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546051181,Exception running first command: IndexError: list index out of range,
https://github.com/dogsheep/github-to-sqlite/issues/16#issuecomment-613611455,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16,613611455,MDEyOklzc3VlQ29tbWVudDYxMzYxMTQ1NQ==,9599,simonw,2020-04-14T18:37:21Z,2020-04-14T18:37:21Z,MEMBER,"This should have been fixed by #20 and #23 

@jayvdb I'm definitely interested in this tool working as a library - it's purely designed as a CLI tool at the moment, but cleaning it up to work better as a dependency is totally in-scope for the project. https://sqlite-utils.readthedocs.io/ is an example of a tool I've built that works for both.

Feel free to open a new issue here with some notes on what you would need for this to work as a library for your project!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546051181,Exception running first command: IndexError: list index out of range,
https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597354514,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17,597354514,MDEyOklzc3VlQ29tbWVudDU5NzM1NDUxNA==,9599,simonw,2020-03-10T22:37:45Z,2020-03-10T22:37:45Z,MEMBER,I should add an option to stop the moment you see a commit you have fetched before.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",578883725,Command for importing commits,
https://github.com/dogsheep/github-to-sqlite/issues/17#issuecomment-597358364,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17,597358364,MDEyOklzc3VlQ29tbWVudDU5NzM1ODM2NA==,9599,simonw,2020-03-10T22:50:20Z,2020-03-11T01:18:36Z,MEMBER,By default it will stop when it sees a commit that has already been stored. You will be able to over-ride that behaviour using `--all`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",578883725,Command for importing commits,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980724,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,601980724,MDEyOklzc3VlQ29tbWVudDYwMTk4MDcyNA==,9599,simonw,2020-03-21T02:28:05Z,2020-03-21T02:28:05Z,MEMBER,"This suggests that `commit[""author""]` can be `None` in some cases?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-601980957,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,601980957,MDEyOklzc3VlQ29tbWVudDYwMTk4MDk1Nw==,9599,simonw,2020-03-21T02:30:03Z,2020-03-21T02:30:03Z,MEMBER,"https://github.community/t5/GitHub-API-Development-and/Request-for-commits-quot-author-quot-null-and-quot-committer/m-p/35842/highlight/true#M3372
> Commits aren't always associated with a GitHub user. For example, perhaps a friend of mine and I were working on a project together. I have a GitHub account and my friend doesn't. If we both add commits to the repository using our own email addresses and names and then I push the repository to GitHub, my commits will be associated with my GitHub user account but my friends' commits will show up with `author` and `committer` as `null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602807178,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602807178,MDEyOklzc3VlQ29tbWVudDYwMjgwNzE3OA==,9599,simonw,2020-03-23T19:24:43Z,2020-03-23T19:24:43Z,MEMBER,I need to find an example before I work on this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811468,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602811468,MDEyOklzc3VlQ29tbWVudDYwMjgxMTQ2OA==,9599,simonw,2020-03-23T19:33:14Z,2020-03-23T19:33:14Z,MEMBER,Found one: https://api.github.com/repos/simonw/simonw.github.com/commits,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602811785,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602811785,MDEyOklzc3VlQ29tbWVudDYwMjgxMTc4NQ==,9599,simonw,2020-03-23T19:33:56Z,2020-03-23T19:34:06Z,MEMBER,"```json
[
  {
    ""sha"": ""a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0"",
    ""node_id"": ""MDY6Q29tbWl0OTMyMDk6YThkYzkxNDA4OWQzOTlkOWI1MjJlYmI1MWI2N2Y5YWMyZThhYTZiMA=="",
    ""commit"": {
      ""author"": {
        ""name"": ""Simon Willison"",
        ""email"": ""simon@..."",
        ""date"": ""2008-12-18T23:17:12Z""
      },
      ""committer"": {
        ""name"": ""Simon Willison"",
        ""email"": ""simon@..."",
        ""date"": ""2008-12-18T23:17:12Z""
      },
      ""message"": ""First commit"",
      ""tree"": {
        ""sha"": ""ac2dfb75e2592c59165c2880f3f7a16dafd452a1"",
        ""url"": ""https://api.github.com/repos/simonw/simonw.github.com/git/trees/ac2dfb75e2592c59165c2880f3f7a16dafd452a1""
      },
      ""url"": ""https://api.github.com/repos/simonw/simonw.github.com/git/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0"",
      ""comment_count"": 0,
      ""verification"": {
        ""verified"": false,
        ""reason"": ""unsigned"",
        ""signature"": null,
        ""payload"": null
      }
    },
    ""url"": ""https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0"",
    ""html_url"": ""https://github.com/simonw/simonw.github.com/commit/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0"",
    ""comments_url"": ""https://api.github.com/repos/simonw/simonw.github.com/commits/a8dc914089d399d9b522ebb51b67f9ac2e8aa6b0/comments"",
    ""author"": null,
    ""committer"": null,
    ""parents"": []
  }
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602813731,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602813731,MDEyOklzc3VlQ29tbWVudDYwMjgxMzczMQ==,9599,simonw,2020-03-23T19:38:03Z,2020-03-23T20:44:49Z,MEMBER,"So it turns out `""author""` and `""committer""` on the commit are `null` if the email address in the nested `""commit""` doesn't match an existing GitHub user.

Maybe I should be storing the nested data somewhere as well?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602815120,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602815120,MDEyOklzc3VlQ29tbWVudDYwMjgxNTEyMA==,9599,simonw,2020-03-23T19:40:55Z,2020-03-23T19:43:19Z,MEMBER,I could pull a pk-hashed version of the name/email into separate `raw_author` and `raw_committer` columns perhaps - against a `commit_authors` table. Could be interesting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author,
https://github.com/dogsheep/github-to-sqlite/issues/18#issuecomment-602846293,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18,602846293,MDEyOklzc3VlQ29tbWVudDYwMjg0NjI5Mw==,9599,simonw,2020-03-23T20:44:40Z,2020-03-23T20:44:40Z,MEMBER,I implemented the `raw_authors` idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585411547,Commits in GitHub API can have null author,
https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602304531,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19,602304531,MDEyOklzc3VlQ29tbWVudDYwMjMwNDUzMQ==,9599,simonw,2020-03-23T00:20:28Z,2020-03-23T00:20:28Z,MEMBER,"Current FTS code:
https://github.com/dogsheep/github-to-sqlite/blob/e35eec4343aa560c58c1634cc228d0d46c442304/github_to_sqlite/utils.py#L136-L143","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585850715,"Enable full-text search for more stuff (like commits, issues and issue_comments)",
https://github.com/dogsheep/github-to-sqlite/issues/19#issuecomment-602782437,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19,602782437,MDEyOklzc3VlQ29tbWVudDYwMjc4MjQzNw==,9599,simonw,2020-03-23T18:36:12Z,2020-03-23T18:51:23Z,MEMBER,"* `commits`: `message`
* `issue_comments`: `body`
* `issues`: `title`, `body`
* `labels`: `name`, `description`
* `licenses`: `name`
* `milestones`: `title`, `description`
* `releases`: `name`, `body`
* `repos`: `name`, `description`
* `users`: `login`, `name` (maybe `company`, `bio`)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585850715,"Enable full-text search for more stuff (like commits, issues and issue_comments)",
https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602896434,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21,602896434,MDEyOklzc3VlQ29tbWVudDYwMjg5NjQzNA==,9599,simonw,2020-03-23T22:43:37Z,2020-03-23T22:43:37Z,MEMBER,"I'm going to do this now to help figure out the latest error in #13:
```
Traceback (most recent call last):
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/bin/github-to-sqlite"", line 11, in <module>
    load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')()
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 237, in commits
    utils.save_commits(db, commits, repo_full[""id""])
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 345, in save_commits
    for commit in commits:
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 207, in fetch_commits
    if stop_when(commit):
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 224, in stop_when
    db[""commits""].get(commit[""sha""])
TypeError: string indices must be integers
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",586561727,Turn GitHub API errors into exceptions,
https://github.com/dogsheep/github-to-sqlite/issues/21#issuecomment-602920163,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21,602920163,MDEyOklzc3VlQ29tbWVudDYwMjkyMDE2Mw==,9599,simonw,2020-03-23T23:48:22Z,2020-03-23T23:48:22Z,MEMBER,"I'm happy with this pattern:
https://github.com/dogsheep/github-to-sqlite/blob/f78c4e9baaf0970ffab266ba780df7240aae9f32/github_to_sqlite/utils.py#L4-L18","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",586561727,Turn GitHub API errors into exceptions,
https://github.com/dogsheep/github-to-sqlite/issues/23#issuecomment-602928533,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23,602928533,MDEyOklzc3VlQ29tbWVudDYwMjkyODUzMw==,9599,simonw,2020-03-24T00:15:49Z,2020-03-24T00:15:49Z,MEMBER,https://github.com/dogsheep/github-to-sqlite/releases/tag/1.0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",586595839,Release 1.0,
https://github.com/dogsheep/github-to-sqlite/issues/25#issuecomment-614810417,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25,614810417,MDEyOklzc3VlQ29tbWVudDYxNDgxMDQxNw==,9599,simonw,2020-04-16T18:07:11Z,2020-04-16T18:07:11Z,MEMBER,Turns out the main problem was #26 - now fixed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601265023,Improvements to demo instance,
https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614794739,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26,614794739,MDEyOklzc3VlQ29tbWVudDYxNDc5NDczOQ==,9599,simonw,2020-04-16T17:38:28Z,2020-04-16T17:38:28Z,MEMBER,I'm already doing this here: https://github.com/dogsheep/github-to-sqlite/blob/c4aaa50e167cfa9021c7c94260bc3e89e10947bf/github_to_sqlite/utils.py#L246-L250,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601271612,Topics are missing from repositories,
https://github.com/dogsheep/github-to-sqlite/issues/26#issuecomment-614795712,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26,614795712,MDEyOklzc3VlQ29tbWVudDYxNDc5NTcxMg==,9599,simonw,2020-04-16T17:40:27Z,2020-04-16T17:40:27Z,MEMBER,Aha! it was missing from the `fetch_repo()` function.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601271612,Topics are missing from repositories,
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831451,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,614831451,MDEyOklzc3VlQ29tbWVudDYxNDgzMTQ1MQ==,9599,simonw,2020-04-16T18:47:25Z,2020-04-16T18:47:25Z,MEMBER,Is it possible for a repo to have an `owner` that differs from its `organization`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,Repos have a big blob of JSON in the organization column,
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614831842,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,614831842,MDEyOklzc3VlQ29tbWVudDYxNDgzMTg0Mg==,9599,simonw,2020-04-16T18:48:18Z,2020-04-16T18:48:18Z,MEMBER,I'm going to make `organization` another foreign key to the `users` table just in case it IS possible (maybe with GitHub Enterprise or similar?),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,Repos have a big blob of JSON in the organization column,
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,614843406,MDEyOklzc3VlQ29tbWVudDYxNDg0MzQwNg==,9599,simonw,2020-04-16T19:11:53Z,2020-04-16T19:20:23Z,MEMBER,"This didn't quite work: the column type is incorrect, so the foreign key relationship isn't sticking:

https://github-to-sqlite.dogsheep.net/github/repos?organization=53015001

`[organization] TEXT REFERENCES [users]([id])` - should be `INTEGER`.

The problem is that if the first repo inserted has no organization it's set to `null`, which `sqlite-utils` derives as a `TEXT` column.

One solution would be to create the column explicitly with a type, but this could get messy.

I think I want a new sqlite-utils feature for this instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,Repos have a big blob of JSON in the organization column,
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615518606,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,615518606,MDEyOklzc3VlQ29tbWVudDYxNTUxODYwNg==,9599,simonw,2020-04-18T00:14:32Z,2020-04-18T00:14:32Z,MEMBER,https://github.com/simonw/sqlite-utils/issues/100 is done and released in sqlite-utils 2.7.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,Repos have a big blob of JSON in the organization column,
https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-615519409,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27,615519409,MDEyOklzc3VlQ29tbWVudDYxNTUxOTQwOQ==,9599,simonw,2020-04-18T00:19:16Z,2020-04-18T00:19:16Z,MEMBER,"```
$ github-to-sqlite repos b.db dogsheep
$ sqlite3 b.db '.schema repos'
CREATE TABLE [repos] (
   [id] INTEGER PRIMARY KEY,
...
   [permissions] TEXT,
   [organization] INTEGER REFERENCES [users]([id]),
   FOREIGN KEY(license) REFERENCES licenses(key)
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601330277,Repos have a big blob of JSON in the organization column,
https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883040,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28,615883040,MDEyOklzc3VlQ29tbWVudDYxNTg4MzA0MA==,9599,simonw,2020-04-18T14:45:38Z,2020-04-18T14:45:38Z,MEMBER,"```
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 219, in contributors
    utils.save_contributors(db, contributors, repo_full[""id""])
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 354, in save_contributors
    for contributor in contributors:
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 228, in fetch_contributors
    for contributors in paginate(url, headers):
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 286, in paginate
    data = response.json()
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/requests/models.py"", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/__init__.py"", line 357, in loads
    return _default_decoder.decode(s)
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py"", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File ""/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/json/decoder.py"", line 355, in raw_decode
    raise JSONDecodeError(""Expecting value"", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601333634,Pull repository contributors,
https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615883687,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28,615883687,MDEyOklzc3VlQ29tbWVudDYxNTg4MzY4Nw==,9599,simonw,2020-04-18T14:49:58Z,2020-04-18T14:49:58Z,MEMBER,"That happened trying to pull contributors for `dogsheep/beta` - an empty repository. Turns out it was returning a `204 no content`:
```
~ $ curl -i 'https://api.github.com/repos/dogsheep/beta/contributors'
HTTP/1.1 204 No Content
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601333634,Pull repository contributors,
https://github.com/dogsheep/github-to-sqlite/issues/28#issuecomment-615886206,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28,615886206,MDEyOklzc3VlQ29tbWVudDYxNTg4NjIwNg==,9599,simonw,2020-04-18T15:04:59Z,2020-04-18T15:04:59Z,MEMBER,"Demo: https://github-to-sqlite.dogsheep.net/github/contributors

Documentation: https://github.com/dogsheep/github-to-sqlite/blob/13f8868fb5efa01c263b24f6dd91c617e6e938e1/README.md#fetching-contributors-to-a-repository ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",601333634,Pull repository contributors,
https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29,616883275,MDEyOklzc3VlQ29tbWVudDYxNjg4MzI3NQ==,9599,simonw,2020-04-21T00:43:28Z,2020-04-21T00:43:28Z,MEMBER,"I'm copying repo from issue, which surprisingly is a string, not an integer ID.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603617013,Milestones should have foreign key to creator and repo,
https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531516956,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3,531516956,MDEyOklzc3VlQ29tbWVudDUzMTUxNjk1Ng==,9599,simonw,2019-09-14T21:56:31Z,2019-09-14T21:56:31Z,MEMBER,"https://api.github.com/users/simonw/repos

It would be useful to be able to fetch stargazers, forks etc as well. Not sure if that should be a separate command or a `--stargazers` option to this command.

Probably a separate command since `issues` is a separate command already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670426,Command to fetch all repos belonging to a user or organization,
https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531517083,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3,531517083,MDEyOklzc3VlQ29tbWVudDUzMTUxNzA4Mw==,9599,simonw,2019-09-14T21:58:42Z,2019-09-14T21:58:42Z,MEMBER,Split stargazers into #4,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670426,Command to fetch all repos belonging to a user or organization,
https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616879753,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30,616879753,MDEyOklzc3VlQ29tbWVudDYxNjg3OTc1Mw==,9599,simonw,2020-04-21T00:29:29Z,2020-04-21T00:29:29Z,MEMBER,`assignee` looks like it's the incorrect type too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603618244,Issues milestone column is the wrong type,
https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616883726,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30,616883726,MDEyOklzc3VlQ29tbWVudDYxNjg4MzcyNg==,9599,simonw,2020-04-21T00:45:23Z,2020-04-21T00:45:23Z,MEMBER,"Demo of fix: https://github-to-sqlite.dogsheep.net/github/issues?assignee__notblank=1&milestone__notblank=1

![github__issues__4_rows_where_where_assignee_is_not_blank_and_milestone_is_not_blank_sorted_by_updated_at_descending](https://user-images.githubusercontent.com/9599/79812758-b49ace80-832e-11ea-81db-bdf993b872cc.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603618244,Issues milestone column is the wrong type,
https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-616884647,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,616884647,MDEyOklzc3VlQ29tbWVudDYxNjg4NDY0Nw==,9599,simonw,2020-04-21T00:49:16Z,2020-04-21T00:50:20Z,MEMBER,"The API just gives us the `repository_url`: https://api.github.com/repos/simonw/datasette/issues

![Mozilla_Firefox_and_Topic__Week_2__Discussion__Submit_your_six_story_points_here](https://user-images.githubusercontent.com/9599/79812950-283cdb80-832f-11ea-8759-9633087d1e7e.png)

We currently turn that into a `simonw/datasette` string here:
https://github.com/dogsheep/github-to-sqlite/blob/e0e8d8caa9657b04bfb8a2cf16c9b580f38b1805/github_to_sqlite/utils.py#L43-L46","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862,Issue and milestone should have foreign key to repo,
https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617348174,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,617348174,MDEyOklzc3VlQ29tbWVudDYxNzM0ODE3NA==,9599,simonw,2020-04-21T18:50:29Z,2020-04-21T18:50:29Z,MEMBER,"Since this represents a breaking schema change for anyone running SQL queries against these tables, I'm going to do a major version bump to 2.0 when I release this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862,Issue and milestone should have foreign key to repo,
https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617491607,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,617491607,MDEyOklzc3VlQ29tbWVudDYxNzQ5MTYwNw==,9599,simonw,2020-04-22T01:20:19Z,2020-04-22T01:20:19Z,MEMBER,"https://github-to-sqlite.dogsheep.net/github/milestones now link to repo:

<img width=""1095"" alt=""Screen Shot 2020-04-21 at 6 19 03 PM"" src=""https://user-images.githubusercontent.com/9599/79930145-956a7280-83fc-11ea-9b47-b0c5674a0d4f.png"">

And so do issues: https://github-to-sqlite.dogsheep.net/github/issues

<img width=""1095"" alt=""Screen Shot 2020-04-21 at 6 19 53 PM"" src=""https://user-images.githubusercontent.com/9599/79930200-bd59d600-83fc-11ea-8fb8-b40772140409.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862,Issue and milestone should have foreign key to repo,
https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617364956,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617364956,MDEyOklzc3VlQ29tbWVudDYxNzM2NDk1Ng==,9599,simonw,2020-04-21T19:24:45Z,2020-04-21T19:24:45Z,MEMBER,"That's because I just broke this code:

https://github.com/dogsheep/github-to-sqlite/blob/2cf75a0a036719eb7e57fdc7c5c2ea0f4c26978a/github_to_sqlite/utils.py#L131-L139

It expects the `repo` column to be `simonw/datasette` but it's now an ID instead.

I should add a test for this as part of the fix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295,Issue comments don't appear to populate issues foreign key,
https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617369247,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617369247,MDEyOklzc3VlQ29tbWVudDYxNzM2OTI0Nw==,9599,simonw,2020-04-21T19:33:03Z,2020-04-21T19:33:03Z,MEMBER,Caused by #31.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295,Issue comments don't appear to populate issues foreign key,
https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617490914,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617490914,MDEyOklzc3VlQ29tbWVudDYxNzQ5MDkxNA==,9599,simonw,2020-04-22T01:17:44Z,2020-04-22T01:17:44Z,MEMBER,"https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101 now shows issues.

And https://github-to-sqlite.dogsheep.net/github/issue_comments links to them:

<img width=""395"" alt=""Screen Shot 2020-04-21 at 6 17 33 PM"" src=""https://user-images.githubusercontent.com/9599/79930066-6227e380-83fc-11ea-8416-8b9d55660cb5.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295,Issue comments don't appear to populate issues foreign key,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622169728,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622169728,MDEyOklzc3VlQ29tbWVudDYyMjE2OTcyOA==,9599,simonw,2020-04-30T23:18:51Z,2020-04-30T23:18:51Z,MEMBER,"Sure, that sounds fine to me.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,Fall back to authentication via ENV,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622171097,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622171097,MDEyOklzc3VlQ29tbWVudDYyMjE3MTA5Nw==,9599,simonw,2020-04-30T23:22:45Z,2020-04-30T23:23:57Z,MEMBER,"The `auth.json` mechanism this uses is standard across all of the other Dogsheep tools - it's actually designed so you can have one `auth.json` with a bunch of different credentials for different tools:

```json
{
    ""goodreads_personal_token"": ""..."",
    ""goodreads_user_id"": ""..."",
    ""github_personal_token"": ""..."",
    ""pocket_consumer_key"": ""..."",
    ""pocket_username"": ""..."",
    ""pocket_access_token"": ""...""
}
```
But... `github-to-sqlite` does feel like it deserves a special case here, since it's such a good fit for running inside of GitHub Actions - which even provide a `GITHUB_TOKEN` for you to use!

So I don't think it will harm the family of tools too much if this has an environment variable alternative to the `-a` file.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,Fall back to authentication via ENV,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622279374,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622279374,MDEyOklzc3VlQ29tbWVudDYyMjI3OTM3NA==,2029,garethr,2020-05-01T07:12:47Z,2020-05-01T07:12:47Z,NONE,"I also go it working with:

```yaml
run: echo ${{ secrets.github_token }} | github-to-sqlite auth
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,Fall back to authentication via ENV,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622997410,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622997410,MDEyOklzc3VlQ29tbWVudDYyMjk5NzQxMA==,9599,simonw,2020-05-02T18:46:10Z,2020-05-02T18:46:10Z,MEMBER,Documented here: https://github.com/dogsheep/github-to-sqlite/blob/10fb34de41aaa35681f08b5991540d65bfcf2e2e/README.md#authentication,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,Fall back to authentication via ENV,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133298,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622133298,MDEyOklzc3VlQ29tbWVudDYyMjEzMzI5OA==,9599,simonw,2020-04-30T21:48:24Z,2020-04-30T21:48:24Z,MEMBER,"Unfortunately it's not available through any GitHub API - I managed to figure out how to get dependencies, but I need dependents. https://github.com/simonw/til/blob/master/github/dependencies-graphql-api.md","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,Command for retrieving dependents for a repo,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133422,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622133422,MDEyOklzc3VlQ29tbWVudDYyMjEzMzQyMg==,9599,simonw,2020-04-30T21:48:39Z,2020-04-30T21:48:39Z,MEMBER,It looks like the only option is to scrape them. I'll do that and then replace with an API as soon as one becomes available.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,Command for retrieving dependents for a repo,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133775,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622133775,MDEyOklzc3VlQ29tbWVudDYyMjEzMzc3NQ==,9599,simonw,2020-04-30T21:49:27Z,2020-04-30T21:49:27Z,MEMBER,"Proposed command:

    github-to-sqlite scrape-dependents github.db simonw/datasette

I'll pull full details of the scraped repos from the regular API. I'll also record when they were ""first seen"" by the command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,Command for retrieving dependents for a repo,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622135654,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622135654,MDEyOklzc3VlQ29tbWVudDYyMjEzNTY1NA==,9599,simonw,2020-04-30T21:53:44Z,2020-04-30T21:56:06Z,MEMBER,"I think this is the neatest scraping pattern:
```python
[a[""href""].lstrip(""/"") for a in soup.select(""a[data-hovercard-type=repository]"")]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,Command for retrieving dependents for a repo,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622136585,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622136585,MDEyOklzc3VlQ29tbWVudDYyMjEzNjU4NQ==,9599,simonw,2020-04-30T21:55:51Z,2020-04-30T21:55:51Z,MEMBER,"And to find the ""Next"" pagination link:
```python
soup.select("".paginate-container"")[0].find(""a"", text=""Next"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,Command for retrieving dependents for a repo,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622162835,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622162835,MDEyOklzc3VlQ29tbWVudDYyMjE2MjgzNQ==,9599,simonw,2020-04-30T22:59:26Z,2020-04-30T22:59:26Z,MEMBER,Documentation: https://github.com/dogsheep/github-to-sqlite/blob/c9f48404481882e8b3af06f35e4801a80ac79ed6/README.md#scraping-dependents-for-a-repository,"{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 2, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,Command for retrieving dependents for a repo,
https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622213950,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622213950,MDEyOklzc3VlQ29tbWVudDYyMjIxMzk1MA==,9599,simonw,2020-05-01T02:09:04Z,2020-05-01T02:09:04Z,MEMBER,"It sped up this query a lot - 2.5s down to 300ms:
```sql
select
  repos.full_name,
  json_object(
    'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number,
    'label', '#' || issues.number
  ) as issue,
  issues.title,
  users.login,
  users.id,
  issues.state,
  issues.locked,
  issues.assignee,
  issues.milestone,
  issues.comments,
  issues.created_at,
  issues.updated_at,
  issues.closed_at,
  issues.author_association,
  issues.pull_request,
  issues.repo,
  issues.type
from
  issues
  join repos on repos.id = issues.repo
  join users on issues.user = users.id
where
  issues.state = 'open'
  and issues.user not in (9599, 27856297)
  and not exists (
    select
      id
    from
      issue_comments
    where
      issue_comments.user = 9599
      and issues.id = issue_comments.issue
  )
order by
  issues.updated_at desc;
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450,Create index on issue_comments(user) and other foreign keys,
https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622214262,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622214262,MDEyOklzc3VlQ29tbWVudDYyMjIxNDI2Mg==,9599,simonw,2020-05-01T02:10:32Z,2020-05-01T02:11:19Z,MEMBER,"This sped that query up even more - down to 4ms.
```sql
create index issue_comments_issue on issue_comments(issue);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450,Create index on issue_comments(user) and other foreign keys,
https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622982667,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622982667,MDEyOklzc3VlQ29tbWVudDYyMjk4MjY2Nw==,9599,simonw,2020-05-02T16:52:53Z,2020-05-02T16:52:53Z,MEMBER,Easiest option: use `db.index_foreign_keys()`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#adding-indexes-for-all-foreign-keys,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450,Create index on issue_comments(user) and other foreign keys,
https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622461025,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36,622461025,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTAyNQ==,9599,simonw,2020-05-01T16:34:24Z,2020-05-01T16:34:24Z,MEMBER,Blocked on #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610842926,Add view for better display of dependent repos,
https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622982346,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36,622982346,MDEyOklzc3VlQ29tbWVudDYyMjk4MjM0Ng==,9599,simonw,2020-05-02T16:50:31Z,2020-05-02T16:50:31Z,MEMBER,Demo: https://github-to-sqlite.dogsheep.net/github/dependent_repos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610842926,Add view for better display of dependent repos,
https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461537,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622461537,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTUzNw==,9599,simonw,2020-05-01T16:35:40Z,2020-05-01T16:35:40Z,MEMBER,"This will check if the view exists and has the exact same matching definition as the one we want. If it doesn't, we will drop it (if it exists) and recreate it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136,Mechanism for creating views if they don't yet exist,
https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461948,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622461948,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTk0OA==,9599,simonw,2020-05-01T16:36:42Z,2020-05-01T16:36:42Z,MEMBER,It should only create views if the underlying tables exist.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136,Mechanism for creating views if they don't yet exist,
https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622978173,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622978173,MDEyOklzc3VlQ29tbWVudDYyMjk3ODE3Mw==,9599,simonw,2020-05-02T16:19:31Z,2020-05-02T16:19:47Z,MEMBER,"I can use the new `.create_view(..., replace=True)` parameter in `sqlite-utils` 2.7.2 for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136,Mechanism for creating views if they don't yet exist,
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623027889,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623027889,MDEyOklzc3VlQ29tbWVudDYyMzAyNzg4OQ==,9599,simonw,2020-05-02T23:15:11Z,2020-05-02T23:15:11Z,MEMBER,"This is one of the use-cases for the `repos_starred` view: it allows you to easily run this kid of query without having to construct the SQL by hand. Here's a demo:

https://github-to-sqlite.dogsheep.net/github/repos_starred?name__contains=twitter

My philosophy here is to keep the raw tables (like `stars`) as normalized as possible, then use SQL views which expose the data in a form that's easier to query.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481,[Feature Request] Support Repo Name in Search 🥺,
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038148,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623038148,MDEyOklzc3VlQ29tbWVudDYyMzAzODE0OA==,5779832,zzeleznick,2020-05-03T01:18:57Z,2020-05-03T01:18:57Z,NONE,"Thanks, @simonw! 

I feel a little foolish in hindsight, but I'm on the same page now and am glad to have discovered first-hand a motivation for this `repos_starred` use case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481,[Feature Request] Support Repo Name in Search 🥺,
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038378,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623038378,MDEyOklzc3VlQ29tbWVudDYyMzAzODM3OA==,9599,simonw,2020-05-03T01:21:13Z,2020-05-03T01:21:13Z,MEMBER,No this is really useful feedback! I'm so close to this project that I miss what's not obvious to people dropping in for the first time.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481,[Feature Request] Support Repo Name in Search 🥺,
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623044643,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623044643,MDEyOklzc3VlQ29tbWVudDYyMzA0NDY0Mw==,5779832,zzeleznick,2020-05-03T02:34:32Z,2020-05-03T02:34:32Z,NONE,"1. More than glad to share feedback from the sidelines as a [starrer](https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++starred_at%2C%0D%0A++starred_by%2C%0D%0A++full_name+as+repo_name%0D%0Afrom%0D%0A++repos_starred%0D%0Awhere%0D%0A++starred_by+%3D+%22zzeleznick%22%0D%0Aorder+by%0D%0A++starred_at+desc). 

```
-- Motivation:
--  Datasette is a data hammer and I'm looking for nails
--  e.g. Find which repos a user has starred => trigger a TBD downstream action
select
  starred_at,
  starred_by,
  full_name as repo_name
from
  repos_starred
where
  starred_by = ""zzeleznick""
order by
  starred_at desc
``` 

| starred_at | starred_by | repo_name |
| --- | --- | --- |
| 2020-02-11T01:08:59Z | zzeleznick | dogsheep/twitter-to-sqlite |
| 2020-01-11T21:57:34Z | zzeleznick | simonw/datasette |

2. In my day job, I use [airflow](https://github.com/apache/airflow), and that's the mental model I'm bringing to [datasette](https://github.com/simonw/datasette). 

3. I see your project like [twitter-to-sqlite](https://github.com/dogsheep/twitter-to-sqlite) akin to [Operators](https://airflow.apache.org/docs/stable/_api/index.html#pythonapi-operators) in Airflow world.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481,[Feature Request] Support Repo Name in Search 🥺,
https://github.com/dogsheep/github-to-sqlite/issues/39#issuecomment-675509550,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/39,675509550,MDEyOklzc3VlQ29tbWVudDY3NTUwOTU1MA==,9599,simonw,2020-08-18T14:23:56Z,2020-08-18T14:23:56Z,MEMBER,I think this is fixed: https://github-to-sqlite.dogsheep.net/github/issues?_facet=repo,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613777056,issues foreign key to repo isn't working,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-531517138,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,531517138,MDEyOklzc3VlQ29tbWVudDUzMTUxNzEzOA==,9599,simonw,2019-09-14T21:59:59Z,2019-09-14T21:59:59Z,MEMBER,"Paginate through https://api.github.com/repos/simonw/datasette/stargazers

Send `Accept: application/vnd.github.v3.star+json` to get the `starred_at` dates.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-550388354,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,550388354,MDEyOklzc3VlQ29tbWVudDU1MDM4ODM1NA==,9599,simonw,2019-11-06T16:26:55Z,2019-11-06T16:26:55Z,MEMBER,"Here's a query I figured out using a window function that shows cumulative stargazers over time:
```sql
select
  yyyymmdd,
  sum(n) over (
    order by
      yyyymmdd rows unbounded preceding
  ) as cumulative_count
from
  (
    select
      substr(starred_at, 0, 11) as yyyymmdd,
      count(*) as n
    from
      stars
    group by
      yyyymmdd
  )
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-622990947,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,622990947,MDEyOklzc3VlQ29tbWVudDYyMjk5MDk0Nw==,9599,simonw,2020-05-02T17:54:16Z,2020-05-02T17:54:16Z,MEMBER,"I could add that window function query as a view, but only if the detected version of SQLite supports window functions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623000814,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623000814,MDEyOklzc3VlQ29tbWVudDYyMzAwMDgxNA==,9599,simonw,2020-05-02T19:15:23Z,2020-05-02T19:15:23Z,MEMBER,"I'm not going to do the `--sql` bit just yet. I have patterns for working around this for other commands which are working fine:

https://github.com/dogsheep/github-to-sqlite/blob/d00a53061556dc403c166b443d141c4e1adbd64a/.github/workflows/deploy-demo.yml#L53-L70","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623004836,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623004836,MDEyOklzc3VlQ29tbWVudDYyMzAwNDgzNg==,9599,simonw,2020-05-02T19:49:51Z,2020-05-02T19:49:51Z,MEMBER,"Alternative pattern:
```
sqlite-utils releases.db 'select full_name from repos' --csv --no-headers \
  | tr -d '\r' \
  | xargs github-to-sqlite stargazers stars.db
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623006154,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623006154,MDEyOklzc3VlQ29tbWVudDYyMzAwNjE1NA==,9599,simonw,2020-05-02T20:01:39Z,2020-05-02T20:01:54Z,MEMBER,Needs tests and documentation. I shipped it early to check that the live demo works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623007441,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623007441,MDEyOklzc3VlQ29tbWVudDYyMzAwNzQ0MQ==,9599,simonw,2020-05-02T20:13:37Z,2020-05-02T20:13:37Z,MEMBER,Datasette cumulative stars over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623010272,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623010272,MDEyOklzc3VlQ29tbWVudDYyMzAxMDI3Mg==,9599,simonw,2020-05-02T20:39:14Z,2020-05-02T20:39:14Z,MEMBER,"Graph of cumulative stars for Datasette over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29#g.mark=line&g.x_column=yyyymmdd&g.x_type=temporal&g.y_column=cumulative_count&g.y_type=quantitative

<img width=""291"" alt=""github__select_yyyymmdd__sum_n__over___order_by_yyyymmdd_rows_unbounded_preceding___as_cumulative_count_from___select_substr_starred_at__0__11__as_yyyymmdd__count____as_n_from_stars_where_repo___107914493_group_by_yyyymmdd__"" src=""https://user-images.githubusercontent.com/9599/80891642-31587180-8c7a-11ea-9169-60eb3cb6d3e0.png"">

Stars per day (as a label bar chart, so very wide):

https://github-to-sqlite.dogsheep.net/github?sql=%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++#g.mark=bar&g.x_column=yyyymmdd&g.x_type=ordinal&g.y_column=n&g.y_type=quantitative

<img width=""1424"" alt=""github__select_substr_starred_at__0__11__as_yyyymmdd__count____as_n_from_stars_where_repo___107914493_group_by_yyyymmdd"" src=""https://user-images.githubusercontent.com/9599/80891656-4e8d4000-8c7a-11ea-9ff2-02c9cf4cb5d5.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643393506,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40,643393506,MDEyOklzc3VlQ29tbWVudDY0MzM5MzUwNg==,9599,simonw,2020-06-12T17:21:14Z,2020-06-12T17:21:14Z,MEMBER,"I only install SQLite for this:

https://github.com/dogsheep/github-to-sqlite/blob/c0d54e0260468be38152293df5abd775c068495d/.github/workflows/deploy-demo.yml#L77-L78

I'm going to remove the need to install sqlite3 by making this possible with sqlite-utils: https://github.com/simonw/sqlite-utils/issues/115","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637899539,Demo deploy is broken,
https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643414646,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40,643414646,MDEyOklzc3VlQ29tbWVudDY0MzQxNDY0Ng==,9599,simonw,2020-06-12T18:06:48Z,2020-06-12T18:06:48Z,MEMBER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637899539,Demo deploy is broken,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653947916,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653947916,MDEyOklzc3VlQ29tbWVudDY1Mzk0NzkxNg==,9599,simonw,2020-07-05T22:40:47Z,2020-07-05T22:40:47Z,MEMBER,Might be that it's not got enough RAM. I'll try deploying to a larger instance.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653960989,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653960989,MDEyOklzc3VlQ29tbWVudDY1Mzk2MDk4OQ==,9599,simonw,2020-07-06T00:32:34Z,2020-07-06T00:32:34Z,MEMBER,Same error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962418,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653962418,MDEyOklzc3VlQ29tbWVudDY1Mzk2MjQxOA==,9599,simonw,2020-07-06T00:41:38Z,2020-07-06T00:41:38Z,MEMBER,"https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/logs?project=datasette-222320 has some clues.

<img width=""818"" alt=""github-to-sqlite_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform"" src=""https://user-images.githubusercontent.com/9599/86545887-92086280-bee6-11ea-8787-b3b85a7bb4b9.png"">
<img width=""1451"" alt=""github-to-sqlite_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform"" src=""https://user-images.githubusercontent.com/9599/86545917-c11ed400-bee6-11ea-9cf8-957911ca9f6a.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962530,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653962530,MDEyOklzc3VlQ29tbWVudDY1Mzk2MjUzMA==,9599,simonw,2020-07-06T00:42:13Z,2020-07-06T00:42:13Z,MEMBER,So it looks like it's the ASGI lifespan change I made in https://github.com/simonw/datasette/commit/16f592247a2a0e140ada487e9972645406dcae69 - It must be incompatible with one of the plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962669,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653962669,MDEyOklzc3VlQ29tbWVudDY1Mzk2MjY2OQ==,9599,simonw,2020-07-06T00:42:57Z,2020-07-06T00:42:57Z,MEMBER,"https://github-to-sqlite.dogsheep.net/-/plugins

```json
[
    {
        ""name"": ""datasette-json-html"",
        ""static"": false,
        ""templates"": false,
        ""version"": ""0.6"",
        ""hooks"": [
            ""prepare_connection"",
            ""render_cell""
        ]
    },
    {
        ""name"": ""datasette-render-markdown"",
        ""static"": false,
        ""templates"": false,
        ""version"": ""1.1.2"",
        ""hooks"": [
            ""extra_template_vars"",
            ""render_cell""
        ]
    },
    {
        ""name"": ""datasette-pretty-json"",
        ""static"": false,
        ""templates"": false,
        ""version"": ""0.2"",
        ""hooks"": [
            ""render_cell""
        ]
    },
    {
        ""name"": ""datasette-search-all"",
        ""static"": false,
        ""templates"": true,
        ""version"": ""0.2.1"",
        ""hooks"": [
            ""asgi_wrapper"",
            ""extra_template_vars""
        ]
    },
    {
        ""name"": ""datasette-vega"",
        ""static"": true,
        ""templates"": false,
        ""version"": ""0.6.2"",
        ""hooks"": [
            ""extra_css_urls"",
            ""extra_js_urls""
        ]
    }
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962708,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653962708,MDEyOklzc3VlQ29tbWVudDY1Mzk2MjcwOA==,9599,simonw,2020-07-06T00:43:10Z,2020-07-06T00:43:10Z,MEMBER,I bet it's datasette-search-all.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653966670,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653966670,MDEyOklzc3VlQ29tbWVudDY1Mzk2NjY3MA==,9599,simonw,2020-07-06T01:07:02Z,2020-07-06T01:07:02Z,MEMBER,OK that fix worked.https://github.com/dogsheep/github-to-sqlite/runs/839764768?check_suite_focus=true,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660536265,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43,660536265,MDEyOklzc3VlQ29tbWVudDY2MDUzNjI2NQ==,9599,simonw,2020-07-18T20:15:12Z,2020-07-18T20:15:12Z,MEMBER,"I want to create a SQL query which shows me all of my repositories that have commits that are NOT in the most recent release.

The releases table doesn't have enough information for this because it doesn't tell you the commit hash associated with each release, just the tag: https://github-to-sqlite.dogsheep.net/github/releases","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660355904,github-to-sqlite tags command for fetching tags,
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660547502,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43,660547502,MDEyOklzc3VlQ29tbWVudDY2MDU0NzUwMg==,9599,simonw,2020-07-18T21:50:37Z,2020-07-18T21:50:37Z,MEMBER,"```
$ github-to-sqlite tags tags.db simonw/datasette dogsheep/github-to-sqlite
$ sqlite-utils tables tags.db --counts
[{""table"": ""users"", ""count"": 2},
 {""table"": ""licenses"", ""count"": 1},
 {""table"": ""repos"", ""count"": 2},
 {""table"": ""tags"", ""count"": 76},
 {""table"": ""licenses_fts"", ""count"": 1},
 {""table"": ""licenses_fts_data"", ""count"": 3},
 {""table"": ""licenses_fts_idx"", ""count"": 1},
 {""table"": ""licenses_fts_docsize"", ""count"": 1},
 {""table"": ""licenses_fts_config"", ""count"": 1},
 {""table"": ""repos_fts"", ""count"": 2},
 {""table"": ""repos_fts_data"", ""count"": 3},
 {""table"": ""repos_fts_idx"", ""count"": 1},
 {""table"": ""repos_fts_docsize"", ""count"": 2},
 {""table"": ""repos_fts_config"", ""count"": 1},
 {""table"": ""users_fts"", ""count"": 2},
 {""table"": ""users_fts_data"", ""count"": 3},
 {""table"": ""users_fts_idx"", ""count"": 1},
 {""table"": ""users_fts_docsize"", ""count"": 2},
 {""table"": ""users_fts_config"", ""count"": 1}]
$ sqlite-utils rows tags.db tags
[{""repo_id"": 107914493, ""name"": ""0.45"", ""sha"": ""f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb""},
 {""repo_id"": 107914493, ""name"": ""0.45a5"", ""sha"": ""676bb64c877d73f8ff496cef4632f5a8a5a9283c""},
 {""repo_id"": 107914493, ""name"": ""0.45a4"", ""sha"": ""265483173bc8341dc02c8b782b9b59d2ce8bbedb""},
 {""repo_id"": 107914493, ""name"": ""0.45a3"", ""sha"": ""1f55a4a2b68fa65e56a28baeb7f44122fdeca7e7""},
 {""repo_id"": 107914493, ""name"": ""0.45a2"", ""sha"": ""1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f""},
 {""repo_id"": 107914493, ""name"": ""0.45a1"", ""sha"": ""b59b92b1b0517cf18fa748ff9d0a0bf86298dd43""},
 {""repo_id"": 107914493, ""name"": ""0.45a0"", ""sha"": ""dda932d818b34ccab11730a76554f0a3748d8348""},
 {""repo_id"": 107914493, ""name"": ""0.44"", ""sha"": ""b906030235efbdff536405d66078f4868ce0d3bd""},
 {""repo_id"": 107914493, ""name"": ""0.43"", ""sha"": ""d56f402822df102f9cf1a9a056449d01a15e3aae""},
 {""repo_id"": 107914493, ""name"": ""0.42"", ""sha"": ""af6c6c5d6f929f951c0e63bfd1c82e37a071b50f""},
 {""repo_id"": 107914493, ""name"": ""0.41"", ""sha"": ""182e5c8745c94576718315f7596ccc81e5e2417b""},
 {""repo_id"": 107914493, ""name"": ""0.40"", ""sha"": ""8da108193b08abf140716f8ac499f32309dfe9cf""},
 {""repo_id"": 107914493, ""name"": ""0.39"", ""sha"": ""dedd775512daee49925882654f252df61a9e3b6d""},
 {""repo_id"": 107914493, ""name"": ""0.38"", ""sha"": ""7e357abbc38dcc9d19a2f1df3252668a48e941e4""},
 {""repo_id"": 107914493, ""name"": ""0.37.1"", ""sha"": ""be20e6991eac2baa9b43e9b26ae209bae805ede5""},
 {""repo_id"": 107914493, ""name"": ""0.37"", ""sha"": ""c9e6841482b299fceadc5ad548c2dbf58a8f1227""},
 {""repo_id"": 107914493, ""name"": ""0.36"", ""sha"": ""b031fe97636b80b05fec409ee1dffb7d044fd4e9""},
 {""repo_id"": 107914493, ""name"": ""0.35"", ""sha"": ""30b6f71b306a43605c99bef79302ed5cb22d1924""},
 {""repo_id"": 107914493, ""name"": ""0.34"", ""sha"": ""e7f60d2a9b59752e20de8412f7b0a3e9a5359a31""},
 {""repo_id"": 107914493, ""name"": ""0.33"", ""sha"": ""59e7014c8a0f4102d7dc79f517540c55c49e1554""},
 {""repo_id"": 107914493, ""name"": ""0.32"", ""sha"": ""a95bedb9c423fa6d772c93ef47bc40f13a5bea50""},
 {""repo_id"": 107914493, ""name"": ""0.31.2"", ""sha"": ""b51f258d00bb3c3b401f15d46a1fbd50394dbe1c""},
 {""repo_id"": 107914493, ""name"": ""0.31.1"", ""sha"": ""a22c7761b61baa61b8e3da7d30887468d61d6b83""},
 {""repo_id"": 107914493, ""name"": ""0.31"", ""sha"": ""7f89928062b1a1fdb2625a946f7cd5161e597401""},
 {""repo_id"": 107914493, ""name"": ""0.30.2"", ""sha"": ""2bf7ce5f517d772a16d7855a35a8a75d4456aad7""},
 {""repo_id"": 107914493, ""name"": ""0.30.1"", ""sha"": ""3ca290e0db03bb4747e24203c445873f74512107""},
 {""repo_id"": 107914493, ""name"": ""0.30"", ""sha"": ""8050f9e1ece9afd0236ad38c6458c12a4ad917e6""},
 {""repo_id"": 107914493, ""name"": ""0.29.3"", ""sha"": ""0fc8afde0eb5ef677f4ac31601540d6168c8208d""},
 {""repo_id"": 107914493, ""name"": ""0.29.2"", ""sha"": ""6abe6faff6b035e9334dd05f8c741ae9b7a47440""},
 {""repo_id"": 107914493, ""name"": ""0.29.1"", ""sha"": ""2a94f3719fb2c4335fcda374fa92f87272b02d34""},
 {""repo_id"": 107914493, ""name"": ""0.29"", ""sha"": ""fb7ee8e0ad59a15083234a48e935525f6e7257dd""},
 {""repo_id"": 107914493, ""name"": ""0.28"", ""sha"": ""e518f76c5f5dd0138032bfb26387f5bb91086a3f""},
 {""repo_id"": 107914493, ""name"": ""0.27.1"", ""sha"": ""3f3f29ac9afe7c41ffc48a3bd2af473a53eecc8a""},
 {""repo_id"": 107914493, ""name"": ""0.27"", ""sha"": ""436b8bc1d17c2ab415800ab209204f94e7f7929e""},
 {""repo_id"": 107914493, ""name"": ""0.26.2"", ""sha"": ""a418c8b44f82d456be523c8690cf7236bb648c22""},
 {""repo_id"": 107914493, ""name"": ""0.26.1"", ""sha"": ""4722acc73ce761556b18f5dcbe36b7fef2ee2c69""},
 {""repo_id"": 107914493, ""name"": ""0.26"", ""sha"": ""424e146697309a54c05d5d1ba1f840849ddbafdc""},
 {""repo_id"": 107914493, ""name"": ""0.25.2"", ""sha"": ""b5128fc53fce6a1bf3b16bad9f318451bc1d1263""},
 {""repo_id"": 107914493, ""name"": ""0.25.1"", ""sha"": ""3dc0b3fa8c9b9bd81540ffe20c8b7e7a72465274""},
 {""repo_id"": 107914493, ""name"": ""0.25"", ""sha"": ""57a71377c992753327a16b417daf79df7f506dd1""},
 {""repo_id"": 107914493, ""name"": ""0.24"", ""sha"": ""28872a1fa789f314b0342f4e6182f1c78d6e2bca""},
 {""repo_id"": 107914493, ""name"": ""0.23.2"", ""sha"": ""6df6f712b36f0fe75694174906e31242427a8d1d""},
 {""repo_id"": 107914493, ""name"": ""0.23.1"", ""sha"": ""dea86b9fba78e032ad09673e884e764387daf209""},
 {""repo_id"": 107914493, ""name"": ""0.23"", ""sha"": ""e04f5b0d348ef7275a0a5ab9eb53527105132885""},
 {""repo_id"": 107914493, ""name"": ""0.22.1"", ""sha"": ""5d6252788230d168ba09f379d1d2af867e3302ab""},
 {""repo_id"": 107914493, ""name"": ""0.22"", ""sha"": ""558d9d7bfef3dd633eb16389281b67d42c9bdeef""},
 {""repo_id"": 107914493, ""name"": ""0.21"", ""sha"": ""403211de632cd15f0820cc9399305fc43c187b47""},
 {""repo_id"": 107914493, ""name"": ""0.20"", ""sha"": ""3a5d7951ce8f35118ffdd7f8d86e09b909e1218c""},
 {""repo_id"": 107914493, ""name"": ""0.19"", ""sha"": ""ba9bfa583179c25aaef94b1f44da7eba74620b9a""},
 {""repo_id"": 107914493, ""name"": ""0.18"", ""sha"": ""43ae15c0d14b3e968e8d5bfef72ac0c39783c3a2""},
 {""repo_id"": 107914493, ""name"": ""0.17"", ""sha"": ""fb988ace7c7e2bee5ac142a0eab22431d0675a77""},
 {""repo_id"": 107914493, ""name"": ""0.16"", ""sha"": ""b6539ff04502536bd1fa96e3b1430bdafc456826""},
 {""repo_id"": 107914493, ""name"": ""0.15"", ""sha"": ""7706fe0c67aba5cfe905c7906cae9e0c43cd75b2""},
 {""repo_id"": 107914493, ""name"": ""0.14"", ""sha"": ""2edc652df6d786e4f2c3f073e3567002d248be09""},
 {""repo_id"": 107914493, ""name"": ""0.13"", ""sha"": ""c160f15c3937f8fbe581276f811e8c58f9137bb1""},
 {""repo_id"": 107914493, ""name"": ""0.12"", ""sha"": ""51bdd67691bd69082ae7690af8b905f06050ee80""},
 {""repo_id"": 107914493, ""name"": ""0.11"", ""sha"": ""b0f3d4e375655f0764f3137dbcede324f9bbc0cb""},
 {""repo_id"": 107914493, ""name"": ""0.10"", ""sha"": ""5928c11ee798a232aa4096706cd47e639d1c9fc2""},
 {""repo_id"": 107914493, ""name"": ""0.9"", ""sha"": ""d75f423b6fcfc074b7c6f8f7679da8876f181edd""},
 {""repo_id"": 107914493, ""name"": ""0.8"", ""sha"": ""fe279ab7b4ae99dab295d5cf4d39ad06d782997e""},
 {""repo_id"": 107914493, ""name"": ""0.7"", ""sha"": ""6b3b05b6db0d2a7b7cec8b8dbb4ddc5e12a376b2""},
 {""repo_id"": 207052882, ""name"": ""2.3"", ""sha"": ""7090e43d804724ef3b31ae5ca9efd6ac05f76cbc""},
 {""repo_id"": 207052882, ""name"": ""2.2"", ""sha"": ""4fe69783b55465e7692a807d3a02a710f69c9c42""},
 {""repo_id"": 207052882, ""name"": ""2.1"", ""sha"": ""9d7aed336c8e62bf372caa800cb4aae3985cbae9""},
 {""repo_id"": 207052882, ""name"": ""2.0"", ""sha"": ""44611df1524a03ce305405e5902c9615e3c73a72""},
 {""repo_id"": 207052882, ""name"": ""1.1"", ""sha"": ""5cd34bd07d704487d48ac741ee5da5317afe88d2""},
 {""repo_id"": 207052882, ""name"": ""1.0.1"", ""sha"": ""3b7ab5685de89fcb6fc92d320c0e24b17be05570""},
 {""repo_id"": 207052882, ""name"": ""1.0"", ""sha"": ""1ea30c8fb1d080bd5e38c577e3ad20bb527a2fe6""},
 {""repo_id"": 207052882, ""name"": ""0.7"", ""sha"": ""e35eec4343aa560c58c1634cc228d0d46c442304""},
 {""repo_id"": 207052882, ""name"": ""0.6"", ""sha"": ""9eb737090fafd0e5a7e314be48402374d99e9828""},
 {""repo_id"": 207052882, ""name"": ""0.5"", ""sha"": ""ae9035f8fe5aff1c54bff4c6b4c2e808a44f0f2a""},
 {""repo_id"": 207052882, ""name"": ""0.4"", ""sha"": ""8c6251c31a05c58c2bfbef114247642d1b3dbb44""},
 {""repo_id"": 207052882, ""name"": ""0.3"", ""sha"": ""f697f247468516aa4ee13b1862b59e0dba18d00f""},
 {""repo_id"": 207052882, ""name"": ""0.2"", ""sha"": ""0fe96bc50fb3d7b1c7e4577db0ddf207eaeebbb9""},
 {""repo_id"": 207052882, ""name"": ""0.1.1"", ""sha"": ""321e0284c64dc48b2143311009886293c05edb07""},
 {""repo_id"": 207052882, ""name"": ""0.1"", ""sha"": ""7387c88a3f84704548e81d43b91615c02b61a957""}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660355904,github-to-sqlite tags command for fetching tags,
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660548780,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43,660548780,MDEyOklzc3VlQ29tbWVudDY2MDU0ODc4MA==,9599,simonw,2020-07-18T22:02:37Z,2020-07-18T23:05:56Z,MEMBER,https://github-to-sqlite.dogsheep.net/github/tags?_facet=repo,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660355904,github-to-sqlite tags command for fetching tags,
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660551397,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43,660551397,MDEyOklzc3VlQ29tbWVudDY2MDU1MTM5Nw==,9599,simonw,2020-07-18T22:27:32Z,2020-07-18T23:05:45Z,MEMBER,"```sql
with most_recent_releases as (
  with ranked as (
    select
      repo,
      tag_name,
      published_at,
      row_number() OVER (
        partition BY repo
        ORDER BY
          published_at DESC
      ) rank
    FROM
      releases
  )
  select
    *
  from
    ranked
  where
    rank = 1
)
select
  repos.full_name as repo,
  most_recent_releases.tag_name as release,
  commits.committer_date as release_commit_date,
  (
    select
      count(*)
    from
      commits c2
    where
      c2.repo = repos.id
      and c2.committer_date > commits.committer_date
  ) as commits_since_release,
  'https://github.com/' || repos.full_name || '/compare/' || most_recent_releases.tag_name || '...' || repos.default_branch as view_commits
from
  most_recent_releases
  join repos on most_recent_releases.repo = repos.id
  join tags on tags.repo = repos.id
  and tags.name = most_recent_releases.tag_name
  join commits on tags.sha = commits.sha
order by
  commits_since_release desc
```


repo | release | release_commit_date | commits_since_release | view_commits
-- | -- | -- | -- | --
simonw/datasette | 0.45 | 2020-07-01T21:43:07Z | 9 | https://github.com/simonw/datasette/compare/0.45...master
dogsheep/twitter-to-sqlite | 0.21.1 | 2020-04-30T18:20:43Z | 2 | https://github.com/dogsheep/twitter-to-sqlite/compare/0.21.1...master
dogsheep/github-to-sqlite | 2.3 | 2020-07-09T23:26:34Z | 2 | https://github.com/dogsheep/github-to-sqlite/compare/2.3...master
dogsheep/dogsheep-photos | 0.4.1 | 2020-05-25T20:11:20Z | 2 | https://github.com/dogsheep/dogsheep-photos/compare/0.4.1...master
dogsheep/swarm-to-sqlite | 0.3.1 | 2020-03-28T02:29:41Z | 1 | https://github.com/dogsheep/swarm-to-sqlite/compare/0.3.1...master
dogsheep/hacker-news-to-sqlite | 0.3.1 | 2020-03-21T22:39:34Z | 1 | https://github.com/dogsheep/hacker-news-to-sqlite/compare/0.3.1...master
simonw/sqlite-utils | 2.11 | 2020-07-08T17:36:07Z | 0 | https://github.com/simonw/sqlite-utils/compare/2.11...master
dogsheep/healthkit-to-sqlite | 0.5 | 2020-03-28T01:50:51Z | 0 | https://github.com/dogsheep/healthkit-to-sqlite/compare/0.5...master
dogsheep/inaturalist-to-sqlite | 0.2 | 2020-03-24T00:35:44Z | 0 | https://github.com/dogsheep/inaturalist-to-sqlite/compare/0.2...master
dogsheep/genome-to-sqlite | 0.1 | 2019-09-19T15:38:10Z | 0 | https://github.com/dogsheep/genome-to-sqlite/compare/0.1...master
dogsheep/pocket-to-sqlite | 0.2 | 2020-03-27T22:23:16Z | 0 | https://github.com/dogsheep/pocket-to-sqlite/compare/0.2...master

https://github-to-sqlite.dogsheep.net/github?sql=with+most_recent_releases+as+%28%0D%0A++with+ranked+as+%28%0D%0A++++select%0D%0A++++++repo%2C%0D%0A++++++tag_name%2C%0D%0A++++++published_at%2C%0D%0A++++++row_number%28%29+OVER+%28%0D%0A++++++++partition+BY+repo%0D%0A++++++++ORDER+BY%0D%0A++++++++++published_at+DESC%0D%0A++++++%29+rank%0D%0A++++FROM%0D%0A++++++releases%0D%0A++%29%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++ranked%0D%0A++where%0D%0A++++rank+%3D+1%0D%0A%29%0D%0Aselect%0D%0A++repos.full_name+as+repo%2C%0D%0A++most_recent_releases.tag_name+as+release%2C%0D%0A++commits.committer_date+as+release_commit_date%2C%0D%0A++%28%0D%0A++++select%0D%0A++++++count%28*%29%0D%0A++++from%0D%0A++++++commits+c2%0D%0A++++where%0D%0A++++++c2.repo+%3D+repos.id%0D%0A++++++and+c2.committer_date+%3E+commits.committer_date%0D%0A++%29+as+commits_since_release%2C%0D%0A++%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos.full_name+%7C%7C+%27%2Fcompare%2F%27+%7C%7C+most_recent_releases.tag_name+%7C%7C+%27...%27+%7C%7C+repos.default_branch+as+view_commits%0D%0Afrom%0D%0A++most_recent_releases%0D%0A++join+repos+on+most_recent_releases.repo+%3D+repos.id%0D%0A++join+tags+on+tags.repo+%3D+repos.id%0D%0A++and+tags.name+%3D+most_recent_releases.tag_name%0D%0A++join+commits+on+tags.sha+%3D+commits.sha%0D%0Aorder+by%0D%0A++commits_since_release+desc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660355904,github-to-sqlite tags command for fetching tags,
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553646,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45,660553646,MDEyOklzc3VlQ29tbWVudDY2MDU1MzY0Ng==,9599,simonw,2020-07-18T22:51:41Z,2020-07-18T22:51:41Z,MEMBER,"I could fix this by putting `REFRESH_DB` in a commit message:

https://github.com/dogsheep/github-to-sqlite/blob/4ae4aa6f172344b19ff3513707195ee6d2654bd4/.github/workflows/deploy-demo.yml#L41-L46

But... doing so would lose the data I've collected in https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc concerning the first time each dependent repo was spotted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660429601,Fix the demo - it breaks because of the tags table change,
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553711,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45,660553711,MDEyOklzc3VlQ29tbWVudDY2MDU1MzcxMQ==,9599,simonw,2020-07-18T22:52:16Z,2020-07-18T22:52:16Z,MEMBER,"I think the best fix is to download the `github.db` database, manually fix it and then manually deploy it to Cloud Run from my laptop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660429601,Fix the demo - it breaks because of the tags table change,
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554162,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45,660554162,MDEyOklzc3VlQ29tbWVudDY2MDU1NDE2Mg==,9599,simonw,2020-07-18T22:56:58Z,2020-07-18T22:56:58Z,MEMBER,"Manually fixing the database:
```
$ wget 'https://github-to-sqlite.dogsheep.net/github.db'
--2020-07-18 15:52:33--  https://github-to-sqlite.dogsheep.net/github.db
Resolving github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)... 172.217.5.115
Connecting to github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)|172.217.5.115|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14626816 (14M) [application/octet-stream]
Saving to: ‘github.db’

github.db                              100%[============================================================================>]  13.95M  1.22MB/s    in 18s     

2020-07-18 15:52:53 (773 KB/s) - ‘github.db’ saved [14626816/14626816]

$ sqlite3 github.db 
SQLite version 3.28.0 2019-04-15 14:49:49
Enter "".help"" for usage hints.
sqlite> drop table tags;
sqlite> ^D
$ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils dogsheep/healthkit-to-sqlite dogsheep/swarm-to-sqlite dogsheep/twitter-to-sqlite dogsheep/inaturalist-to-sqlite dogsheep/google-takeout-to-sqlite dogsheep/github-to-sqlite dogsheep/genome-to-sqlite dogsheep/pocket-to-sqlite dogsheep/hacker-news-to-sqlite dogsheep/dogsheep-photos 
$ sqlite-utils tables github.db --counts
[{""table"": ""users"", ""count"": 4048},
 {""table"": ""repos"", ""count"": 210},
 ...
 {""table"": ""stars"", ""count"": 4140},
 {""table"": ""tags"", ""count"": 188}]
$ sqlite-utils rows github.db tags    
[{""repo"": 107914493, ""name"": ""0.45"", ""sha"": ""f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb""},
 {""repo"": 107914493, ""name"": ""0.45a5"", ""sha"": ""676bb64c877d73f8ff496cef4632f5a8a5a9283c""},
 ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660429601,Fix the demo - it breaks because of the tags table change,
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554299,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45,660554299,MDEyOklzc3VlQ29tbWVudDY2MDU1NDI5OQ==,9599,simonw,2020-07-18T22:58:24Z,2020-07-18T23:02:52Z,MEMBER,"Deploying the fixed version like this:
```
$ gcloud config set run/region us-central1
$ gcloud config set project datasette-222320
$ datasette publish cloudrun /tmp/github.db \
            -m demo-metadata.json \
            --service github-to-sqlite \
            --install=py-gfm \
            --install='datasette-search-all>=0.3' \
            --install='datasette-render-markdown>=1.1.2' \
            --install=datasette-pretty-json \
            --install=datasette-json-html \
            --install=datasette-vega
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660429601,Fix the demo - it breaks because of the tags table change,
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554811,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45,660554811,MDEyOklzc3VlQ29tbWVudDY2MDU1NDgxMQ==,9599,simonw,2020-07-18T23:03:13Z,2020-07-18T23:03:13Z,MEMBER,https://github-to-sqlite.dogsheep.net/github/tags now shows a `repo` column instead of a `repo_id` column.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660429601,Fix the demo - it breaks because of the tags table change,
https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-1359468823,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46,1359468823,IC_kwDODFdgUs5RB9kX,1839645,choldgraf,2022-12-20T14:39:39Z,2022-12-20T14:40:15Z,NONE,"Just a quick +1 to this one from me - I would like to do a better job of tracking who is reviewing one another's pull requests in repositories, since this is a specific kind of maintenance work that I think often goes unrewarded. I can't seem to figure this out just by looking at the `pull_request` or `issue_comments` tables, so I think it would be helpful to support PR reviews natively (even if just for summary statistics). Alternatively if there is a way in the API to tell if an issue comment is part of a review, then perhaps you could quickly calculate the number of unique reviews that an author performed. But that was beyond my SQL-foo :-) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",664485022,Feature: pull request reviews and comments,
https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-675259273,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46,675259273,MDEyOklzc3VlQ29tbWVudDY3NTI1OTI3Mw==,9599,simonw,2020-08-18T05:28:32Z,2020-08-18T05:28:32Z,MEMBER,"Oh that's interesting - i didn't realize ""reviews"" were a separate concept. I'd definitely accept a pull request adding those!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",664485022,Feature: pull request reviews and comments,
https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482187,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46,735482187,MDEyOklzc3VlQ29tbWVudDczNTQ4MjE4Nw==,9599,simonw,2020-11-30T00:20:11Z,2020-11-30T00:20:11Z,MEMBER,"Pull request are now added, thanks to  @adamjonas.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",664485022,Feature: pull request reviews and comments,
https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482546,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46,735482546,MDEyOklzc3VlQ29tbWVudDczNTQ4MjU0Ng==,9599,simonw,2020-11-30T00:22:02Z,2020-11-30T00:22:02Z,MEMBER,"As for reviews... here's the output of `github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews --accept 'application/vnd.github.v3+json'`

```json
[
    {
        ""id"": 503368921,
        ""node_id"": ""MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTAzMzY4OTIx"",
        ""user"": {
            ""login"": ""simonw"",
            ""id"": 9599,
            ""node_id"": ""MDQ6VXNlcjk1OTk="",
            ""avatar_url"": ""https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/simonw"",
            ""html_url"": ""https://github.com/simonw"",
            ""followers_url"": ""https://api.github.com/users/simonw/followers"",
            ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/simonw/orgs"",
            ""repos_url"": ""https://api.github.com/users/simonw/repos"",
            ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/simonw/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""body"": """",
        ""state"": ""CHANGES_REQUESTED"",
        ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-503368921"",
        ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"",
        ""author_association"": ""MEMBER"",
        ""_links"": {
            ""html"": {
                ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-503368921""
            },
            ""pull_request"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48""
            }
        },
        ""submitted_at"": ""2020-10-06T21:28:40Z"",
        ""commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c""
    },
    {
        ""id"": 513118561,
        ""node_id"": ""MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTEzMTE4NTYx"",
        ""user"": {
            ""login"": ""adamjonas"",
            ""id"": 755825,
            ""node_id"": ""MDQ6VXNlcjc1NTgyNQ=="",
            ""avatar_url"": ""https://avatars1.githubusercontent.com/u/755825?v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/adamjonas"",
            ""html_url"": ""https://github.com/adamjonas"",
            ""followers_url"": ""https://api.github.com/users/adamjonas/followers"",
            ""following_url"": ""https://api.github.com/users/adamjonas/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/adamjonas/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/adamjonas/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/adamjonas/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/adamjonas/orgs"",
            ""repos_url"": ""https://api.github.com/users/adamjonas/repos"",
            ""events_url"": ""https://api.github.com/users/adamjonas/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/adamjonas/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""body"": """",
        ""state"": ""COMMENTED"",
        ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513118561"",
        ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"",
        ""author_association"": ""CONTRIBUTOR"",
        ""_links"": {
            ""html"": {
                ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513118561""
            },
            ""pull_request"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48""
            }
        },
        ""submitted_at"": ""2020-10-20T20:45:05Z"",
        ""commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c""
    },
    {
        ""id"": 513127529,
        ""node_id"": ""MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTEzMTI3NTI5"",
        ""user"": {
            ""login"": ""adamjonas"",
            ""id"": 755825,
            ""node_id"": ""MDQ6VXNlcjc1NTgyNQ=="",
            ""avatar_url"": ""https://avatars1.githubusercontent.com/u/755825?v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/adamjonas"",
            ""html_url"": ""https://github.com/adamjonas"",
            ""followers_url"": ""https://api.github.com/users/adamjonas/followers"",
            ""following_url"": ""https://api.github.com/users/adamjonas/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/adamjonas/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/adamjonas/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/adamjonas/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/adamjonas/orgs"",
            ""repos_url"": ""https://api.github.com/users/adamjonas/repos"",
            ""events_url"": ""https://api.github.com/users/adamjonas/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/adamjonas/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""body"": """",
        ""state"": ""COMMENTED"",
        ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513127529"",
        ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"",
        ""author_association"": ""CONTRIBUTOR"",
        ""_links"": {
            ""html"": {
                ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513127529""
            },
            ""pull_request"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48""
            }
        },
        ""submitted_at"": ""2020-10-20T20:57:33Z"",
        ""commit_id"": ""3a0d5c498f9faae4e40aab204cd01b965a4f61f3""
    }
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",664485022,Feature: pull request reviews and comments,
https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483604,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46,735483604,MDEyOklzc3VlQ29tbWVudDczNTQ4MzYwNA==,9599,simonw,2020-11-30T00:26:50Z,2020-11-30T00:26:50Z,MEMBER,"It seems like there's a lot missing from that - those aren't particularly interesting given the data that is returned.

From the docs at https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#reviews it looks like each review consists of multiple comments, and the comments are where the useful material is - https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#list-comments-for-a-pull-request-review

`github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews/503368921/comments --accept 'application/vnd.github.v3+json'`

```json
[
    {
        ""id"": 500603838,
        ""node_id"": ""MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwMzgzOA=="",
        ""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500603838"",
        ""pull_request_review_id"": 503368921,
        ""diff_hunk"": ""@@ -0,0 +1,370 @@\n+[\n+    {\n+        \""url\"": \""https://api.github.com/repos/simonw/datasette/pulls/571\"",\n+        \""id\"": 313384926,\n+        \""node_id\"": \""MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2\"",\n+        \""html_url\"": \""https://github.com/simonw/datasette/pull/571\"",\n+        \""diff_url\"": \""https://github.com/simonw/datasette/pull/571.diff\"",\n+        \""patch_url\"": \""https://github.com/simonw/datasette/pull/571.patch\"",\n+        \""issue_url\"": \""https://api.github.com/repos/simonw/datasette/issues/571\"",\n+        \""number\"": 571,\n+        \""state\"": \""closed\"",\n+        \""locked\"": false,\n+        \""title\"": \""detect_fts now works with alternative table escaping\"",\n+        \""user\"": {\n+          \""login\"": \""simonw\"",\n+          \""id\"": 9599,\n+          \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+          \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+          \""gravatar_id\"": \""\"",\n+          \""url\"": \""https://api.github.com/users/simonw\"",\n+          \""html_url\"": \""https://github.com/simonw\"",\n+          \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+          \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+          \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+          \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+          \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+          \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+          \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+          \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+          \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+          \""type\"": \""User\"",\n+          \""site_admin\"": false\n+        },\n+        \""body\"": \""Fixes #570\"",\n+        \""created_at\"": \""2019-09-03T00:23:39Z\"",\n+        \""updated_at\"": \""2019-09-03T00:32:28Z\"",\n+        \""closed_at\"": \""2019-09-03T00:32:28Z\"",\n+        \""merged_at\"": \""2019-09-03T00:32:28Z\"",\n+        \""merge_commit_sha\"": \""2dc5c8dc259a0606162673d394ba8cc1c6f54428\"",\n+        \""assignee\"": null,\n+        \""assignees\"": [\n+\n+        ],\n+        \""requested_reviewers\"": [\n+\n+        ],\n+        \""requested_teams\"": [\n+\n+        ],\n+        \""labels\"": [\n+\n+        ],\n+        \""milestone\"": null,\n+        \""draft\"": false,\n+        \""commits_url\"": \""https://api.github.com/repos/simonw/datasette/pulls/571/commits\"",\n+        \""review_comments_url\"": \""https://api.github.com/repos/simonw/datasette/pulls/571/comments\"",\n+        \""review_comment_url\"": \""https://api.github.com/repos/simonw/datasette/pulls/comments{/number}\"",\n+        \""comments_url\"": \""https://api.github.com/repos/simonw/datasette/issues/571/comments\"",\n+        \""statuses_url\"": \""https://api.github.com/repos/simonw/datasette/statuses/a85239f69261c10f1a9f90514c8b5d113cb94585\"",\n+        \""head\"": {\n+          \""label\"": \""simonw:detect-fts\"",\n+          \""ref\"": \""detect-fts\"",\n+          \""sha\"": \""a85239f69261c10f1a9f90514c8b5d113cb94585\"",\n+          \""user\"": {\n+            \""login\"": \""simonw\"",\n+            \""id\"": 9599,\n+            \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+            \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+            \""gravatar_id\"": \""\"",\n+            \""url\"": \""https://api.github.com/users/simonw\"",\n+            \""html_url\"": \""https://github.com/simonw\"",\n+            \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+            \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+            \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+            \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+            \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+            \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+            \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+            \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+            \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+            \""type\"": \""User\"",\n+            \""site_admin\"": false\n+          },\n+          \""repo\"": {\n+            \""id\"": 107914493,\n+            \""node_id\"": \""MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=\"",\n+            \""name\"": \""datasette\"",\n+            \""full_name\"": \""simonw/datasette\"",\n+            \""private\"": false,\n+            \""owner\"": {\n+              \""login\"": \""simonw\"",\n+              \""id\"": 9599,\n+              \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+              \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+              \""gravatar_id\"": \""\"",\n+              \""url\"": \""https://api.github.com/users/simonw\"",\n+              \""html_url\"": \""https://github.com/simonw\"",\n+              \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+              \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+              \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+              \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+              \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+              \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+              \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+              \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+              \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+              \""type\"": \""User\"",\n+              \""site_admin\"": false\n+            },\n+            \""html_url\"": \""https://github.com/simonw/datasette\"",\n+            \""description\"": \""An open source multi-tool for exploring and publishing data\"",\n+            \""fork\"": false,\n+            \""url\"": \""https://api.github.com/repos/simonw/datasette\"",\n+            \""forks_url\"": \""https://api.github.com/repos/simonw/datasette/forks\"",\n+            \""keys_url\"": \""https://api.github.com/repos/simonw/datasette/keys{/key_id}\"",\n+            \""collaborators_url\"": \""https://api.github.com/repos/simonw/datasette/collaborators{/collaborator}\"",\n+            \""teams_url\"": \""https://api.github.com/repos/simonw/datasette/teams\"",\n+            \""hooks_url\"": \""https://api.github.com/repos/simonw/datasette/hooks\"",\n+            \""issue_events_url\"": \""https://api.github.com/repos/simonw/datasette/issues/events{/number}\"",\n+            \""events_url\"": \""https://api.github.com/repos/simonw/datasette/events\"",\n+            \""assignees_url\"": \""https://api.github.com/repos/simonw/datasette/assignees{/user}\"",\n+            \""branches_url\"": \""https://api.github.com/repos/simonw/datasette/branches{/branch}\"",\n+            \""tags_url\"": \""https://api.github.com/repos/simonw/datasette/tags\"",\n+            \""blobs_url\"": \""https://api.github.com/repos/simonw/datasette/git/blobs{/sha}\"",\n+            \""git_tags_url\"": \""https://api.github.com/repos/simonw/datasette/git/tags{/sha}\"",\n+            \""git_refs_url\"": \""https://api.github.com/repos/simonw/datasette/git/refs{/sha}\"",\n+            \""trees_url\"": \""https://api.github.com/repos/simonw/datasette/git/trees{/sha}\"",\n+            \""statuses_url\"": \""https://api.github.com/repos/simonw/datasette/statuses/{sha}\"",\n+            \""languages_url\"": \""https://api.github.com/repos/simonw/datasette/languages\"",\n+            \""stargazers_url\"": \""https://api.github.com/repos/simonw/datasette/stargazers\"",\n+            \""contributors_url\"": \""https://api.github.com/repos/simonw/datasette/contributors\"",\n+            \""subscribers_url\"": \""https://api.github.com/repos/simonw/datasette/subscribers\"",\n+            \""subscription_url\"": \""https://api.github.com/repos/simonw/datasette/subscription\"",\n+            \""commits_url\"": \""https://api.github.com/repos/simonw/datasette/commits{/sha}\"",\n+            \""git_commits_url\"": \""https://api.github.com/repos/simonw/datasette/git/commits{/sha}\"",\n+            \""comments_url\"": \""https://api.github.com/repos/simonw/datasette/comments{/number}\"",\n+            \""issue_comment_url\"": \""https://api.github.com/repos/simonw/datasette/issues/comments{/number}\"",\n+            \""contents_url\"": \""https://api.github.com/repos/simonw/datasette/contents/{+path}\"",\n+            \""compare_url\"": \""https://api.github.com/repos/simonw/datasette/compare/{base}...{head}\"",\n+            \""merges_url\"": \""https://api.github.com/repos/simonw/datasette/merges\"",\n+            \""archive_url\"": \""https://api.github.com/repos/simonw/datasette/{archive_format}{/ref}\"",\n+            \""downloads_url\"": \""https://api.github.com/repos/simonw/datasette/downloads\"",\n+            \""issues_url\"": \""https://api.github.com/repos/simonw/datasette/issues{/number}\"",\n+            \""pulls_url\"": \""https://api.github.com/repos/simonw/datasette/pulls{/number}\"",\n+            \""milestones_url\"": \""https://api.github.com/repos/simonw/datasette/milestones{/number}\"",\n+            \""notifications_url\"": \""https://api.github.com/repos/simonw/datasette/notifications{?since,all,participating}\"",\n+            \""labels_url\"": \""https://api.github.com/repos/simonw/datasette/labels{/name}\"",\n+            \""releases_url\"": \""https://api.github.com/repos/simonw/datasette/releases{/id}\"",\n+            \""deployments_url\"": \""https://api.github.com/repos/simonw/datasette/deployments\"",\n+            \""created_at\"": \""2017-10-23T00:39:03Z\"",\n+            \""updated_at\"": \""2020-07-27T20:42:15Z\"",\n+            \""pushed_at\"": \""2020-07-26T01:21:05Z\"",\n+            \""git_url\"": \""git://github.com/simonw/datasette.git\"",\n+            \""ssh_url\"": \""git@github.com:simonw/datasette.git\"",\n+            \""clone_url\"": \""https://github.com/simonw/datasette.git\"",\n+            \""svn_url\"": \""https://github.com/simonw/datasette\"",\n+            \""homepage\"": \""http://datasette.readthedocs.io/\"",\n+            \""size\"": 3487,\n+            \""stargazers_count\"": 3642,\n+            \""watchers_count\"": 3642,\n+            \""language\"": \""Python\"",\n+            \""has_issues\"": true,\n+            \""has_projects\"": false,\n+            \""has_downloads\"": true,\n+            \""has_wiki\"": true,\n+            \""has_pages\"": false,\n+            \""forks_count\"": 206,\n+            \""mirror_url\"": null,\n+            \""archived\"": false,\n+            \""disabled\"": false,\n+            \""open_issues_count\"": 190,\n+            \""license\"": {\n+              \""key\"": \""apache-2.0\"",\n+              \""name\"": \""Apache License 2.0\"",\n+              \""spdx_id\"": \""Apache-2.0\"",\n+              \""url\"": \""https://api.github.com/licenses/apache-2.0\"",\n+              \""node_id\"": \""MDc6TGljZW5zZTI=\""\n+            },\n+            \""forks\"": 206,\n+            \""open_issues\"": 190,\n+            \""watchers\"": 3642,\n+            \""default_branch\"": \""master\""\n+          }\n+        },\n+        \""base\"": {\n+          \""label\"": \""simonw:master\"",\n+          \""ref\"": \""master\"",\n+          \""sha\"": \""f04deebec4f3842f7bd610cd5859de529f77d50e\"",\n+          \""user\"": {\n+            \""login\"": \""simonw\"",\n+            \""id\"": 9599,\n+            \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+            \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+            \""gravatar_id\"": \""\"",\n+            \""url\"": \""https://api.github.com/users/simonw\"",\n+            \""html_url\"": \""https://github.com/simonw\"",\n+            \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+            \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+            \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+            \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+            \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+            \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+            \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+            \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+            \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+            \""type\"": \""User\"",\n+            \""site_admin\"": false\n+          },\n+          \""repo\"": {\n+            \""id\"": 107914493,\n+            \""node_id\"": \""MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=\"",\n+            \""name\"": \""datasette\"",\n+            \""full_name\"": \""simonw/datasette\"",\n+            \""private\"": false,\n+            \""owner\"": {\n+              \""login\"": \""simonw\"",\n+              \""id\"": 9599,\n+              \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+              \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+              \""gravatar_id\"": \""\"",\n+              \""url\"": \""https://api.github.com/users/simonw\"",\n+              \""html_url\"": \""https://github.com/simonw\"",\n+              \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+              \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+              \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+              \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+              \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+              \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+              \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+              \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+              \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+              \""type\"": \""User\"",\n+              \""site_admin\"": false\n+            },\n+            \""html_url\"": \""https://github.com/simonw/datasette\"",\n+            \""description\"": \""An open source multi-tool for exploring and publishing data\"",\n+            \""fork\"": false,\n+            \""url\"": \""https://api.github.com/repos/simonw/datasette\"",\n+            \""forks_url\"": \""https://api.github.com/repos/simonw/datasette/forks\"",\n+            \""keys_url\"": \""https://api.github.com/repos/simonw/datasette/keys{/key_id}\"",\n+            \""collaborators_url\"": \""https://api.github.com/repos/simonw/datasette/collaborators{/collaborator}\"",\n+            \""teams_url\"": \""https://api.github.com/repos/simonw/datasette/teams\"",\n+            \""hooks_url\"": \""https://api.github.com/repos/simonw/datasette/hooks\"",\n+            \""issue_events_url\"": \""https://api.github.com/repos/simonw/datasette/issues/events{/number}\"",\n+            \""events_url\"": \""https://api.github.com/repos/simonw/datasette/events\"",\n+            \""assignees_url\"": \""https://api.github.com/repos/simonw/datasette/assignees{/user}\"",\n+            \""branches_url\"": \""https://api.github.com/repos/simonw/datasette/branches{/branch}\"",\n+            \""tags_url\"": \""https://api.github.com/repos/simonw/datasette/tags\"",\n+            \""blobs_url\"": \""https://api.github.com/repos/simonw/datasette/git/blobs{/sha}\"",\n+            \""git_tags_url\"": \""https://api.github.com/repos/simonw/datasette/git/tags{/sha}\"",\n+            \""git_refs_url\"": \""https://api.github.com/repos/simonw/datasette/git/refs{/sha}\"",\n+            \""trees_url\"": \""https://api.github.com/repos/simonw/datasette/git/trees{/sha}\"",\n+            \""statuses_url\"": \""https://api.github.com/repos/simonw/datasette/statuses/{sha}\"",\n+            \""languages_url\"": \""https://api.github.com/repos/simonw/datasette/languages\"",\n+            \""stargazers_url\"": \""https://api.github.com/repos/simonw/datasette/stargazers\"",\n+            \""contributors_url\"": \""https://api.github.com/repos/simonw/datasette/contributors\"",\n+            \""subscribers_url\"": \""https://api.github.com/repos/simonw/datasette/subscribers\"",\n+            \""subscription_url\"": \""https://api.github.com/repos/simonw/datasette/subscription\"",\n+            \""commits_url\"": \""https://api.github.com/repos/simonw/datasette/commits{/sha}\"",\n+            \""git_commits_url\"": \""https://api.github.com/repos/simonw/datasette/git/commits{/sha}\"",\n+            \""comments_url\"": \""https://api.github.com/repos/simonw/datasette/comments{/number}\"",\n+            \""issue_comment_url\"": \""https://api.github.com/repos/simonw/datasette/issues/comments{/number}\"",\n+            \""contents_url\"": \""https://api.github.com/repos/simonw/datasette/contents/{+path}\"",\n+            \""compare_url\"": \""https://api.github.com/repos/simonw/datasette/compare/{base}...{head}\"",\n+            \""merges_url\"": \""https://api.github.com/repos/simonw/datasette/merges\"",\n+            \""archive_url\"": \""https://api.github.com/repos/simonw/datasette/{archive_format}{/ref}\"",\n+            \""downloads_url\"": \""https://api.github.com/repos/simonw/datasette/downloads\"",\n+            \""issues_url\"": \""https://api.github.com/repos/simonw/datasette/issues{/number}\"",\n+            \""pulls_url\"": \""https://api.github.com/repos/simonw/datasette/pulls{/number}\"",\n+            \""milestones_url\"": \""https://api.github.com/repos/simonw/datasette/milestones{/number}\"",\n+            \""notifications_url\"": \""https://api.github.com/repos/simonw/datasette/notifications{?since,all,participating}\"",\n+            \""labels_url\"": \""https://api.github.com/repos/simonw/datasette/labels{/name}\"",\n+            \""releases_url\"": \""https://api.github.com/repos/simonw/datasette/releases{/id}\"",\n+            \""deployments_url\"": \""https://api.github.com/repos/simonw/datasette/deployments\"",\n+            \""created_at\"": \""2017-10-23T00:39:03Z\"",\n+            \""updated_at\"": \""2020-07-27T20:42:15Z\"",\n+            \""pushed_at\"": \""2020-07-26T01:21:05Z\"",\n+            \""git_url\"": \""git://github.com/simonw/datasette.git\"",\n+            \""ssh_url\"": \""git@github.com:simonw/datasette.git\"",\n+            \""clone_url\"": \""https://github.com/simonw/datasette.git\"",\n+            \""svn_url\"": \""https://github.com/simonw/datasette\"",\n+            \""homepage\"": \""http://datasette.readthedocs.io/\"",\n+            \""size\"": 3487,\n+            \""stargazers_count\"": 3642,\n+            \""watchers_count\"": 3642,\n+            \""language\"": \""Python\"",\n+            \""has_issues\"": true,\n+            \""has_projects\"": false,\n+            \""has_downloads\"": true,\n+            \""has_wiki\"": true,\n+            \""has_pages\"": false,\n+            \""forks_count\"": 206,\n+            \""mirror_url\"": null,\n+            \""archived\"": false,\n+            \""disabled\"": false,\n+            \""open_issues_count\"": 190,\n+            \""license\"": {\n+              \""key\"": \""apache-2.0\"",\n+              \""name\"": \""Apache License 2.0\"",\n+              \""spdx_id\"": \""Apache-2.0\"",\n+              \""url\"": \""https://api.github.com/licenses/apache-2.0\"",\n+              \""node_id\"": \""MDc6TGljZW5zZTI=\""\n+            },\n+            \""forks\"": 206,\n+            \""open_issues\"": 190,\n+            \""watchers\"": 3642,\n+            \""default_branch\"": \""master\""\n+          }\n+        },\n+        \""_links\"": {\n+          \""self\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/pulls/571\""\n+          },\n+          \""html\"": {\n+            \""href\"": \""https://github.com/simonw/datasette/pull/571\""\n+          },\n+          \""issue\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/issues/571\""\n+          },\n+          \""comments\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/issues/571/comments\""\n+          },\n+          \""review_comments\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/pulls/571/comments\""\n+          },\n+          \""review_comment\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/pulls/comments{/number}\""\n+          },\n+          \""commits\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/pulls/571/commits\""\n+          },\n+          \""statuses\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/statuses/a85239f69261c10f1a9f90514c8b5d113cb94585\""\n+          }\n+        },\n+        \""author_association\"": \""OWNER\"",\n+        \""active_lock_reason\"": null,\n+        \""merged\"": true,\n+        \""mergeable\"": null,\n+        \""rebaseable\"": null,\n+        \""mergeable_state\"": \""unknown\"",\n+        \""merged_by\"": {"",
        ""path"": ""tests/pull_requests.json"",
        ""position"": 342,
        ""original_position"": 342,
        ""commit_id"": ""3a0d5c498f9faae4e40aab204cd01b965a4f61f3"",
        ""user"": {
            ""login"": ""simonw"",
            ""id"": 9599,
            ""node_id"": ""MDQ6VXNlcjk1OTk="",
            ""avatar_url"": ""https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/simonw"",
            ""html_url"": ""https://github.com/simonw"",
            ""followers_url"": ""https://api.github.com/users/simonw/followers"",
            ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/simonw/orgs"",
            ""repos_url"": ""https://api.github.com/users/simonw/repos"",
            ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/simonw/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""body"": ""Running this should create a `merged_by` column on the `pull_requests` table which is a foreign key to the `users` table."",
        ""created_at"": ""2020-10-06T21:22:47Z"",
        ""updated_at"": ""2020-10-20T20:56:33Z"",
        ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500603838"",
        ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"",
        ""author_association"": ""MEMBER"",
        ""_links"": {
            ""self"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500603838""
            },
            ""html"": {
                ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500603838""
            },
            ""pull_request"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48""
            }
        },
        ""original_commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c""
    },
    {
        ""id"": 500606198,
        ""node_id"": ""MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwNjE5OA=="",
        ""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606198"",
        ""pull_request_review_id"": 503368921,
        ""diff_hunk"": ""@@ -0,0 +1,124 @@\n+from github_to_sqlite import utils\n+import pytest\n+import pathlib\n+import sqlite_utils\n+from sqlite_utils.db import ForeignKey\n+import json\n+\n+\n+@pytest.fixture\n+def pull_requests():\n+    return json.load(open(pathlib.Path(__file__).parent / \""pull_requests.json\""))\n+\n+\n+@pytest.fixture\n+def db(pull_requests):\n+    db = sqlite_utils.Database(memory=True)\n+    db[\""repos\""].insert(\n+        {\""id\"": 1},\n+        pk=\""id\"",\n+        columns={\""organization\"": int, \""topics\"": str, \""name\"": str, \""description\"": str},\n+    )\n+    utils.save_pull_requests(db, pull_requests, {\""id\"": 1})\n+    return db\n+\n+\n+def test_tables(db):\n+    assert {\""pull_requests\"", \""users\"", \""repos\"", \""milestones\""} == set(\n+        db.table_names()\n+    )\n+    assert {\n+        ForeignKey(\n+            table=\""pull_requests\"", column=\""repo\"", other_table=\""repos\"", other_column=\""id\""\n+        ),\n+        ForeignKey(\n+            table=\""pull_requests\"",\n+            column=\""milestone\"",\n+            other_table=\""milestones\"",\n+            other_column=\""id\"",\n+        ),\n+        ForeignKey(\n+            table=\""pull_requests\"", column=\""assignee\"", other_table=\""users\"", other_column=\""id\""\n+        ),\n+        ForeignKey(\n+            table=\""pull_requests\"", column=\""user\"", other_table=\""users\"", other_column=\""id\""\n+        ),\n+    } == set(db[\""pull_requests\""].foreign_keys)\n+\n+\n+def test_pull_requests(db):\n+    pull_request_rows = list(db[\""pull_requests\""].rows)\n+    assert [\n+        {\n+            'id': 313384926,\n+            'node_id': 'MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2',\n+            'number': 571,\n+            'state': 'closed',\n+            'locked': 0,\n+            'title': 'detect_fts now works with alternative table escaping',\n+            'user': 9599,\n+            'body': 'Fixes #570',\n+            'created_at': '2019-09-03T00:23:39Z',\n+            'updated_at': '2019-09-03T00:32:28Z',\n+            'closed_at': '2019-09-03T00:32:28Z',\n+            'merged_at': '2019-09-03T00:32:28Z',\n+            'merge_commit_sha': '2dc5c8dc259a0606162673d394ba8cc1c6f54428',\n+            'assignee': None,\n+            'milestone': None,\n+            'draft': 0,\n+            'head': 'a85239f69261c10f1a9f90514c8b5d113cb94585',\n+            'base': 'f04deebec4f3842f7bd610cd5859de529f77d50e',\n+            'author_association': 'OWNER',\n+            'merged': 1,\n+            'mergeable': None,\n+            'rebaseable': None,\n+            'mergeable_state': 'unknown',\n+            'merged_by': '{\""login\"": \""simonw\"", \""id\"": 9599, \""node_id\"": \""MDQ6VXNlcjk1OTk=\"", \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"", \""gravatar_id\"": \""\"", \""url\"": \""https://api.github.com/users/simonw\"", \""html_url\"": \""https://github.com/simonw\"", \""followers_url\"": \""https://api.github.com/users/simonw/followers\"", \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"", \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"", \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"", \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"", \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"", \""repos_url\"": \""https://api.github.com/users/simonw/repos\"", \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"", \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"", \""type\"": \""User\"", \""site_admin\"": false}',"",
        ""path"": ""tests/test_pull_requests.py"",
        ""position"": null,
        ""original_position"": 76,
        ""commit_id"": ""3a0d5c498f9faae4e40aab204cd01b965a4f61f3"",
        ""user"": {
            ""login"": ""simonw"",
            ""id"": 9599,
            ""node_id"": ""MDQ6VXNlcjk1OTk="",
            ""avatar_url"": ""https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/simonw"",
            ""html_url"": ""https://github.com/simonw"",
            ""followers_url"": ""https://api.github.com/users/simonw/followers"",
            ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/simonw/orgs"",
            ""repos_url"": ""https://api.github.com/users/simonw/repos"",
            ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/simonw/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""body"": ""See above - this should be 9599, an integer reference to the row in the users table."",
        ""created_at"": ""2020-10-06T21:27:43Z"",
        ""updated_at"": ""2020-10-20T20:56:33Z"",
        ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606198"",
        ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"",
        ""author_association"": ""MEMBER"",
        ""_links"": {
            ""self"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606198""
            },
            ""html"": {
                ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606198""
            },
            ""pull_request"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48""
            }
        },
        ""original_commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c""
    },
    {
        ""id"": 500606665,
        ""node_id"": ""MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwNjY2NQ=="",
        ""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606665"",
        ""pull_request_review_id"": 503368921,
        ""diff_hunk"": ""@@ -0,0 +1,124 @@\n+from github_to_sqlite import utils\n+import pytest\n+import pathlib\n+import sqlite_utils\n+from sqlite_utils.db import ForeignKey\n+import json\n+\n+\n+@pytest.fixture\n+def pull_requests():\n+    return json.load(open(pathlib.Path(__file__).parent / \""pull_requests.json\""))\n+\n+\n+@pytest.fixture\n+def db(pull_requests):\n+    db = sqlite_utils.Database(memory=True)\n+    db[\""repos\""].insert(\n+        {\""id\"": 1},\n+        pk=\""id\"",\n+        columns={\""organization\"": int, \""topics\"": str, \""name\"": str, \""description\"": str},\n+    )\n+    utils.save_pull_requests(db, pull_requests, {\""id\"": 1})\n+    return db\n+\n+\n+def test_tables(db):\n+    assert {\""pull_requests\"", \""users\"", \""repos\"", \""milestones\""} == set(\n+        db.table_names()\n+    )\n+    assert {\n+        ForeignKey(\n+            table=\""pull_requests\"", column=\""repo\"", other_table=\""repos\"", other_column=\""id\""\n+        ),\n+        ForeignKey(\n+            table=\""pull_requests\"",\n+            column=\""milestone\"",\n+            other_table=\""milestones\"",\n+            other_column=\""id\"",\n+        ),\n+        ForeignKey(\n+            table=\""pull_requests\"", column=\""assignee\"", other_table=\""users\"", other_column=\""id\""\n+        ),\n+        ForeignKey(\n+            table=\""pull_requests\"", column=\""user\"", other_table=\""users\"", other_column=\""id\""\n+        ),\n+    } == set(db[\""pull_requests\""].foreign_keys)\n+\n+\n+def test_pull_requests(db):\n+    pull_request_rows = list(db[\""pull_requests\""].rows)\n+    assert [\n+        {\n+            'id': 313384926,"",
        ""path"": ""tests/test_pull_requests.py"",
        ""position"": null,
        ""original_position"": 53,
        ""commit_id"": ""3a0d5c498f9faae4e40aab204cd01b965a4f61f3"",
        ""user"": {
            ""login"": ""simonw"",
            ""id"": 9599,
            ""node_id"": ""MDQ6VXNlcjk1OTk="",
            ""avatar_url"": ""https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/simonw"",
            ""html_url"": ""https://github.com/simonw"",
            ""followers_url"": ""https://api.github.com/users/simonw/followers"",
            ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/simonw/orgs"",
            ""repos_url"": ""https://api.github.com/users/simonw/repos"",
            ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/simonw/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""body"": ""Minor detail: I use Black for this repo, which requires double quotes - running \""black .\"" in the root directory (with the latest version of Black) should handle this for you."",
        ""created_at"": ""2020-10-06T21:28:31Z"",
        ""updated_at"": ""2020-10-20T20:56:33Z"",
        ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606665"",
        ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"",
        ""author_association"": ""MEMBER"",
        ""_links"": {
            ""self"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606665""
            },
            ""html"": {
                ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606665""
            },
            ""pull_request"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48""
            }
        },
        ""original_commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c""
    }
]
```
That's a lot more interesting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",664485022,Feature: pull request reviews and comments,
https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483820,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46,735483820,MDEyOklzc3VlQ29tbWVudDczNTQ4MzgyMA==,9599,simonw,2020-11-30T00:27:47Z,2020-11-30T00:27:47Z,MEMBER,"So it looks like anything that pulls reviews needs to pull each review, then for each one pull the comments.

I'm going to consider this blocked on smarter rate limit handling in #51.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",664485022,Feature: pull request reviews and comments,
https://github.com/dogsheep/github-to-sqlite/issues/47#issuecomment-675523053,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/47,675523053,MDEyOklzc3VlQ29tbWVudDY3NTUyMzA1Mw==,9599,simonw,2020-08-18T14:45:53Z,2020-08-18T14:45:53Z,MEMBER,"```
% github-to-sqlite emojis emojis.db --fetch
  [########----------------------------]  397/1682   23%  00:03:43
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681086659,emojis command,
https://github.com/dogsheep/github-to-sqlite/issues/5#issuecomment-549094229,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5,549094229,MDEyOklzc3VlQ29tbWVudDU0OTA5NDIyOQ==,9599,simonw,2019-11-03T00:44:03Z,2019-11-03T00:44:03Z,MEMBER,Might not need an incomplete boolean - may be possible to handle this with `alter=True` and then by filtering for users with null values in certain columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493671014,"Add ""incomplete"" boolean to users table for incomplete profiles",
https://github.com/dogsheep/github-to-sqlite/issues/5#issuecomment-602806530,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5,602806530,MDEyOklzc3VlQ29tbWVudDYwMjgwNjUzMA==,9599,simonw,2020-03-23T19:23:30Z,2020-03-23T19:23:30Z,MEMBER,Yeah `alter=True` means that this isn't necessary.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493671014,"Add ""incomplete"" boolean to users table for incomplete profiles",
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693773191,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693773191,MDEyOklzc3VlQ29tbWVudDY5Mzc3MzE5MQ==,9599,simonw,2020-09-17T02:39:26Z,2020-09-17T02:39:26Z,MEMBER,I'm going to start with `github-to-sqlite get` and `github-to-sqlite post` - I may add `put` and suchlike later on.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693775622,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693775622,MDEyOklzc3VlQ29tbWVudDY5Mzc3NTYyMg==,9599,simonw,2020-09-17T02:48:34Z,2020-09-17T02:48:34Z,MEMBER,I'd like a `--paginate` option that does the same thing as https://github.com/simonw/paginate-json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693788032,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693788032,MDEyOklzc3VlQ29tbWVudDY5Mzc4ODAzMg==,9599,simonw,2020-09-17T03:35:22Z,2020-09-17T03:35:22Z,MEMBER,Documentation: https://github.com/dogsheep/github-to-sqlite/blob/b02bf135485c0a7a3768868967f45a6b5e515289/README.md#making-authenticated-api-calls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693788387,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693788387,MDEyOklzc3VlQ29tbWVudDY5Mzc4ODM4Nw==,9599,simonw,2020-09-17T03:36:47Z,2020-09-17T03:36:58Z,MEMBER,"Fun demo of the `--nl` option:

    github-to-sqlite get /users/simonw/repos --paginate --nl | sqlite-utils insert simonw.db repos - --nl
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693789129,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693789129,MDEyOklzc3VlQ29tbWVudDY5Mzc4OTEyOQ==,9599,simonw,2020-09-17T03:40:01Z,2020-09-17T03:40:01Z,MEMBER,"Bug with endpoints that return dictionaries rather than arrays:
```
github-to-sqlite get /users/simonw
[
    ""login"",
    ""id"",
    ""node_id"",
    ""avatar_url"",
    ""gravatar_id"",
    ""url"",
    ""html_url"",
    ""followers_url"",
    ""following_url"",
    ""gists_url"",
    ""starred_url"",
    ""subscriptions_url"",
    ""organizations_url"",
    ""repos_url"",
    ""events_url"",
    ""received_events_url"",
    ""type"",
    ""site_admin"",
    ""name"",
    ""company"",
    ""blog"",
    ""location"",
    ""email"",
    ""hireable"",
    ""bio"",
    ""twitter_username"",
    ""public_repos"",
    ""public_gists"",
    ""followers"",
    ""following"",
    ""created_at"",
    ""updated_at""
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693794700,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693794700,MDEyOklzc3VlQ29tbWVudDY5Mzc5NDcwMA==,9599,simonw,2020-09-17T04:02:39Z,2020-09-17T04:02:39Z,MEMBER,It would be useful if you could pass an `--accept` option to this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-711569063,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,711569063,MDEyOklzc3VlQ29tbWVudDcxMTU2OTA2Mw==,9599,simonw,2020-10-19T05:01:29Z,2020-10-19T05:01:29Z,MEMBER,"Demo of `--accept`:

    github-to-sqlite get /repos/simonw/datasette/readme --accept 'application/vnd.github.VERSION.html'
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-1208757153,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51,1208757153,IC_kwDODFdgUs5IDCuh,9020979,hydrosquall,2022-08-09T00:29:44Z,2022-08-09T00:29:44Z,NONE,"I've been looking into how to to get this data out of Github (especially now there are ""secondary rate limits"" without an advertised allowance separate from the regular rate limits. 

I've had decent success with the Airbyte github extractor (aside from one data quality issue https://github.com/airbytehq/airbyte/pull/15420 ). Airbyte splits data extraction between the GraphQL and REST endpoints depending on the resource type, but they're very comprehensive. 

https://github.com/airbytehq/airbyte/blob/306a75ef5370728e0912cf52a1a898a530db0c90/airbyte-integrations/connectors/source-github/source_github/streams.py#L22-L122

Before this, I tried a few solutions in my own custom wrapper mentioned in this thread + its children https://github.com/PyGithub/PyGithub/issues/1989 , but they weren't working as expected.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703246031,github-to-sqlite should handle rate limits better,
https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-1279224780,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51,1279224780,IC_kwDODFdgUs5MP2vM,7908073,chapmanjacobd,2022-10-14T16:34:07Z,2022-10-14T16:34:07Z,NONE,"also, it says that authenticated requests have a much higher ""rate limit"". Unauthenticated requests only get 60 req/hour ?? seems more like a quota than a ""rate limit"" (although I guess that is semantic equivalence)

You would want to use `x-ratelimit-reset`

```
time.sleep(r['x-ratelimit-reset'] + 1 - time.time())
```

But a more complete solution would bring authenticated requests to the other subcommands. I'm surprised only `github-to-sqlite get` is using the `--auth=` CLI flag","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703246031,github-to-sqlite should handle rate limits better,
https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-735484186,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51,735484186,MDEyOklzc3VlQ29tbWVudDczNTQ4NDE4Ng==,9599,simonw,2020-11-30T00:29:31Z,2020-11-30T00:29:31Z,MEMBER,"This just caused a failure in deploying the demo: https://github.com/dogsheep/github-to-sqlite/runs/1471304407?check_suite_focus=true

```
  File ""/opt/hostedtoolcache/Python/3.8.6/x64/bin/github-to-sqlite"", line 33, in <module>
    sys.exit(load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')())
  File ""/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 142, in issue_comments
    for comment in utils.fetch_issue_comments(repo, token, issue):
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 380, in fetch_issue_comments
    for comments in paginate(url, headers):
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 472, in paginate
    raise GitHubError.from_response(response)
github_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403)
Error: Process completed with exit code 1.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703246031,github-to-sqlite should handle rate limits better,
https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-770150526,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51,770150526,MDEyOklzc3VlQ29tbWVudDc3MDE1MDUyNg==,22578954,daniel-butler,2021-01-30T03:44:19Z,2021-01-30T03:47:24Z,CONTRIBUTOR,I don't have much experience with github's rate limiting. In my day job we use the [tenacity library](https://github.com/jd/tenacity) to handle http errors we get.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703246031,github-to-sqlite should handle rate limits better,
https://github.com/dogsheep/github-to-sqlite/issues/53#issuecomment-735485677,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/53,735485677,MDEyOklzc3VlQ29tbWVudDczNTQ4NTY3Nw==,9599,simonw,2020-11-30T00:36:09Z,2020-11-30T00:36:09Z,MEMBER,Given rate limits (see #51) this command might be better implemented by running a `git clone` into a temporary directory - doing so would retrieve all of the files in one go.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753000405,Command for fetching file contents,
https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735464438,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54,735464438,MDEyOklzc3VlQ29tbWVudDczNTQ2NDQzOA==,9599,simonw,2020-11-29T21:57:08Z,2020-11-29T21:57:08Z,MEMBER,Inspired by this tweet from Michael Heap https://twitter.com/mheap/status/1333108608817631238,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753026003,github-to-sqlite workflows command,
https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735464493,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54,735464493,MDEyOklzc3VlQ29tbWVudDczNTQ2NDQ5Mw==,9599,simonw,2020-11-29T21:57:32Z,2020-11-29T21:57:32Z,MEMBER,`$ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753026003,github-to-sqlite workflows command,
https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735465708,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54,735465708,MDEyOklzc3VlQ29tbWVudDczNTQ2NTcwOA==,9599,simonw,2020-11-29T22:08:46Z,2020-11-29T22:08:46Z,MEMBER,"Demo:
- https://github-to-sqlite.dogsheep.net/github/steps?_facet=repo
- https://github-to-sqlite.dogsheep.net/github/workflows
- https://github-to-sqlite.dogsheep.net/github/jobs","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753026003,github-to-sqlite workflows command,
https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746734412,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58,746734412,MDEyOklzc3VlQ29tbWVudDc0NjczNDQxMg==,9599,simonw,2020-12-16T17:58:56Z,2020-12-16T17:58:56Z,MEMBER,"I'm going to rewrite those `<a href=""#filtering-tables"">` links to `<a href=""#user-content-filtering-tables"">` - but only if a corresponding `id=""user-content-filtering-tables""` element exists.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769150394,Readme HTML has broken internal links,
https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746735889,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58,746735889,MDEyOklzc3VlQ29tbWVudDc0NjczNTg4OQ==,9599,simonw,2020-12-16T17:59:50Z,2020-12-16T17:59:50Z,MEMBER,"I don't want to add a full HTML parser (like BeautifulSoup) as a dependency for this feature. Since the HTML comes from a single, trusted source (GitHub) I could probably handle this using [regular expressions](https://stackoverflow.com/a/1732454).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769150394,Readme HTML has broken internal links,
https://github.com/dogsheep/github-to-sqlite/issues/6#issuecomment-541387822,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6,541387822,MDEyOklzc3VlQ29tbWVudDU0MTM4NzgyMg==,9599,simonw,2019-10-13T05:27:39Z,2019-10-13T05:27:39Z,MEMBER,This should be fixed by https://github.com/dogsheep/github-to-sqlite/commit/552543a74970f8a3a3f87f887be23a0c6eb1cb5b,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",504238461,sqlite3.OperationalError: table users has no column named bio,
https://github.com/dogsheep/github-to-sqlite/issues/6#issuecomment-541387941,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6,541387941,MDEyOklzc3VlQ29tbWVudDU0MTM4Nzk0MQ==,9599,simonw,2019-10-13T05:30:19Z,2019-10-13T05:30:19Z,MEMBER,Fix released in 0.5: https://github.com/dogsheep/github-to-sqlite/releases/tag/0.5,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",504238461,sqlite3.OperationalError: table users has no column named bio,
https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770069864,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60,770069864,MDEyOklzc3VlQ29tbWVudDc3MDA2OTg2NA==,22578954,daniel-butler,2021-01-29T21:52:05Z,2021-02-12T18:29:43Z,CONTRIBUTOR,"For the purposes below I am assuming the organization I would get all the repositories and their related commits from is called `gh-organization`. The github's owner id of gh-orgnization is `123456789`.

```bash
github-to-sqlite  repos github.db gh-organization
```

I'm on a windows computer running git bash to be able to use the `|` command. This works for me
```bash
sqlite3 github.db ""SELECT full_name FROM repos WHERE owner = '123456789';"" | tr '\n\r' ' ' | xargs | { read repos; github-to-sqlite commits github.db $repos; }
```

On a pure linux system I think this would work because the new line character is normally `\n`
```bash
sqlite3 github.db ""SELECT full_name FROM repos WHERE owner = '123456789';"" | tr '\n' ' ' | xargs | { read repos; github-to-sqlite commits github.db $repos; }`
```

As expected I ran into rate limit issues #51 
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797097140,Use Data from SQLite in other commands,
https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770071568,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60,770071568,MDEyOklzc3VlQ29tbWVudDc3MDA3MTU2OA==,9599,simonw,2021-01-29T21:56:15Z,2021-01-29T21:56:15Z,MEMBER,"I really like the way you're using pipes here - really smart. It's similar to how I build the demo database in this GitHub Actions workflow:

https://github.com/dogsheep/github-to-sqlite/blob/62dfd3bc4014b108200001ef4bc746feb6f33b45/.github/workflows/deploy-demo.yml#L52-L82

`twitter-to-sqlite` actually has a mechanism for doing this kind of thing, documented at https://github.com/dogsheep/twitter-to-sqlite#providing-input-from-a-sql-query-with---sql-and---attach

It lets you do things like:

```
$ twitter-to-sqlite users-lookup my.db --sql=""select follower_id from following"" --ids
```
Maybe I should add something similar to `github-to-sqlite`? Feels like it could be really useful.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797097140,Use Data from SQLite in other commands,
https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770112248,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60,770112248,MDEyOklzc3VlQ29tbWVudDc3MDExMjI0OA==,22578954,daniel-butler,2021-01-30T00:01:03Z,2021-01-30T01:14:42Z,CONTRIBUTOR,"Yes that would be cool! I wouldn't mind helping. Is this the meat of it? https://github.com/dogsheep/twitter-to-sqlite/blob/21fc1cad6dd6348c67acff90a785b458d3a81275/twitter_to_sqlite/utils.py#L512

It looks like the cli option is added with this decorator : https://github.com/dogsheep/twitter-to-sqlite/blob/21fc1cad6dd6348c67acff90a785b458d3a81275/twitter_to_sqlite/cli.py#L14

I looked a bit at utils.py in the GitHub repository. I was surprised at the amount of manual mapping of the API response you had to do to get this to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797097140,Use Data from SQLite in other commands,
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-860895838,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,860895838,MDEyOklzc3VlQ29tbWVudDg2MDg5NTgzOA==,231498,khimaros,2021-06-14T18:23:21Z,2021-06-14T21:37:35Z,NONE,"i have a basic working version at https://github.com/khimaros/github-to-sqlite

this can be tested with `github-to-sqlite events.db khimaros/events`

caveat: the GitHub API doesn't seem to provide a complete history of events.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861035862,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,861035862,MDEyOklzc3VlQ29tbWVudDg2MTAzNTg2Mg==,231498,khimaros,2021-06-14T22:29:20Z,2021-06-14T22:29:20Z,NONE,"it looks like the v4 GraphQL API is the only way to get data beyond 90 days from GitHub.

this is significant change, but may be worth considering in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861041597,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,861041597,MDEyOklzc3VlQ29tbWVudDg2MTA0MTU5Nw==,9599,simonw,2021-06-14T22:44:54Z,2021-06-14T22:44:54Z,MEMBER,Have you found a way to access events in GraphQL? I can only see way to access a timeline of events for a single issue or a single pull request. See also https://github.community/t/get-event-equivalent-for-v4/13600/2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861042050,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,861042050,MDEyOklzc3VlQ29tbWVudDg2MTA0MjA1MA==,9599,simonw,2021-06-14T22:45:42Z,2021-06-14T22:45:42Z,MEMBER,I'm definitely interested in supporting events in this tool - see #14.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861087651,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,861087651,MDEyOklzc3VlQ29tbWVudDg2MTA4NzY1MQ==,231498,khimaros,2021-06-15T00:48:37Z,2021-06-15T00:48:37Z,NONE,"@simonw -- i've created an omega-query that fetched most of what was interesting to me for a single user.

found by poking around in the ""Explorer"" tab in https://docs.github.com/en/graphql/overview/explorer

note: pagination is still required via `first` and `last` but it seems to allow unlimited history.

```
query MyQuery {
  __typename
  user(login: ""<user>"") {
    id
    pinnedItems(first: 100) {
      edges {
        node
      }
    }
    pullRequests(first: 100) {
      nodes {
        body
        title
        state
        createdAt
      }
    }
    createdAt
    issues(first: 100) {
      pageInfo {
        endCursor
        startCursor
      }
      nodes {
        title
        url
        createdAt
        body
      }
    }
    issueComments(first: 100) {
      edges {
        node {
          id
          updatedAt
          url
          body
        }
      }
    }
    repositories(first: 100) {
      nodes {
        createdAt
        description
        parent {
          name
        }
        pinnedIssues(first: 100) {
          edges {
            node {
              id
            }
          }
        }
        pinnedDiscussions(first: 100) {
          edges {
            node {
              id
            }
          }
        }
      }
    }
    starredRepositories(first: 100) {
      edges {
        node {
          id
        }
      }
    }
  }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/dogsheep/github-to-sqlite/issues/69#issuecomment-985928838,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69,985928838,IC_kwDODFdgUs46xBSG,9599,simonw,2021-12-04T00:34:52Z,2021-12-04T00:34:52Z,MEMBER,"First attempt at this:
```sql
select 'issues' as ""table"", id, node_id, title, user, created_at, body, repo
from issues
union all
select 'issue_comments' as ""table"", issue_comments.id, issue_comments.node_id, '' as title, issue_comments.user, issue_comments.created_at, issue_comments.body, issues.repo
from issue_comments join issues on issues.id = issue_comments.issue
order by created_at desc
```
https://github-to-sqlite.dogsheep.net/github?sql=select+%27issues%27+as+%22table%22%2C+id%2C+node_id%2C+title%2C+user%2C+created_at%2C+body%2C+repo%0D%0Afrom+issues%0D%0Aunion+all%0D%0Aselect+%27issue_comments%27+as+%22table%22%2C+issue_comments.id%2C+issue_comments.node_id%2C+%27%27+as+title%2C+issue_comments.user%2C+issue_comments.created_at%2C+issue_comments.body%2C+issues.repo%0D%0Afrom+issue_comments+join+issues+on+issues.id+%3D+issue_comments.issue%0D%0Aorder+by+created_at+desc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1071071397,View that combines issues and issue comments,
https://github.com/dogsheep/github-to-sqlite/issues/7#issuecomment-541721437,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/7,541721437,MDEyOklzc3VlQ29tbWVudDU0MTcyMTQzNw==,9599,simonw,2019-10-14T14:44:12Z,2019-10-14T14:44:12Z,MEMBER,Docs: https://github.com/dogsheep/github-to-sqlite/blob/0.5/README.md#retrieving-issue-comments-for-a-repository,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506276893,issue-comments command for importing issue comments,
https://github.com/dogsheep/github-to-sqlite/issues/72#issuecomment-1105474232,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/72,1105474232,IC_kwDODFdgUs5B5DK4,9599,simonw,2022-04-21T17:02:15Z,2022-04-21T17:02:15Z,MEMBER,"That's interesting - yeah it looks like the number of pages can be derived from the `Link` header, which is enough information to show a progress bar, probably using Click just to avoid adding another dependency.

https://docs.github.com/en/rest/guides/traversing-with-pagination","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1211283427,feature: display progress bar when downloading multi-page responses,
https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188223933,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74,1188223933,IC_kwDODFdgUs5G0tu9,9599,simonw,2022-07-18T19:40:50Z,2022-07-18T19:42:41Z,MEMBER,"Here's how the demo is deployed: https://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/.github/workflows/deploy-demo.yml#L103-L119

I'm suspicious of `py-gfm`, which is used like this:

https://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/demo-metadata.json#L49-L51","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1308461063,500 error in github-to-sqlite demo,
https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188225625,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74,1188225625,IC_kwDODFdgUs5G0uJZ,9599,simonw,2022-07-18T19:41:52Z,2022-07-18T19:41:52Z,MEMBER,"https://github.com/Zopieux/py-gfm says that library is no longer maintained, and suggests https://github.com/Zopieux/pycmarkgfm as an alternative.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1308461063,500 error in github-to-sqlite demo,
https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188228964,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74,1188228964,IC_kwDODFdgUs5G0u9k,9599,simonw,2022-07-18T19:45:30Z,2022-07-18T19:47:35Z,MEMBER,"`pycmarkgfm` doesn't implement the Markdown plugin extension I was using.

I'm going to drop the GFM rendering from the demo, and just treat it as regular markdown.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1308461063,500 error in github-to-sqlite demo,
https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188233729,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74,1188233729,IC_kwDODFdgUs5G0wIB,9599,simonw,2022-07-18T19:51:02Z,2022-07-18T19:51:02Z,MEMBER,Takes 30m to deploy the demo!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1308461063,500 error in github-to-sqlite demo,
https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188317682,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74,1188317682,IC_kwDODFdgUs5G1Eny,9599,simonw,2022-07-18T21:14:22Z,2022-07-18T21:14:22Z,MEMBER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1308461063,500 error in github-to-sqlite demo,
https://github.com/dogsheep/github-to-sqlite/issues/79#issuecomment-1498167714,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79,1498167714,IC_kwDODFdgUs5ZTDmi,9599,simonw,2023-04-05T21:12:55Z,2023-04-05T21:12:55Z,MEMBER,"It's a rate limiting problem: https://github.com/dogsheep/github-to-sqlite/actions/runs/4622674009/jobs/8175633155

```
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 407, in commits
    utils.save_commits(db, commits, repo_full[""id""])
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 635, in save_commits
    for commit in commits:
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 417, in fetch_commits
    for commits in paginate(url, headers):
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 478, in paginate
    raise GitHubError.from_response(response)
github_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403)
Error: Process completed with exit code 1.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1570375808,Deploy demo job is failing due to rate limit,
https://github.com/dogsheep/github-to-sqlite/pull/48#issuecomment-704503719,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48,704503719,MDEyOklzc3VlQ29tbWVudDcwNDUwMzcxOQ==,755825,adamjonas,2020-10-06T19:26:59Z,2020-10-06T19:26:59Z,CONTRIBUTOR,ref #46 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681228542,Add pull requests,
https://github.com/dogsheep/github-to-sqlite/pull/48#issuecomment-704553385,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48,704553385,MDEyOklzc3VlQ29tbWVudDcwNDU1MzM4NQ==,9599,simonw,2020-10-06T21:07:44Z,2020-10-06T21:07:44Z,MEMBER,"Sorry for not looking at this sooner, trying it out now - pull request looks great!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681228542,Add pull requests,
https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-751375487,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59,751375487,MDEyOklzc3VlQ29tbWVudDc1MTM3NTQ4Nw==,631242,frosencrantz,2020-12-26T17:08:44Z,2020-12-26T17:08:44Z,CONTRIBUTOR,"Hi @simonw,  do I need to do anything else for this PR to be considered to be included? I've tried using this project and it is quite nice to be able to explore a repository, but noticed that a couple commands don't allow you to use authorization from the environment variable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771872303,Remove unneeded exists=True for -a/--auth flag.,
https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-844250232,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59,844250232,MDEyOklzc3VlQ29tbWVudDg0NDI1MDIzMg==,9599,simonw,2021-05-19T16:08:10Z,2021-05-19T16:08:10Z,MEMBER,Thanks for catching this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771872303,Remove unneeded exists=True for -a/--auth flag.,
https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-846413174,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59,846413174,MDEyOklzc3VlQ29tbWVudDg0NjQxMzE3NA==,631242,frosencrantz,2021-05-22T14:06:19Z,2021-05-22T14:06:19Z,CONTRIBUTOR,Thanks Simon!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771872303,Remove unneeded exists=True for -a/--auth flag.,
https://github.com/dogsheep/github-to-sqlite/pull/61#issuecomment-844249385,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/61,844249385,MDEyOklzc3VlQ29tbWVudDg0NDI0OTM4NQ==,9599,simonw,2021-05-19T16:07:06Z,2021-05-19T16:07:06Z,MEMBER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797108702,fixing typo in get cli help text,
https://github.com/dogsheep/github-to-sqlite/pull/65#issuecomment-1266141699,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65,1266141699,IC_kwDODFdgUs5Ld8oD,231498,khimaros,2022-10-03T22:35:03Z,2022-10-03T22:35:03Z,NONE,"@simonw rebased against latest, please let me know if i should drop this PR.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923270900,basic support for events,
https://github.com/dogsheep/github-to-sqlite/pull/65#issuecomment-885964242,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65,885964242,IC_kwDODFdgUs40zr3S,231498,khimaros,2021-07-23T23:45:35Z,2021-07-23T23:45:35Z,NONE,@simonw is this PR of interest to you?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923270900,basic support for events,
https://github.com/dogsheep/github-to-sqlite/pull/66#issuecomment-929651819,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/66,929651819,IC_kwDODFdgUs43aVxr,30531572,sarcasticadmin,2021-09-28T21:50:31Z,2021-09-28T21:50:31Z,NONE,@simonw any feedback/thoughts? ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",975161924,Add --merged-by flag to pull-requests sub command,
https://github.com/dogsheep/github-to-sqlite/pull/73#issuecomment-1188223299,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/73,1188223299,IC_kwDODFdgUs5G0tlD,9599,simonw,2022-07-18T19:40:06Z,2022-07-18T19:40:06Z,MEMBER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1261884917,Fixing 'NoneType' object has no attribute 'items',
https://github.com/dogsheep/github-to-sqlite/pull/76#issuecomment-1238190601,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/76,1238190601,IC_kwDODFdgUs5JzUoJ,2757699,OverkillGuy,2022-09-06T13:58:20Z,2022-09-06T13:59:08Z,NONE,"Tested PR just now in private org, fetched >2k repos infos flawlessly!

    poetry run github-to-sqlite repos --organization github.db MYORG","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363280254,Add organization support to repos command,
https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549094195,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8,549094195,MDEyOklzc3VlQ29tbWVudDU0OTA5NDE5NQ==,9599,simonw,2019-11-03T00:43:16Z,2019-11-03T00:43:28Z,MEMBER,"Also need to take #5 into account - if this command creates incomplete user records, how do we repair them?

And make sure that if we run this command first any future commands that populate users don't break (probably just a case of using `alter=True` in a few places).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516763727,"stargazers command, refs #4",
https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549230583,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8,549230583,MDEyOklzc3VlQ29tbWVudDU0OTIzMDU4Mw==,9599,simonw,2019-11-04T05:49:26Z,2019-11-04T05:49:26Z,MEMBER,Adding the view from #10 would be useful here too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516763727,"stargazers command, refs #4",
https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549233778,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8,549233778,MDEyOklzc3VlQ29tbWVudDU0OTIzMzc3OA==,9599,simonw,2019-11-04T06:14:40Z,2019-11-04T06:14:40Z,MEMBER,"Spotted a tricky problem: running `github-to-sqlite starred stargazers.db` results in an incomplete `simonw` record. It creates a proper record for me thanks to this bit:

https://github.com/dogsheep/github-to-sqlite/blob/ea07274667a08c67907e8bfbbccb6f0fb95ce817/github_to_sqlite/cli.py#L120-L126

But then... when it gets to the `datasette` repository which I have starred it over-writes my full user record with one that's missing most of the details, thanks to this bit:

https://github.com/dogsheep/github-to-sqlite/blob/ea07274667a08c67907e8bfbbccb6f0fb95ce817/github_to_sqlite/utils.py#L117-L124

I need to find a way of NOT over-writing a good record with a thinner one.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516763727,"stargazers command, refs #4",
https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-594154644,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8,594154644,MDEyOklzc3VlQ29tbWVudDU5NDE1NDY0NA==,9599,simonw,2020-03-03T20:33:57Z,2020-03-03T20:33:57Z,MEMBER,`sqlite-utils` supports proper upserts now so this problem should be easy to fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516763727,"stargazers command, refs #4",
https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-623006004,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8,623006004,MDEyOklzc3VlQ29tbWVudDYyMzAwNjAwNA==,9599,simonw,2020-05-02T20:00:26Z,2020-05-02T20:00:26Z,MEMBER,I'm abandoning this in favour of a new implementation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516763727,"stargazers command, refs #4",
https://github.com/dogsheep/google-takeout-to-sqlite/issues/10#issuecomment-1073152522,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/10,1073152522,IC_kwDODFE5qs4_9wIK,9290214,csusanu,2022-03-20T02:38:07Z,2022-03-20T02:38:07Z,NONE,"[This line](https://github.com/dogsheep/google-takeout-to-sqlite/blob/e54e544427f1cc3ea8189f0e95f54046301a8645/google_takeout_to_sqlite/utils.py) needs to say `""MyActivity.json""` instead of `""My Activity.json""`. Google must have changed the file name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1123393829,sqlite3.OperationalError: no such table: main.my_activity,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/2#issuecomment-747126777,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2,747126777,MDEyOklzc3VlQ29tbWVudDc0NzEyNjc3Nw==,9599,simonw,2020-12-17T00:36:52Z,2020-12-17T00:36:52Z,MEMBER,The memory profiler tricks I used in https://github.com/dogsheep/healthkit-to-sqlite/issues/7 could help figure out what's going on here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769376447,killed by oomkiller on large location-history,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/2#issuecomment-747130908,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2,747130908,MDEyOklzc3VlQ29tbWVudDc0NzEzMDkwOA==,231498,khimaros,2020-12-17T00:47:04Z,2020-12-17T00:47:43Z,NONE,"it looks like almost all of the memory consumption is coming from `json.load()`.

another direction here may be to use the new ""Semantic Location History"" data which is already broken down by year and month.

it also provides much more interesting data, such as estimated address, form of travel, etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769376447,killed by oomkiller on large location-history,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-780817596,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4,780817596,MDEyOklzc3VlQ29tbWVudDc4MDgxNzU5Ng==,306240,UtahDave,2021-02-17T20:01:35Z,2021-02-17T20:01:35Z,NONE,I've got this almost working. Just needs some polish,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778380836,Feature Request: Gmail,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-781451701,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4,781451701,MDEyOklzc3VlQ29tbWVudDc4MTQ1MTcwMQ==,203343,Btibert3,2021-02-18T16:06:21Z,2021-02-18T16:06:21Z,NONE,Awesome!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778380836,Feature Request: Gmail,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-783688547,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4,783688547,MDEyOklzc3VlQ29tbWVudDc4MzY4ODU0Nw==,306240,UtahDave,2021-02-22T21:31:28Z,2021-02-22T21:31:28Z,NONE,"@Btibert3 I've opened a PR with my initial attempt at this.  Would you be willing to give this a try?

https://github.com/dogsheep/google-takeout-to-sqlite/pull/5","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778380836,Feature Request: Gmail,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790198930,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4,790198930,MDEyOklzc3VlQ29tbWVudDc5MDE5ODkzMA==,203343,Btibert3,2021-03-04T00:58:40Z,2021-03-04T00:58:40Z,NONE,"I am just seeing this sorry, yes!  I will kick the tires later on tonight.  My apologies for the delay.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778380836,Feature Request: Gmail,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790934616,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4,790934616,MDEyOklzc3VlQ29tbWVudDc5MDkzNDYxNg==,203343,Btibert3,2021-03-04T20:54:44Z,2021-03-04T20:54:44Z,NONE,"Sorry for the delay, I got sidetracked after class last night.  I am getting the following error:

```
/content# google-takeout-to-sqlite mbox takeout.db Takeout/Mail/gmail.mbox 
Usage: google-takeout-to-sqlite [OPTIONS] COMMAND [ARGS]...Try 'google-takeout-to-sqlite --help' for help.

Error: No such command 'mbox'.
```

On the box, I installed with pip after cloning: https://github.com/UtahDave/google-takeout-to-sqlite.git","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778380836,Feature Request: Gmail,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/6#issuecomment-790384087,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6,790384087,MDEyOklzc3VlQ29tbWVudDc5MDM4NDA4Nw==,9599,simonw,2021-03-04T07:22:51Z,2021-03-04T07:22:51Z,MEMBER,#3 also mentions the conflicting version with other tools.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",821841046,Upgrade to latest sqlite-utils,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-783794520,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,783794520,MDEyOklzc3VlQ29tbWVudDc4Mzc5NDUyMA==,306240,UtahDave,2021-02-23T01:13:54Z,2021-02-23T01:13:54Z,NONE,"Also, @simonw I created a test based off the existing tests. I think it's working correctly","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-784638394,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,784638394,MDEyOklzc3VlQ29tbWVudDc4NDYzODM5NA==,306240,UtahDave,2021-02-24T00:36:18Z,2021-02-24T00:36:18Z,NONE,I noticed that @simonw is using black for formatting. I ran black on my additions in this PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-786925280,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,786925280,MDEyOklzc3VlQ29tbWVudDc4NjkyNTI4MA==,9599,simonw,2021-02-26T22:23:10Z,2021-02-26T22:23:10Z,MEMBER,"Thanks!

I requested my Gmail export from takeout - once that arrives I'll test it against this and then merge the PR.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790312268,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790312268,MDEyOklzc3VlQ29tbWVudDc5MDMxMjI2OA==,9599,simonw,2021-03-04T05:48:16Z,2021-03-04T05:48:16Z,MEMBER,"Wow, my mbox is a 10.35 GB download!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790369076,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790369076,MDEyOklzc3VlQ29tbWVudDc5MDM2OTA3Ng==,9599,simonw,2021-03-04T06:54:46Z,2021-03-04T06:54:46Z,MEMBER,"The Rich-powered progress bar is pretty:

![rich](https://user-images.githubusercontent.com/9599/109923307-71f69200-7c73-11eb-9ee2-8f0a240f3994.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790370485,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790370485,MDEyOklzc3VlQ29tbWVudDc5MDM3MDQ4NQ==,9599,simonw,2021-03-04T06:57:25Z,2021-03-04T06:57:48Z,MEMBER,"The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count:

https://github.com/dogsheep/google-takeout-to-sqlite/blob/a3de045eba0fae4b309da21aa3119102b0efc576/google_takeout_to_sqlite/utils.py#L66-L67

I'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790372621,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790372621,MDEyOklzc3VlQ29tbWVudDc5MDM3MjYyMQ==,9599,simonw,2021-03-04T07:01:18Z,2021-03-04T07:01:18Z,MEMBER,"I'm not sure if it would work, but there is an alternative pattern for showing a progress bar against a really large file that I've used in `healthkit-to-sqlite` - you set the progress bar size to the size of the file in bytes, then update a counter as you read the file.

https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/cli.py#L24-L57 and https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/utils.py#L4-L19 (the `progress_callback()` bit) is where that happens.

It can be a bit of a convoluted pattern, and I'm not at all sure it would work for `mbox` files since it looks like that library has other reasons it needs to do a file scan rather than streaming it through one chunk of bytes at a time. So I imagine this would not work here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790373024,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790373024,MDEyOklzc3VlQ29tbWVudDc5MDM3MzAyNA==,9599,simonw,2021-03-04T07:01:58Z,2021-03-04T07:04:06Z,MEMBER,"I got 9 warnings that look like this:
```
Errors: 1
Traceback (most recent call last):
  File ""/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py"", line 103, in get_mbox
    message[""date""] = get_message_date(email.get(""Date""), email.get_from())
  File ""/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py"", line 167, in get_message_date
    datetime_tuple = email.utils.parsedate_tz(mail_date)
  File ""/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py"", line 50, in parsedate_tz
    res = _parsedate_tz(data)
  File ""/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py"", line 69, in _parsedate_tz
    data = data.split()
AttributeError: 'Header' object has no attribute 'split'
```
It would be useful if those warnings told me the message ID (or similar) of the affected message so I could grep for it in the `mbox` and see what was going on.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790378658,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790378658,MDEyOklzc3VlQ29tbWVudDc5MDM3ODY1OA==,9599,simonw,2021-03-04T07:12:48Z,2021-03-04T07:12:48Z,MEMBER,"It looks like the `body` is being loaded into a BLOB column - so in Datasette default it looks like this:

<img width=""1650"" alt=""mbox__mbox_emails__753_446_rows"" src=""https://user-images.githubusercontent.com/9599/109924808-b4b96980-7c75-11eb-8c9e-307f2ae32d5a.png"">

If I `datasette install datasette-render-binary` and then try again I get this:

<img width=""1487"" alt=""mbox__mbox_emails__753_446_rows"" src=""https://user-images.githubusercontent.com/9599/109924944-ea5e5280-7c75-11eb-9a32-404f3d68455f.png"">

It would be great if we could store the `body` as unicode text instead. May have to do something clever to decode it based on some kind of charset header?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790379629,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790379629,MDEyOklzc3VlQ29tbWVudDc5MDM3OTYyOQ==,9599,simonw,2021-03-04T07:14:41Z,2021-03-04T07:14:41Z,MEMBER,"Confirmed: removing the `len()` call does not speed things up, so it's reading through the entire file for some other purpose too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790380839,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790380839,MDEyOklzc3VlQ29tbWVudDc5MDM4MDgzOQ==,9599,simonw,2021-03-04T07:17:05Z,2021-03-04T07:17:05Z,MEMBER,"Looks like you're doing this:
```python
    elif message.get_content_type() == ""text/plain"":
        body = message.get_payload(decode=True)
```
So presumably that decodes to a unicode string?

I imagine the reason the column is a `BLOB` for me is that `sqlite-utils` determines the column type based on the first batch of items - https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1927-L1928 - and I got unlucky and had something in my first batch that wasn't a unicode string.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790389335,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790389335,MDEyOklzc3VlQ29tbWVudDc5MDM4OTMzNQ==,306240,UtahDave,2021-03-04T07:32:04Z,2021-03-04T07:32:04Z,NONE,"> The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count:
> 
> https://github.com/dogsheep/google-takeout-to-sqlite/blob/a3de045eba0fae4b309da21aa3119102b0efc576/google_takeout_to_sqlite/utils.py#L66-L67
> 
> I'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long.

The wait is from python loading the mbox file. This happens regardless if you're getting the length of the mbox.  The mbox module is on the slow side. It is possible to do one's own parsing of the mbox, but I kind of wanted to avoid doing that.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790391711,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790391711,MDEyOklzc3VlQ29tbWVudDc5MDM5MTcxMQ==,306240,UtahDave,2021-03-04T07:36:24Z,2021-03-04T07:36:24Z,NONE,"> Looks like you're doing this:
> 
> ```python
>     elif message.get_content_type() == ""text/plain"":
>         body = message.get_payload(decode=True)
> ```
> 
> So presumably that decodes to a unicode string?
> 
> I imagine the reason the column is a `BLOB` for me is that `sqlite-utils` determines the column type based on the first batch of items - https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1927-L1928 - and I got unlucky and had something in my first batch that wasn't a unicode string.

Ah, that's good to know. I think explicitly creating the tables will be a great improvement. I'll add that.

Also, I noticed after I opened this PR that the `message.get_payload()` is being deprecated in favor of `message.get_content()` or something like that.  I'll see if that handles the decoding better, too.

Thanks for the feedback. I should have time tomorrow to put together some improvements.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790668263,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790668263,MDEyOklzc3VlQ29tbWVudDc5MDY2ODI2Mw==,9599,simonw,2021-03-04T14:43:58Z,2021-03-04T14:43:58Z,MEMBER,"I added this code to output a message ID on errors:
```diff
             print(""Errors: {}"".format(num_errors))
             print(traceback.format_exc())
+            print(""Message-Id: {}"".format(email.get(""Message-Id"", ""None"")))
             continue
```
Having found a message ID that had an error, I ran this command to see the context:

    rg --text --context 20 '44F289B0.000001.02100@SCHWARZE-DWFXMI' ~/gmail.mbox

This was for the following error:
```
  File ""/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py"", line 102, in get_mbox
    message[""date""] = get_message_date(email.get(""Date""), email.get_from())
  File ""/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py"", line 178, in get_message_date
    datetime_tuple = email.utils.parsedate_tz(mail_date)
  File ""/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py"", line 50, in parsedate_tz
    res = _parsedate_tz(data)
  File ""/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py"", line 69, in _parsedate_tz
    data = data.split()
AttributeError: 'Header' object has no attribute 'split'
```
Here's what I spotted in the `ripgrep` output:
```
177133570:Message-Id: <44F289B0.000001.02100@SCHWARZE-DWFXMI>
177133571-Date: Mon, 28 Aug 2006 08:14:08 +0200 (Westeurop�ische Sommerzeit)
177133572-X-Mailer: IncrediMail (5002253)
```
So it could it be that `_parsedate_tz` is having trouble with that `Mon, 28 Aug 2006 08:14:08 +0200 (Westeurop�ische Sommerzeit)` string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790669767,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790669767,MDEyOklzc3VlQ29tbWVudDc5MDY2OTc2Nw==,9599,simonw,2021-03-04T14:46:06Z,2021-03-04T14:46:06Z,MEMBER,"Solution could be to pre-process that string by splitting on `(` and dropping everything afterwards, assuming that the `(...)` bit isn't necessary for correctly parsing the date.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790693674,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790693674,MDEyOklzc3VlQ29tbWVudDc5MDY5MzY3NA==,9599,simonw,2021-03-04T15:18:36Z,2021-03-04T15:18:36Z,MEMBER,"I imported my 10GB mbox with 750,000 emails in it, ran this tool (with a hacked fix for the blob column problem) - and now a search that returns 92 results takes 25.37ms! This is fantastic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790695126,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790695126,MDEyOklzc3VlQ29tbWVudDc5MDY5NTEyNg==,9599,simonw,2021-03-04T15:20:42Z,2021-03-04T15:20:42Z,MEMBER,"I'm not sure why but my most recent import, when displayed in Datasette, looks like this:

<img width=""574"" alt=""mbox__mbox_emails__753_446_rows"" src=""https://user-images.githubusercontent.com/9599/109985836-0ab00080-7cba-11eb-97d5-0631a0835b61.png"">

Sorting by `id` in the opposite order gives me the data I would expect - so it looks like a bunch of null/blank messages are being imported at some point and showing up first due to ID ordering.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791089881,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,791089881,MDEyOklzc3VlQ29tbWVudDc5MTA4OTg4MQ==,28565,maxhawkins,2021-03-05T02:03:19Z,2021-03-05T02:03:19Z,NONE,"I just tried to run this on a small VPS instance with 2GB of memory and it crashed out of memory while processing a 12GB mbox from Takeout.

Is it possible to stream the emails to sqlite instead of loading it all into memory and upserting at once?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791530093,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,791530093,MDEyOklzc3VlQ29tbWVudDc5MTUzMDA5Mw==,306240,UtahDave,2021-03-05T16:28:07Z,2021-03-05T16:28:07Z,NONE,"> I just tried to run this on a small VPS instance with 2GB of memory and it crashed out of memory while processing a 12GB mbox from Takeout.
> 
> Is it possible to stream the emails to sqlite instead of loading it all into memory and upserting at once?

@maxhawkins a limitation of the python mbox module is it loads the entire mbox into memory.  I did find another approach to this problem that didn't use the builtin python mbox module and created a generator so that it didn't have to load the whole mbox into memory.  I was hoping to use standard library modules, but this might be a good reason to investigate that approach a bit more. My worry is making sure a custom processor handles all the ins and outs of the mbox format correctly.

Hm. As I'm writing this, I thought of something.  I think I can parse each message one at a time, and then use an mbox function to load each message using the python mbox module. That way the mbox module can still deal with the specifics of the mbox format, but I can use a generator.

I'll give that a try.  Thanks for the feedback @maxhawkins and @simonw. I'll give that a try.

@simonw can we hold off on merging this until I can test this new approach?","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-849708617,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,849708617,MDEyOklzc3VlQ29tbWVudDg0OTcwODYxNw==,28565,maxhawkins,2021-05-27T15:01:42Z,2021-05-27T15:01:42Z,NONE,Any updates?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-884672647,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,884672647,IC_kwDODFE5qs40uwiH,28565,maxhawkins,2021-07-22T05:56:31Z,2021-07-22T14:03:08Z,NONE,"How does this commit look? https://github.com/maxhawkins/google-takeout-to-sqlite/commit/72802a83fee282eb5d02d388567731ba4301050d

It seems that Takeout's mbox format is pretty simple, so we can get away with just splitting the file on lines begining with `From `. My commit just splits the file every time a line starts with `From ` and uses `email.message_from_bytes` to parse each chunk.

I was able to load a 12GB takeout mbox without the program using more than a couple hundred MB of memory during the import process. It does make us lose the progress bar, but maybe I can add that back in a later commit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885022230,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,885022230,IC_kwDODFE5qs40wF4W,28565,maxhawkins,2021-07-22T15:51:46Z,2021-07-22T15:51:46Z,NONE,One thing I noticed is this importer doesn't save attachments along with the body of the emails. It would be nice if those got stored as blobs in a separate attachments table so attachments can be included while fetching search results.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885094284,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,885094284,IC_kwDODFE5qs40wXeM,28565,maxhawkins,2021-07-22T17:41:32Z,2021-07-22T17:41:32Z,NONE,I added a follow-up commit that deals with emails that don't have a `Date` header: https://github.com/maxhawkins/google-takeout-to-sqlite/commit/4bc70103582c10802c85a523ef1e99a8a2154aa9,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885098025,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,885098025,IC_kwDODFE5qs40wYYp,306240,UtahDave,2021-07-22T17:47:50Z,2021-07-22T17:47:50Z,NONE,"Hi @maxhawkins , I'm sorry, I haven't had any time to work on this. I'll have some time tomorrow to test your commits. I think they look great.  I'm great with your commits superseding my initial attempt here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-888075098,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,888075098,IC_kwDODFE5qs407vNa,28565,maxhawkins,2021-07-28T07:18:56Z,2021-07-28T07:18:56Z,NONE,"> I'm not sure why but my most recent import, when displayed in Datasette, looks like this:
> 
> <img alt=""mbox__mbox_emails__753_446_rows"" width=""574"" src=""https://user-images.githubusercontent.com/9599/109985836-0ab00080-7cba-11eb-97d5-0631a0835b61.png"">

I did some investigation into this issue and made a fix [here](https://github.com/dogsheep/google-takeout-to-sqlite/pull/8/commits/8ee555c2889a38ff42b95664ee074b4a01a82f06). The problem was that some messages (like gchat logs) don't have a `Message-Id` and we need to use `X-GM-THRID` as the pkey instead.

@simonw While looking into this I found something unexpected about how sqlite_utils handles upserts if the pkey column is `None`. When the pkey is NULL I'd expect the function to either use rowid or throw an exception. Instead, it seems upsert_all creates a row where all columns are NULL instead of using the values provided as parameters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1002735370,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,1002735370,IC_kwDODFE5qs47xIcK,203343,Btibert3,2021-12-29T18:58:23Z,2021-12-29T18:58:23Z,NONE,"@maxhawkins how hard would it be to add an entry to the table that includes the HTML version of the email, if it exists?  I just attempted your the PR branch on a very small mbox file, and it worked great.  My use case is a research project and I need to access more than just the body plain text.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1003437288,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,1003437288,IC_kwDODFE5qs47zzzo,28565,maxhawkins,2021-12-31T19:06:20Z,2021-12-31T19:06:20Z,NONE,"> @maxhawkins how hard would it be to add an entry to the table that includes the HTML version of the email, if it exists? I just attempted your the PR branch on a very small mbox file, and it worked great. My use case is a research project and I need to access more than just the body plain text.

Shouldn't be hard. The easiest way is probably to remove the `if body.content_type == ""text/html""` clause from [utils.py:254](https://github.com/dogsheep/google-takeout-to-sqlite/pull/8/commits/8e6d487b697ce2e8ad885acf613a157bfba84c59#diff-25ad9dd1ced1b8bfc37fda8444819c803232c08891e4af3d4064aa205d8174eaR254) and just return content directly without parsing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-894581223,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,894581223,IC_kwDODFE5qs41Ujnn,28565,maxhawkins,2021-08-07T00:57:48Z,2021-08-07T00:57:48Z,NONE,"Just added two more fixes:

* Added parsing for rfc 2047 encoded unicode headers
* Body is now stored as TEXT rather than a BLOB regardless of what order the messages are parsed in.

I was able to run this on my Takeout export and everything seems to work fine. @simonw let me know if this looks good to merge.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-896378525,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,896378525,IC_kwDODFE5qs41baad,28565,maxhawkins,2021-08-10T23:28:45Z,2021-08-10T23:28:45Z,NONE,"I added parsing of text/html emails using BeautifulSoup.

Around half of the emails in my archive don't include a text/plain payload so adding html parsing makes a good chunk of them searchable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/dogsheep/hacker-news-to-sqlite/issues/1#issuecomment-602094386,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/1,602094386,MDEyOklzc3VlQ29tbWVudDYwMjA5NDM4Ng==,9599,simonw,2020-03-21T19:47:46Z,2020-03-21T19:47:46Z,MEMBER,Released as 0.3.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585526292,Set up full text search,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135562,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2,886135562,IC_kwDODtX3eM400VsK,9599,simonw,2021-07-25T02:01:11Z,2021-07-25T02:01:11Z,MEMBER,"That page doesn't have an API but does look easy to scrape.

The other option here is the HN Search API powered by Algolia, documented at https://hn.algolia.com/api","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952179830,Command for fetching Hacker News threads from the search API,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135922,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2,886135922,IC_kwDODtX3eM400Vxy,9599,simonw,2021-07-25T02:06:20Z,2021-07-25T02:06:20Z,MEMBER,"https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url looks like it does what I want.

https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000 - returns 1000 at once.

Otherwise you have to paginate using `&page=2` etc - up to `nbPages` pages.

https://www.algolia.com/doc/api-reference/api-parameters/hitsPerPage/ says 1000 is the maximum.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952179830,Command for fetching Hacker News threads from the search API,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886136224,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2,886136224,IC_kwDODtX3eM400V2g,9599,simonw,2021-07-25T02:08:29Z,2021-07-25T02:08:29Z,MEMBER,"Prototype:

    curl ""https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000"" | \
      jq .hits | sqlite-utils insert hn.db items - --pk objectID --alter","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952179830,Command for fetching Hacker News threads from the search API,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886140431,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2,886140431,IC_kwDODtX3eM400W4P,9599,simonw,2021-07-25T03:12:57Z,2021-07-25T03:12:57Z,MEMBER,"I'm going to build a general-purpose `hacker-new-to-sqlite search ...` command, where one of the options is to search within the URL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952179830,Command for fetching Hacker News threads from the search API,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886142671,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3,886142671,IC_kwDODtX3eM400XbP,9599,simonw,2021-07-25T03:51:05Z,2021-07-25T03:51:05Z,MEMBER,"Prototype:

    curl 'https://hn.algolia.com/api/v1/items/27941108' \
      | jq '[recurse(.children[]) | del(.children)]' \
      | sqlite-utils insert hn.db items - --pk id
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952189173,Use HN algolia endpoint to retrieve trees,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886237834,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3,886237834,IC_kwDODtX3eM400uqK,9599,simonw,2021-07-25T18:05:32Z,2021-07-25T18:05:32Z,MEMBER,"If you hit the endpoint for a comment that's part of a thread you get that comment and its recursive children: https://hn.algolia.com/api/v1/items/27941552

You can tell that it's not the top-level because the `parent_id` isn't `null`. You can use `story_id` to figure out what the top-level item is.

```json
{
  ""id"": 27941552,
  ""created_at"": ""2021-07-24T15:08:39.000Z"",
  ""created_at_i"": 1627139319,
  ""type"": ""comment"",
  ""author"": ""nine_k"",
  ""title"": null,
  ""url"": null,
  ""text"": ""<p>I wish ..."",
  ""points"": null,
  ""parent_id"": 27941108,
  ""story_id"": 27941108
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952189173,Use HN algolia endpoint to retrieve trees,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886241674,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3,886241674,IC_kwDODtX3eM400vmK,9599,simonw,2021-07-25T18:41:17Z,2021-07-25T18:41:17Z,MEMBER,Got a TIL out of this: https://til.simonwillison.net/jq/extracting-objects-recursively,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952189173,Use HN algolia endpoint to retrieve trees,
https://github.com/dogsheep/hacker-news-to-sqlite/pull/6#issuecomment-1489110168,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/6,1489110168,IC_kwDODtX3eM5YwgSY,1231935,xavdid,2023-03-29T18:36:16Z,2023-03-29T18:36:16Z,NONE,@simonw can you take a look when you have a chance?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1641117021,Add permalink virtual field to items table,
https://github.com/dogsheep/healthkit-to-sqlite/issues/1#issuecomment-513437463,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/1,513437463,MDEyOklzc3VlQ29tbWVudDUxMzQzNzQ2Mw==,9599,simonw,2019-07-20T05:19:59Z,2019-07-20T05:19:59Z,MEMBER,"I ran xml_analyser against the XML HealthKit `export.xml` file and got the following results:

```python
{
  'ActivitySummary': {'attr_counts': {'activeEnergyBurned': 980,
                                     'activeEnergyBurnedGoal': 980,
                                     'activeEnergyBurnedUnit': 980,
                                     'appleExerciseTime': 980,
                                     'appleExerciseTimeGoal': 980,
                                     'appleStandHours': 980,
                                     'appleStandHoursGoal': 980,
                                     'dateComponents': 980},
                     'child_counts': {},
                     'count': 980,
                     'parent_counts': {'HealthData': 980}},
 'Correlation': {'attr_counts': {'creationDate': 1,
                                 'endDate': 1,
                                 'sourceName': 1,
                                 'sourceVersion': 1,
                                 'startDate': 1,
                                 'type': 1},
                 'child_counts': {'MetadataEntry': 1, 'Record': 2},
                 'count': 1,
                 'parent_counts': {'HealthData': 1}},
 'ExportDate': {'attr_counts': {'value': 1},
                'child_counts': {},
                'count': 1,
                'parent_counts': {'HealthData': 1}},
 'HealthData': {'attr_counts': {'locale': 1},
                'child_counts': {'ActivitySummary': 980,
                                 'Correlation': 1,
                                 'ExportDate': 1,
                                 'Me': 1,
                                 'Record': 2672231,
                                 'Workout': 663},
                'count': 1,
                'parent_counts': {}},
 'HeartRateVariabilityMetadataList': {'attr_counts': {},
                                      'child_counts': {'InstantaneousBeatsPerMinute': 93653},
                                      'count': 2318,
                                      'parent_counts': {'Record': 2318}},
 'InstantaneousBeatsPerMinute': {'attr_counts': {'bpm': 93653, 'time': 93653},
                                 'child_counts': {},
                                 'count': 93653,
                                 'parent_counts': {'HeartRateVariabilityMetadataList': 93653}},
 'Location': {'attr_counts': {'altitude': 398683,
                              'course': 398683,
                              'date': 398683,
                              'horizontalAccuracy': 398683,
                              'latitude': 398683,
                              'longitude': 398683,
                              'speed': 398683,
                              'verticalAccuracy': 398683},
              'child_counts': {},
              'count': 398683,
              'parent_counts': {'WorkoutRoute': 398683}},
 'Me': {'attr_counts': {'HKCharacteristicTypeIdentifierBiologicalSex': 1,
                        'HKCharacteristicTypeIdentifierBloodType': 1,
                        'HKCharacteristicTypeIdentifierDateOfBirth': 1,
                        'HKCharacteristicTypeIdentifierFitzpatrickSkinType': 1},
        'child_counts': {},
        'count': 1,
        'parent_counts': {'HealthData': 1}},
 'MetadataEntry': {'attr_counts': {'key': 290449, 'value': 290449},
                   'child_counts': {},
                   'count': 290449,
                   'parent_counts': {'Correlation': 1,
                                     'Record': 287974,
                                     'Workout': 1928,
                                     'WorkoutRoute': 546}},
 'Record': {'attr_counts': {'creationDate': 2672233,
                            'device': 2665111,
                            'endDate': 2672233,
                            'sourceName': 2672233,
                            'sourceVersion': 2671779,
                            'startDate': 2672233,
                            'type': 2672233,
                            'unit': 2650012,
                            'value': 2672232},
            'child_counts': {'HeartRateVariabilityMetadataList': 2318,
                             'MetadataEntry': 287974},
            'count': 2672233,
            'parent_counts': {'Correlation': 2, 'HealthData': 2672231}},
 'Workout': {'attr_counts': {'creationDate': 663,
                             'device': 230,
                             'duration': 663,
                             'durationUnit': 663,
                             'endDate': 663,
                             'sourceName': 663,
                             'sourceVersion': 663,
                             'startDate': 663,
                             'totalDistance': 663,
                             'totalDistanceUnit': 663,
                             'totalEnergyBurned': 663,
                             'totalEnergyBurnedUnit': 663,
                             'workoutActivityType': 663},
             'child_counts': {'MetadataEntry': 1928,
                              'WorkoutEvent': 2094,
                              'WorkoutRoute': 340},
             'count': 663,
             'parent_counts': {'HealthData': 663}},
 'WorkoutEvent': {'attr_counts': {'date': 2094,
                                  'duration': 837,
                                  'durationUnit': 837,
                                  'type': 2094},
                  'child_counts': {},
                  'count': 2094,
                  'parent_counts': {'Workout': 2094}},
 'WorkoutRoute': {'attr_counts': {'creationDate': 340,
                                  'endDate': 340,
                                  'sourceName': 340,
                                  'sourceVersion': 340,
                                  'startDate': 340},
                  'child_counts': {'Location': 398683, 'MetadataEntry': 546},
                  'count': 340,
                  'parent_counts': {'Workout': 340}}}
```

The most interesting bit is this:

```python
 'HealthData': {'attr_counts': {'locale': 1},
                'child_counts': {'ActivitySummary': 980,
                                 'Correlation': 1,
                                 'ExportDate': 1,
                                 'Me': 1,
                                 'Record': 2672231,
                                 'Workout': 663},
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470637068,Use XML Analyser to figure out the structure of the export XML,
https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550783316,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550783316,MDEyOklzc3VlQ29tbWVudDU1MDc4MzMxNg==,9599,simonw,2019-11-07T05:16:56Z,2019-11-07T05:34:29Z,MEMBER,"It looks like Apple changed the location of these in iOS 13 - they are now in separate `.gpx` files:

![2FF70E95-CDEE-4241-A5C5-EE95A862E519](https://user-images.githubusercontent.com/9599/68362042-be12e000-00da-11ea-8925-7397410332d8.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points,
https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550806302,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550806302,MDEyOklzc3VlQ29tbWVudDU1MDgwNjMwMg==,9599,simonw,2019-11-07T05:33:31Z,2019-11-07T05:33:31Z,MEMBER,"The XML now includes references to these new files:

![CBBA54FC-51FB-4BB3-927C-C2CA99237B04](https://user-images.githubusercontent.com/9599/68362716-121ec400-00dd-11ea-9846-387c7cd64c8b.jpeg)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points,
https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550824838,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550824838,MDEyOklzc3VlQ29tbWVudDU1MDgyNDgzOA==,9599,simonw,2019-11-07T05:47:07Z,2019-11-07T05:47:07Z,MEMBER,"Relevant code:

https://github.com/dogsheep/healthkit-to-sqlite/blob/d16f45f06fbae6ec8a78cc9ca7b5b7db0413f139/healthkit_to_sqlite/utils.py#L58-L64","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points,
https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550828084,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550828084,MDEyOklzc3VlQ29tbWVudDU1MDgyODA4NA==,9599,simonw,2019-11-07T05:49:24Z,2019-11-07T05:49:24Z,MEMBER,"So the fix there is going to be to detect the new `FileReference` element and load the corresponding points data from it.

This will be a little tricky because that function will need access to the zip file.

It probably won't work at all for the mode where the `export.xml` file is passed directly using the `--xml` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711074031,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711074031,MDEyOklzc3VlQ29tbWVudDcxMTA3NDAzMQ==,9599,simonw,2020-10-17T20:14:01Z,2020-10-17T20:14:01Z,MEMBER,I'd be happy to teach the tool to look for `export.xml` or `eksport.xml` - and then expand that list to other languages.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711074306,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711074306,MDEyOklzc3VlQ29tbWVudDcxMTA3NDMwNg==,9599,simonw,2020-10-17T20:16:22Z,2020-10-17T20:16:22Z,MEMBER,"The ""first XML file in the root"" solution is probably easier though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711078917,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711078917,MDEyOklzc3VlQ29tbWVudDcxMTA3ODkxNw==,9599,simonw,2020-10-17T20:51:55Z,2020-10-17T20:52:03Z,MEMBER,"I switched my phone to Spanish and ran an export - I got a file called `exportar.zip`. Unzipped I still got a `apple_
health_export` folder but the root contained:
```
electrocardiograms/
export_cda.xml
exportar.xml
workout-routes/
```
It looks like `export_cda.xml` does not have a translated name, so maybe I can ignore it and look for the _other_ `.xml` file in that directory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711079056,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711079056,MDEyOklzc3VlQ29tbWVudDcxMTA3OTA1Ng==,9599,simonw,2020-10-17T20:53:00Z,2020-10-17T20:53:00Z,MEMBER,"I think the safest thing is to sniff the first few lines of the file. Those should be the same no matter the language that was used:
```xml
<?xml version=""1.0"" encoding=""UTF-8""?>
<!DOCTYPE HealthData [
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711079760,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711079760,MDEyOklzc3VlQ29tbWVudDcxMTA3OTc2MA==,9599,simonw,2020-10-17T21:00:05Z,2020-10-17T21:00:05Z,MEMBER,Checking for either `<!DOCTYPE HealthData` or `<HealthData` in the first 1000 bytes should do it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711081703,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711081703,MDEyOklzc3VlQ29tbWVudDcxMTA4MTcwMw==,9599,simonw,2020-10-17T21:18:35Z,2020-10-17T21:18:35Z,MEMBER,"OK, if you upgrade to the just-released 1.0 this should work (it worked against my Spanish export).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711083698,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711083698,MDEyOklzc3VlQ29tbWVudDcxMTA4MzY5OA==,572,jarib,2020-10-17T21:39:15Z,2020-10-17T21:39:15Z,NONE,Nice! Works perfectly. Thanks for the quick response and great tooling in general.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-1163917719,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12,1163917719,IC_kwDOC8tyDs5FX_mX,956433,Mjboothaus,2022-06-23T04:35:02Z,2022-06-23T04:35:02Z,NONE,In terms of unique identifiers - could you use values stored in `HKMetadataKeySyncIdentifier`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727848625,"Some workout columns should be float, not text",
https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-877805513,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12,877805513,MDEyOklzc3VlQ29tbWVudDg3NzgwNTUxMw==,956433,Mjboothaus,2021-07-11T14:03:01Z,2021-07-11T14:03:01Z,NONE,"Hi Simon -- just experimenting with your excellent software!

Up to this point in time I have been using the (paid) [HealthFit App](https://apps.apple.com/au/app/healthfit/id1202650514) to export my workouts from my Apple Watch, one walk at the time into either .GPX or .FIT format and then using another library to suck it into Python and eventually here to my ""Emmaus Walking"" app:

https://share.streamlit.io/mjboothaus/emmaus_walking/emmaus_walking/app.py

I just used `healthkit-to-sqlite` to convert my export.zip file and it all ""just worked"".

I did notice the issue with various numeric fields being stored in the SQLite db as TEXT for now and just thought I'd flag it - but you're already self-reported this issue.

Keep up the great work!

I was curious if you have any thoughts about periodically exporting ""export.zip"" and how to just update the SQLite file instead of re-creating it each time. Hopefully Apple will give some thought to managing this data in a more sensible fashion as it grows over time.  Ideally one could pull it from iCloud (where it is allegedly being backed up).


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727848625,"Some workout columns should be float, not text",
https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-877874117,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12,877874117,MDEyOklzc3VlQ29tbWVudDg3Nzg3NDExNw==,956433,Mjboothaus,2021-07-11T23:03:37Z,2021-07-11T23:03:37Z,NONE,P.s. wondering if you have explored using the spatialite functionality with the location data in workouts?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727848625,"Some workout columns should be float, not text",
https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-879477586,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12,879477586,MDEyOklzc3VlQ29tbWVudDg3OTQ3NzU4Ng==,9599,simonw,2021-07-13T23:50:06Z,2021-07-13T23:50:06Z,MEMBER,"Unfortunately I don't think updating the database is practical, because the export doesn't include unique identifiers which can be used to update existing records and create new ones. Recreating from scratch works around that limitation.

I've not explored workouts with SpatiaLite but that's a really good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727848625,"Some workout columns should be float, not text",
https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1073123231,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14,1073123231,IC_kwDOC8tyDs4_9o-f,343884,lchski,2022-03-19T22:39:29Z,2022-03-19T22:39:29Z,NONE,"I have this issue, too, with a fresh export. None of my `Workout` entries in `export.xml` have an `id` key, though [the sample `export.xml` in the tests folder doesn’t either](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/tests/zip_contents/apple_health_export/export.xml#L14-L21), so I don’t think this is the culprit. Indeed, it seems @simonw is using the [`hash_id` function from `sqlite_utils`](https://sqlite-utils.datasette.io/en/stable/python-api.html#setting-an-id-based-on-the-hash-of-the-row-contents), which creates a column (`id`, in this case) based on a hash of the row’s contents.

When I run the script, a `workouts` table is created, with one entry: my first workout. No `workout_points` table is created, as [I’d expect from `utils.py`](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/healthkit_to_sqlite/utils.py#L89-L90). I then get essentially the same error as noted in this thread:

```Importing from HealthKit  [###################################-]   98%  00:00:01
Traceback (most recent call last):
  File ""/Users/lchski/.pyenv/versions/3.10.3/bin/healthkit-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py"", line 1128, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py"", line 1053, in main
    rv = self.invoke(ctx)
  File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py"", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py"", line 754, in invoke
    return __callback(*args, **kwargs)
  File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/cli.py"", line 57, in cli
    convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)
  File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/utils.py"", line 34, in convert_xml_to_sqlite
    workout_to_db(el, db, zipfile)
  File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/utils.py"", line 57, in workout_to_db
    pk = db[""workouts""].insert(record, alter=True, hash_id=""id"").last_pk
  File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py"", line 2822, in insert
    return self.insert_all(
  File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py"", line 2950, in insert_all
    self.insert_chunk(
  File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py"", line 2715, in insert_chunk
    result = self.db.execute(query, params)
  File ""/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py"", line 458, in execute
    return self.conn.execute(sql, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: workouts.id
```

Are there maybe duplicate workouts in the data, which’d cause multiple rows to share the same `id`? It’s strange, though, that no `workout_points` is created at all. Export created from iOS 15.3.1.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771608692,UNIQUE constraint failed: workouts.id,
https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1073139067,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14,1073139067,IC_kwDOC8tyDs4_9s17,343884,lchski,2022-03-20T00:54:18Z,2022-03-20T00:54:18Z,NONE,"Update: this appears to be because of running the command twice without clearing the DB in between. Tries to insert a Workout that already exists, causing a collision on the (auto-generated) `id` column. Had a different error with a clean DB, likely due to the workout points format; will make a new issue for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771608692,UNIQUE constraint failed: workouts.id,
https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1629123734,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14,1629123734,IC_kwDOC8tyDs5hGnSW,44622670,philipp-heinrich,2023-07-10T14:46:52Z,2023-07-10T14:46:52Z,NONE,@simonw any chance to get this fixed soon? ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771608692,UNIQUE constraint failed: workouts.id,
https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798436026,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14,798436026,MDEyOklzc3VlQ29tbWVudDc5ODQzNjAyNg==,1234956,n8henrie,2021-03-13T14:23:16Z,2021-03-13T14:23:16Z,NONE,"This PR allows my import to succeed.

It looks like some events don't have an `id`, but do have `HKExternalUUID` (which gets turned into `metadata_HKExternalUUID`), so I use this as a fallback.

If a record has neither of these, I changed it to just print the record (for debugging) and `return`.

For some odd reason this ran fine at first, and now (after removing the generated db and trying again) I'm getting a different error (duplicate column name).

Looks like it may have run when I had two successive runs without remembering to delete the db in between. Will try to refactor.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771608692,UNIQUE constraint failed: workouts.id,
https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798468572,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14,798468572,MDEyOklzc3VlQ29tbWVudDc5ODQ2ODU3Mg==,1234956,n8henrie,2021-03-13T14:47:31Z,2021-03-13T14:47:31Z,NONE,"Ok, new PR works. I'm not `git` enough so I just force-pushed over the old one.

I still end up with a lot of activities that are missing an `id` and therefore skipped (since this is used as the primary key). For example:

```
{'workoutActivityType': 'HKWorkoutActivityTypeRunning', 'duration': '35.31666666666667', 'durationUnit': 'min', 'totalDistance': '4.010870267636999', 'totalDistanceUnit': 'mi', 'totalEnergyBurned': '660.3516235351562', 'totalEnergyBurnedUnit': 'Cal', 'sourceName': 'Strava', 'sourceVersion': '22810', 'creationDate': '2020-07-16 13:38:26 -0700', 'startDate': '2020-07-16 06:38:26 -0700', 'endDate': '2020-07-16 07:13:45 -0700'}
```

I also end up with some unhappy characters (in the skipped events), such as: `'sourceName': 'Nathan’s Apple\xa0Watch',`.

But it's successfully making it through the file, and the resulting db opens in datasette, so I'd call that progress.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771608692,UNIQUE constraint failed: workouts.id,
https://github.com/dogsheep/healthkit-to-sqlite/issues/2#issuecomment-513439411,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/2,513439411,MDEyOklzc3VlQ29tbWVudDUxMzQzOTQxMQ==,9599,simonw,2019-07-20T05:58:57Z,2019-07-20T05:58:57Z,MEMBER,"```python
 'Workout': {'attr_counts': {'creationDate': 663,
                             'device': 230,
                             'duration': 663,
                             'durationUnit': 663,
                             'endDate': 663,
                             'sourceName': 663,
                             'sourceVersion': 663,
                             'startDate': 663,
                             'totalDistance': 663,
                             'totalDistanceUnit': 663,
                             'totalEnergyBurned': 663,
                             'totalEnergyBurnedUnit': 663,
                             'workoutActivityType': 663},
             'child_counts': {'MetadataEntry': 1928,
                              'WorkoutEvent': 2094,
                              'WorkoutRoute': 340},
             'count': 663,
             'parent_counts': {'HealthData': 663}},
 'WorkoutEvent': {'attr_counts': {'date': 2094,
                                  'duration': 837,
                                  'durationUnit': 837,
                                  'type': 2094},
                  'child_counts': {},
                  'count': 2094,
                  'parent_counts': {'Workout': 2094}},
 'WorkoutRoute': {'attr_counts': {'creationDate': 340,
                                  'endDate': 340,
                                  'sourceName': 340,
                                  'sourceVersion': 340,
                                  'startDate': 340},
                  'child_counts': {'Location': 398683, 'MetadataEntry': 546},
                  'count': 340,
                  'parent_counts': {'Workout': 340}}}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470637152,Import workouts,
https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902355471,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20,902355471,IC_kwDOC8tyDs41yNoP,9599,simonw,2021-08-20T01:09:07Z,2021-08-20T01:09:07Z,MEMBER,"Workaround:

    sqlite-utils create-index healthkit.db workout_points -- -date

See https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",975166271,Add index on workout_points.date,
https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902356871,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20,902356871,IC_kwDOC8tyDs41yN-H,9599,simonw,2021-08-20T01:12:48Z,2021-08-20T01:12:48Z,MEMBER,Also on `workout_points.workout_id` to speed up queries to show all points in a specific workout.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",975166271,Add index on workout_points.date,
https://github.com/dogsheep/healthkit-to-sqlite/issues/21#issuecomment-903950096,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/21,903950096,IC_kwDOC8tyDs414S8Q,32016596,FabianHertwig,2021-08-23T17:00:59Z,2021-08-23T17:00:59Z,NONE,"I think the issue is that I have records like these:

```xml
 <Record type=""HKQuantityTypeIdentifierDietaryCholesterol"" sourceName=""MyFitnessPal"" sourceVersion=""35120"" unit=""mg"" creationDate=""2021-07-04 20:55:27 +0200"" startDate=""2021-07-04 20:55:00 +0200"" endDate=""2021-07-04 20:55:00 +0200"" value=""124"">
  <MetadataEntry key=""meal"" value=""Dinner""/>
  <MetadataEntry key=""Meal"" value=""Dinner""/>
 </Record>
```

And if sqlite is case insensitive, then `metadata_meal` and `metadata_Meal` result in the same column.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977128935,Duplicate Column,
https://github.com/dogsheep/healthkit-to-sqlite/issues/24#issuecomment-1464786643,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24,1464786643,IC_kwDOC8tyDs5XTt7T,956433,Mjboothaus,2023-03-11T02:01:27Z,2023-03-11T02:01:27Z,NONE,Thanks for reporting this and providing a solution -- I was puzzled by this error when I revisited my walking data and experienced this issues. I haven't tried the fix yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515883470,DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 ,
https://github.com/dogsheep/healthkit-to-sqlite/issues/24#issuecomment-1464796494,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24,1464796494,IC_kwDOC8tyDs5XTwVO,956433,Mjboothaus,2023-03-11T02:23:42Z,2023-03-11T02:23:42Z,NONE,@simonw - maybe put in some error handling to trap for poorly formed XML (from Apple engineers) so that it suggests that there are problems with export.zip rather than odd looking Python errors :),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515883470,DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 ,
https://github.com/dogsheep/healthkit-to-sqlite/issues/4#issuecomment-513440090,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/4,513440090,MDEyOklzc3VlQ29tbWVudDUxMzQ0MDA5MA==,9599,simonw,2019-07-20T06:11:50Z,2019-07-20T06:11:50Z,MEMBER,"Some examples:

https://github.com/dogsheep/healthkit-to-sqlite/blob/d016e70c31cf84ba0f5ec3102546db54a51aaffb/tests/export.xml#L4-L13","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470640505,Import Records,
https://github.com/dogsheep/healthkit-to-sqlite/issues/5#issuecomment-513514978,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5,513514978,MDEyOklzc3VlQ29tbWVudDUxMzUxNDk3OA==,9599,simonw,2019-07-21T02:55:12Z,2019-07-21T02:55:12Z,MEMBER,I'm going to show this by default. Users can pass `-s` or `--silent` to disable the progress bar.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470691622,Add progress bar,
https://github.com/dogsheep/healthkit-to-sqlite/issues/5#issuecomment-513625406,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5,513625406,MDEyOklzc3VlQ29tbWVudDUxMzYyNTQwNg==,9599,simonw,2019-07-22T03:20:16Z,2019-07-22T03:20:16Z,MEMBER,"It now renders like this:
```
Importing from HealthKit  [#-----------------------------------]    5%  00:01:33
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470691622,Add progress bar,
https://github.com/dogsheep/healthkit-to-sqlite/issues/6#issuecomment-513626742,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/6,513626742,MDEyOklzc3VlQ29tbWVudDUxMzYyNjc0Mg==,9599,simonw,2019-07-22T03:28:55Z,2019-07-22T03:28:55Z,MEMBER,"Here's what it looks like now as separate tables:

<img width=""1050"" alt=""hello9_and_Populate__endpoint__key_in_ASGI_scope_·_Issue__537_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/61604322-eba05000-abf5-11e9-8f8b-d0204e9a705a.png"">


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470856782,Break up records into different tables for each type,
https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514496725,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7,514496725,MDEyOklzc3VlQ29tbWVudDUxNDQ5NjcyNQ==,9599,simonw,2019-07-24T06:20:59Z,2019-07-24T06:20:59Z,MEMBER,"I'm using https://pypi.org/project/memory-profiler/ to explore this in more detail:

```
$ pip install memory-profiler matplotlib
```

Then:

```
$ mprof run healthkit-to-sqlite ~/Downloads/healthkit-export.zip healthkit.db
$ mprof plot
```

<img width=""1128"" alt=""Screen Shot 2019-07-24 at 8 17 06 AM"" src=""https://user-images.githubusercontent.com/9599/61769849-f3294a00-adeb-11e9-9026-02c24209530f.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472097220,Script uses a lot of RAM,
https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514498221,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7,514498221,MDEyOklzc3VlQ29tbWVudDUxNDQ5ODIyMQ==,9599,simonw,2019-07-24T06:26:49Z,2019-07-24T06:26:49Z,MEMBER,"Adding `el.clear()` got me a huge improvement:

<img width=""1128"" alt=""Screen Shot 2019-07-24 at 8 23 26 AM"" src=""https://user-images.githubusercontent.com/9599/61770130-c3c70d00-adec-11e9-886a-93a2cea5bf4b.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472097220,Script uses a lot of RAM,
https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514500253,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7,514500253,MDEyOklzc3VlQ29tbWVudDUxNDUwMDI1Mw==,9599,simonw,2019-07-24T06:34:28Z,2019-07-24T06:34:28Z,MEMBER,"Clearing the root element each time saved even more:

<img width=""1128"" alt=""Screen Shot 2019-07-24 at 8 30 38 AM"" src=""https://user-images.githubusercontent.com/9599/61770555-d3932100-aded-11e9-8ffe-bebd682f94ed.png"">
","{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 2, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472097220,Script uses a lot of RAM,
https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-514745798,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9,514745798,MDEyOklzc3VlQ29tbWVudDUxNDc0NTc5OA==,166463,tholo,2019-07-24T18:25:36Z,2019-07-24T18:25:36Z,NONE,"This is on macOS 10.14.6, with Python 3.7.4, packages in the virtual environment:

```
Package             Version
------------------- -------
aiofiles            0.4.0
Click               7.0
click-default-group 1.2.1
datasette           0.29.2
h11                 0.8.1
healthkit-to-sqlite 0.3.1
httptools           0.0.13
hupper              1.8.1
importlib-metadata  0.18
Jinja2              2.10.1
MarkupSafe          1.1.1
Pint                0.8.1
pip                 19.2.1
pluggy              0.12.0
setuptools          41.0.1
sqlite-utils        1.7
tabulate            0.8.3
uvicorn             0.8.4
uvloop              0.12.2
websockets          7.0
zipp                0.5.2
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472429048,Too many SQL variables,
https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515226724,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9,515226724,MDEyOklzc3VlQ29tbWVudDUxNTIyNjcyNA==,9599,simonw,2019-07-25T21:46:01Z,2019-07-25T21:46:01Z,MEMBER,I can work around this here (prior to the fix in sqlite-utils) by setting the batch size to something a bit lower here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472429048,Too many SQL variables,
https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515322294,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9,515322294,MDEyOklzc3VlQ29tbWVudDUxNTMyMjI5NA==,9599,simonw,2019-07-26T06:07:12Z,2019-07-26T06:07:12Z,MEMBER,@tholo this should be fixed in just-released version 0.3.2 - could you run a `pip install -U healthkit-to-sqlite` and let me know if it works for you now?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472429048,Too many SQL variables,
https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515370687,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9,515370687,MDEyOklzc3VlQ29tbWVudDUxNTM3MDY4Nw==,166463,tholo,2019-07-26T09:01:19Z,2019-07-26T09:01:19Z,NONE,"Yes, that did fix the issue I was seeing — it will now import my complete HealthKit data.

Thorsten

> On Jul 25, 2019, at 23:07, Simon Willison <notifications@github.com> wrote:
> 
> @tholo <https://github.com/tholo> this should be fixed in just-released version 0.3.2 - could you run a pip install -U healthkit-to-sqlite and let me know if it works for you now?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/dogsheep/healthkit-to-sqlite/issues/9?email_source=notifications&email_token=AABIUPYTWYOYSLEAFS7TLMLQBKIBBA5CNFSM4IGSXNNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD23TDNQ#issuecomment-515322294>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABIUP7DCBQ37SESQL7D4WTQBKIBBANCNFSM4IGSXNNA>.
> 

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472429048,Too many SQL variables,
https://github.com/dogsheep/healthkit-to-sqlite/pull/13#issuecomment-904642396,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/13,904642396,IC_kwDOC8tyDs41679c,32016596,FabianHertwig,2021-08-24T13:27:40Z,2021-08-24T13:28:26Z,NONE,This would fix #21 and make #22 obsolete.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743071410,SQLite does not have case sensitive columns,
https://github.com/dogsheep/healthkit-to-sqlite/pull/22#issuecomment-904641261,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/22,904641261,IC_kwDOC8tyDs4167rt,32016596,FabianHertwig,2021-08-24T13:26:20Z,2021-08-24T13:26:20Z,NONE,Did not see that #13 fixes the same issue in a similar way. You can decide which one to merge ;),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978086284,Make sure that case-insensitive column names are unique,
https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-538847446,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1,538847446,MDEyOklzc3VlQ29tbWVudDUzODg0NzQ0Ng==,9599,simonw,2019-10-07T05:41:17Z,2019-10-07T05:41:17Z,MEMBER,"Prototype code:
```python
offset = 0
fetched = []
size = 500
while True:
    page = requests.get(""https://getpocket.com/v3/get"", {
        ""consumer_key"": consumer_key,
        ""access_token"": access_token,
        ""sort"": ""oldest"",
        ""detailType"": ""complete"",
        ""count"": size,
        ""offset"": offset,
    }).json()
    print(offset)
    fetched.append(page)
    offset += size
    if not len(page[""list""]):
        break
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503233021,Use better pagination (and implement progress bar),
https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605316146,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1,605316146,MDEyOklzc3VlQ29tbWVudDYwNTMxNjE0Ng==,9599,simonw,2020-03-27T21:09:15Z,2020-03-27T21:09:22Z,MEMBER,"For a progress bar I need to know how many total items there are.

I found an undocumented API for this! `/v3/stats` which returns:
```json
{
    ""count_list"": 7394,
    ""count_read"": 1016,
    ""count_unread"": 6378,
    ""status"": 1
}
```
I guessed this based on the documented v2 API: https://getpocket.com/api/v2_docs/#stats","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503233021,Use better pagination (and implement progress bar),
https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605325897,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1,605325897,MDEyOklzc3VlQ29tbWVudDYwNTMyNTg5Nw==,9599,simonw,2020-03-27T21:37:26Z,2020-03-27T21:38:37Z,MEMBER,"I keep getting 503 errors even though I appear to be staying within the rate limit:
```
{'Date': 'Fri, 27 Mar 2020 21:35:57 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding':
'chunked', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location':
'get.php', 'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...;  httponly',
'X-Frame-Options': 'SAMEORIGIN', 'Status': '200 OK', 'X-Limit-Key-Limit': '10000',
'X-Limit-Key-Remaining': '9960', 'X-Limit-Key-Reset': '282', 'X-Source': 'Pocket',
'P3P': 'policyref=""/w3c/p3p.xml"", CP=""ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE""'}

  [##----------------------------------]    6%  06:49:27

{'Date': 'Fri, 27 Mar 2020 21:36:06 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Content-Length':
'23', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location': 'get.php',
'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...', 'X-Frame-Options': 'SAMEORIGIN',
'X-Error': 'Pocket is currently under heavy load. Please wait a moment and try again.',
'X-Error-Code': '199', 'Status': '503 Service Unavailable', 'X-Source': 'Pocket',
'P3P': 'policyref=""/w3c/p3p.xml"", CP=""ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE""'}
```
I'm going to try doing a few automatic retries any time I see a 503 error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503233021,Use better pagination (and implement progress bar),
https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605327655,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1,605327655,MDEyOklzc3VlQ29tbWVudDYwNTMyNzY1NQ==,9599,simonw,2020-03-27T21:42:49Z,2020-03-27T21:42:49Z,MEMBER,Or maybe it was because of the current Google Cloud outage? https://news.ycombinator.com/item?id=22706677,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503233021,Use better pagination (and implement progress bar),
https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1221622873,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10,1221622873,IC_kwDODLZ_YM5I0HxZ,9599,simonw,2022-08-21T21:19:25Z,2022-08-21T21:19:25Z,MEMBER,"Agreed, that would be a much better implementation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1246826792,"When running `auth` command, don't overwrite an existing auth.json file",
https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1221623052,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10,1221623052,IC_kwDODLZ_YM5I0H0M,9599,simonw,2022-08-21T21:20:33Z,2022-08-21T21:20:33Z,MEMBER,"That was clearly the intention from the description of this issue:
- #4","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1246826792,"When running `auth` command, don't overwrite an existing auth.json file",
https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1239516561,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10,1239516561,IC_kwDODLZ_YM5J4YWR,11887,ashanan,2022-09-07T15:07:38Z,2022-09-07T15:07:38Z,NONE,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1246826792,"When running `auth` command, don't overwrite an existing auth.json file",
https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221521377,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11,1221521377,IC_kwDODLZ_YM5Izu_h,2467,fernand0,2022-08-21T10:51:37Z,2022-08-21T10:51:37Z,NONE,I didn't see there is a PR about this: https://github.com/dogsheep/pocket-to-sqlite/pull/7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1345452427,"-a option is used for ""--auth"" and for ""--all""",
https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221621466,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11,1221621466,IC_kwDODLZ_YM5I0Hba,9599,simonw,2022-08-21T21:09:47Z,2022-08-21T21:09:47Z,MEMBER,"Great catch, thanks.

I'm going to use it to mean `--auth` - since other tools in the Dogsheep family have the same convention.

 `--all` will be the only way to specify all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1345452427,"-a option is used for ""--auth"" and for ""--all""",
https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221621529,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11,1221621529,IC_kwDODLZ_YM5I0HcZ,9599,simonw,2022-08-21T21:10:15Z,2022-08-21T21:11:26Z,MEMBER,"Just saw that's what's implemented here already!
- #7","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1345452427,"-a option is used for ""--auth"" and for ""--all""",
https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627563202,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12,1627563202,IC_kwDODLZ_YM5hAqTC,9599,simonw,2023-07-09T01:14:27Z,2023-07-09T01:14:27Z,MEMBER,I tested this locally with `python -m build` and then `pip install ...whl` in a fresh virtual environment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795187493,Switch to pyproject.toml,
https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627564127,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12,1627564127,IC_kwDODLZ_YM5hAqhf,9599,simonw,2023-07-09T01:19:42Z,2023-07-09T01:19:42Z,MEMBER,https://github.com/dogsheep/pocket-to-sqlite/tree/0.2.3 and https://pypi.org/project/pocket-to-sqlite/0.2.3/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795187493,Switch to pyproject.toml,
https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-538847796,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2,538847796,MDEyOklzc3VlQ29tbWVudDUzODg0Nzc5Ng==,9599,simonw,2019-10-07T05:43:30Z,2019-10-07T05:43:30Z,MEMBER,We can persist the `since` value in its own single-row table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503234169,Track and use the 'since' value,
https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-605337941,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2,605337941,MDEyOklzc3VlQ29tbWVudDYwNTMzNzk0MQ==,9599,simonw,2020-03-27T22:16:32Z,2020-03-27T22:16:32Z,MEMBER,"Need to test this. I have 7,394 items in my database right now. I'm going to save a new thing.

Then I ran this:
```
pocket-to-sqlite fetch pocket-simon.db
```
And it worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503234169,Track and use the 'since' value,
https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-605338322,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2,605338322,MDEyOklzc3VlQ29tbWVudDYwNTMzODMyMg==,9599,simonw,2020-03-27T22:18:02Z,2020-03-27T22:18:02Z,MEMBER,Just needs documentation now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503234169,Track and use the 'since' value,
https://github.com/dogsheep/pocket-to-sqlite/issues/3#issuecomment-684424396,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3,684424396,MDEyOklzc3VlQ29tbWVudDY4NDQyNDM5Ng==,9599,simonw,2020-09-01T06:17:45Z,2020-09-01T06:17:45Z,MEMBER,It looks like I could ignore the `image` column and synthesize a unique key from the data in the `images` column using `$item_id/$image_id`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503243784,Extract images into separate tables,
https://github.com/dogsheep/pocket-to-sqlite/issues/5#issuecomment-684425714,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5,684425714,MDEyOklzc3VlQ29tbWVudDY4NDQyNTcxNA==,9599,simonw,2020-09-01T06:18:32Z,2020-09-01T06:18:32Z,MEMBER,"Good suggestion, I'll setup a demo somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629473827,Set up a demo,
https://github.com/dogsheep/pocket-to-sqlite/issues/9#issuecomment-774726123,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9,774726123,MDEyOklzc3VlQ29tbWVudDc3NDcyNjEyMw==,12669260,jfeiwell,2021-02-07T18:21:08Z,2021-02-07T18:21:08Z,NONE,@simonw any ideas here?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",801780625,SSL Error,
https://github.com/dogsheep/pocket-to-sqlite/issues/9#issuecomment-774730656,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9,774730656,MDEyOklzc3VlQ29tbWVudDc3NDczMDY1Ng==,635179,merwok,2021-02-07T18:45:04Z,2021-02-07T18:45:04Z,NONE,"That URL uses TLS 1.3, but maybe only if the client supports it.
It could be your Python version or your SSL library that’s not recent enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",801780625,SSL Error,
https://github.com/dogsheep/pocket-to-sqlite/pull/7#issuecomment-1221621700,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/7,1221621700,IC_kwDODLZ_YM5I0HfE,9599,simonw,2022-08-21T21:11:12Z,2022-08-21T21:11:12Z,MEMBER,I thought this might need a documentation update but `--all` is already covered: https://github.com/dogsheep/pocket-to-sqlite/blob/0.2.1/README.md,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750141615,Fixed conflicting  CLI flags,
https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-727692413,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11,727692413,MDEyOklzc3VlQ29tbWVudDcyNzY5MjQxMw==,9599,simonw,2020-11-16T02:15:22Z,2020-11-16T02:15:22Z,MEMBER,"Thanks, I'll look into this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743400216,Error thrown: sqlite3.OperationalError: table users has no column named lastName,
https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-761967094,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11,761967094,MDEyOklzc3VlQ29tbWVudDc2MTk2NzA5NA==,9599,simonw,2021-01-18T04:11:13Z,2021-01-18T04:11:13Z,MEMBER,"I just got a similar error:

```
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/swarm_to_sqlite/utils.py"", line 79, in save_checkin
    checkins_table.m2m(""users"", user, m2m_table=""with"", pk=""id"")
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 2048, in m2m
    id = other_table.insert(record, pk=pk, replace=True).last_pk
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1781, in insert
    return self.insert_all(
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1899, in insert_all
    self.insert_chunk(
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1709, in insert_chunk
    result = self.db.execute(query, params)
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 226, in execute
    return self.conn.execute(sql, parameters)
pysqlite3.dbapi2.OperationalError: table users has no column named countryCode

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743400216,Error thrown: sqlite3.OperationalError: table users has no column named lastName,
https://github.com/dogsheep/swarm-to-sqlite/issues/12#issuecomment-941274088,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/12,941274088,IC_kwDODD6af844GrPo,33631,fs111,2021-10-12T18:31:57Z,2021-10-12T18:31:57Z,NONE,I am running into the same problem. Is there any workaround?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",951817328,403 when getting token,
https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502543165,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13,1502543165,IC_kwDODD6af85Zjv09,9599,simonw,2023-04-11T01:10:36Z,2023-04-11T01:11:47Z,MEMBER,"I just had that error myself on macOS while running the tests:
```
ERROR tests/test_save_checkin.py::test_tables - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_venue - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_event - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_sticker - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_likes - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_with_ - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_users - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_photos - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_posts - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_view - sqlite3.OperationalError: table sqlite_master may not be modified
```
`pytest --pdb` shows it happening in the bit that adds foreign keys:
```
> /Users/simon/.local/share/virtualenvs/swarm-to-sqlite-daPW7yIJ/lib/python3.9/site-packages/sqlite_utils/db.py(1096)add_foreign_keys()
-> cursor.execute(
(Pdb) list
1096 >>	                cursor.execute(
1097 	                    ""UPDATE sqlite_master SET sql = ? WHERE name = ?"",
1098 	                    (new_sql, table_name),
1099 	                )
1100 	            cursor.execute(""PRAGMA schema_version = %d"" % (schema_version + 1))
1101 ->	            cursor.execute(""PRAGMA writable_schema = 0"")
1102 	        # Have to VACUUM outside the transaction to ensure .foreign_keys property
1103 	        # can see the newly created foreign key.
1104 	        self.vacuum()
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373210675,fails before generating views. ERR:  table sqlite_master may not be modified,
https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502546045,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13,1502546045,IC_kwDODD6af85Zjwh9,9599,simonw,2023-04-11T01:14:50Z,2023-04-11T01:14:50Z,MEMBER,"Related:
- https://github.com/simonw/sqlite-utils/issues/235","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373210675,fails before generating views. ERR:  table sqlite_master may not be modified,
https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502629219,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13,1502629219,IC_kwDODD6af85ZkE1j,9599,simonw,2023-04-11T03:15:26Z,2023-04-11T03:15:26Z,MEMBER,"OK, I figured this out. Unfortunately it's an error that occurs on Python versions that have defensive mode turned on, and it doesn't look like there's a way to turn that mode off. See notes above.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373210675,fails before generating views. ERR:  table sqlite_master may not be modified,
https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502629404,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13,1502629404,IC_kwDODD6af85ZkE4c,9599,simonw,2023-04-11T03:15:47Z,2023-04-11T03:46:17Z,MEMBER,"I think `swarm-to-sqlite` needs to avoid this error, maybe by setting up foreign keys in another way - or even by skipping foreign keys entirely on databases that don't support this kind of operation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373210675,fails before generating views. ERR:  table sqlite_master may not be modified,
https://github.com/dogsheep/swarm-to-sqlite/issues/2#issuecomment-526701674,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/2,526701674,MDEyOklzc3VlQ29tbWVudDUyNjcwMTY3NA==,9599,simonw,2019-08-30T18:24:26Z,2019-08-30T18:24:26Z,MEMBER,I renamed `--file` to `--load` in 0e5b6025c6f9823ff81aa8aae1cbff5c45e57baf,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487598468,--save option to dump checkins to a JSON file on disk,
https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-527200332,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3,527200332,MDEyOklzc3VlQ29tbWVudDUyNzIwMDMzMg==,9599,simonw,2019-09-02T16:32:20Z,2019-09-02T16:32:39Z,MEMBER,"Also needed: an option for ""fetch all checkins created within the last X days"".

This should help provide support for that Swarm feature where you can retroactively checkin to places in the past.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487600595,Option to fetch only checkins more recent than the current max checkin,
https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542875885,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3,542875885,MDEyOklzc3VlQ29tbWVudDU0Mjg3NTg4NQ==,9599,simonw,2019-10-16T20:23:08Z,2019-10-16T20:23:08Z,MEMBER,"https://developer.foursquare.com/docs/api/users/checkins documents `afterTimestamp`:
> Retrieve the first results to follow these seconds since epoch. This should be useful for paging forward in time, or when polling for changes. To avoid missing results when polling, we recommend subtracting several seconds from the last poll time and then de-duplicating.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487600595,Option to fetch only checkins more recent than the current max checkin,
https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542876047,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3,542876047,MDEyOklzc3VlQ29tbWVudDU0Mjg3NjA0Nw==,9599,simonw,2019-10-16T20:23:36Z,2019-10-16T20:23:36Z,MEMBER,I'm going to go with `--since=1d/2w/3h` for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487600595,Option to fetch only checkins more recent than the current max checkin,
https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542882604,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3,542882604,MDEyOklzc3VlQ29tbWVudDU0Mjg4MjYwNA==,9599,simonw,2019-10-16T20:41:23Z,2019-10-16T20:41:23Z,MEMBER,Documented here: https://github.com/dogsheep/swarm-to-sqlite/blob/0.2/README.md#usage,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487600595,Option to fetch only checkins more recent than the current max checkin,
https://github.com/dogsheep/swarm-to-sqlite/issues/4#issuecomment-526853542,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/4,526853542,MDEyOklzc3VlQ29tbWVudDUyNjg1MzU0Mg==,9599,simonw,2019-08-31T18:06:32Z,2019-08-31T18:06:32Z,MEMBER,"https://your-foursquare-oauth-token.glitch.me/

Source code: https://glitch.com/~your-foursquare-oauth-token","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487601121,Online tool for getting a Foursquare OAuth token,
https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-669241341,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8,669241341,MDEyOklzc3VlQ29tbWVudDY2OTI0MTM0MQ==,9599,simonw,2020-08-05T14:55:14Z,2020-08-05T14:55:14Z,MEMBER,"Looks like there's a column that's not consistently there, so the table got created without it.

Easiest fix is to add `alter=True` to this line:

https://github.com/dogsheep/swarm-to-sqlite/blob/f4a82633da927cde672c9d9af92930bfca2e3ddf/swarm_to_sqlite/utils.py#L94

That will cause `sqlite-utils` to notice if there's a missing column and add it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648245071,Error thrown: table photos has no column named hasSticker,
https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-707332912,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8,707332912,MDEyOklzc3VlQ29tbWVudDcwNzMzMjkxMg==,9599,simonw,2020-10-12T20:35:06Z,2020-10-12T20:35:06Z,MEMBER,Shipped a fix for this in [swarm-to-sqlite 0.3.2](https://github.com/dogsheep/swarm-to-sqlite/releases/tag/0.3.2).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648245071,Error thrown: table photos has no column named hasSticker,
https://github.com/dogsheep/swarm-to-sqlite/pull/10#issuecomment-707326192,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/10,707326192,MDEyOklzc3VlQ29tbWVudDcwNzMyNjE5Mg==,29426418,mattiaborsoi,2020-10-12T20:20:02Z,2020-10-12T20:20:02Z,CONTRIBUTOR,This closes issue #8 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",719637258,Update utils.py to fix sqlite3.OperationalError,
https://github.com/dogsheep/swarm-to-sqlite/pull/6#issuecomment-605382373,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/6,605382373,MDEyOklzc3VlQ29tbWVudDYwNTM4MjM3Mw==,9599,simonw,2020-03-28T02:27:32Z,2020-03-28T02:27:32Z,MEMBER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",543355051,don't break if source is missing,
https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-541748580,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,541748580,MDEyOklzc3VlQ29tbWVudDU0MTc0ODU4MA==,9599,simonw,2019-10-14T15:30:44Z,2019-10-14T15:30:44Z,MEMBER,Had several recommendations for https://github.com/tqdm/tqdm which is what goodreads-to-sqlite uses.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands,
https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543269396,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,543269396,MDEyOklzc3VlQ29tbWVudDU0MzI2OTM5Ng==,9599,simonw,2019-10-17T17:02:07Z,2019-10-17T17:02:07Z,MEMBER,A neat trick that Click does is detecting if an interactive terminal is attached and NOT showing a progress bar if there isn't one. Need to figure out how to do that with tqdm.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands,
https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543270714,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,543270714,MDEyOklzc3VlQ29tbWVudDU0MzI3MDcxNA==,9599,simonw,2019-10-17T17:05:16Z,2019-10-17T17:05:16Z,MEMBER,"https://github.com/pallets/click/blob/716a5be90f56ce6cd506bb53d5739d09374b1636/click/_termui_impl.py#L93 is how Click does this:
```
        self.is_hidden = not isatty(self.file)
```
Where `isatty` is a Click utility function: `from ._compat import isatty`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands,
https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543271000,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,543271000,MDEyOklzc3VlQ29tbWVudDU0MzI3MTAwMA==,9599,simonw,2019-10-17T17:05:59Z,2019-10-17T17:05:59Z,MEMBER,Looks like tqdm already does a TTY check here: https://github.com/tqdm/tqdm/blob/89b73bdc30c099c5b53725806e7edf3a121c9b3a/tqdm/std.py#L889-L890,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands,
https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-607010791,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,607010791,MDEyOklzc3VlQ29tbWVudDYwNzAxMDc5MQ==,9599,simonw,2020-04-01T03:45:48Z,2020-04-01T03:45:48Z,MEMBER,I'm happy with the recent work I did on this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands,
https://github.com/dogsheep/twitter-to-sqlite/issues/11#issuecomment-538711918,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/11,538711918,MDEyOklzc3VlQ29tbWVudDUzODcxMTkxOA==,9599,simonw,2019-10-06T04:54:17Z,2019-10-06T04:54:17Z,MEMBER,Shipped in 0.6. Here's the documentation: https://github.com/dogsheep/twitter-to-sqlite#capturing-tweets-in-real-time-with-track-and-follow,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503045221,Commands for recording real-time tweets from the streaming API,
https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855081,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12,542855081,MDEyOklzc3VlQ29tbWVudDU0Mjg1NTA4MQ==,9599,simonw,2019-10-16T19:26:56Z,2019-10-16T19:26:56Z,MEMBER,This may be the first case where I want to be able to repair existing databases rather than discarding their contents.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503053800,"Extract ""source"" into a separate lookup table",
https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855427,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12,542855427,MDEyOklzc3VlQ29tbWVudDU0Mjg1NTQyNw==,9599,simonw,2019-10-16T19:27:55Z,2019-10-16T19:27:55Z,MEMBER,I can do that by keeping `source` as a `TEXT` column but turning it into a non-enforced foreign key against a new `sources` table. Then I can run code that scans that column for any values beginning with a `<` and converts them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503053800,"Extract ""source"" into a separate lookup table",
https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542858025,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12,542858025,MDEyOklzc3VlQ29tbWVudDU0Mjg1ODAyNQ==,9599,simonw,2019-10-16T19:35:31Z,2019-10-16T19:36:09Z,MEMBER,Maybe this means I need an `upgrade` command to apply these kinds of migrations? Total feature creep!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503053800,"Extract ""source"" into a separate lookup table",
https://github.com/dogsheep/twitter-to-sqlite/issues/13#issuecomment-538804815,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/13,538804815,MDEyOklzc3VlQ29tbWVudDUzODgwNDgxNQ==,9599,simonw,2019-10-07T00:33:49Z,2019-10-07T00:33:49Z,MEMBER,Documentation: https://github.com/dogsheep/twitter-to-sqlite#retrieve-tweets-in-bulk,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503085013,statuses-lookup command,
https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112108,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17,541112108,MDEyOklzc3VlQ29tbWVudDU0MTExMjEwOA==,9599,simonw,2019-10-11T15:30:15Z,2019-10-11T15:30:15Z,MEMBER,It should delete the tables entirely. That way it will work even if the table schema has changed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505674949,import command should empty all archive-* tables first,
https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112588,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17,541112588,MDEyOklzc3VlQ29tbWVudDU0MTExMjU4OA==,9599,simonw,2019-10-11T15:31:30Z,2019-10-11T15:31:30Z,MEMBER,"No need for an option:

> This command will delete and recreate all of your `archive-*` tables every time you run it. If this is not what you want, run the command against a fresh SQLite database rather than running it again one that already exists.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505674949,import command should empty all archive-* tables first,
https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541118773,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18,541118773,MDEyOklzc3VlQ29tbWVudDU0MTExODc3Mw==,9599,simonw,2019-10-11T15:48:31Z,2019-10-11T15:48:31Z,MEMBER,https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-home_timeline,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505928530,Command to import home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541118934,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18,541118934,MDEyOklzc3VlQ29tbWVudDU0MTExODkzNA==,9599,simonw,2019-10-11T15:48:54Z,2019-10-11T15:48:54Z,MEMBER,Rate limit is tight: 15 requests every 15 mins!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505928530,Command to import home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541119834,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18,541119834,MDEyOklzc3VlQ29tbWVudDU0MTExOTgzNA==,9599,simonw,2019-10-11T15:51:22Z,2019-10-11T16:51:33Z,MEMBER,"In order to support multiple user timelines being saved in the same database, I'm going to import the tweets into the `tweets` table AND add a new `timeline_tweets` table recording that a specific tweet showed up in a specific user's timeline.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505928530,Command to import home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541141169,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18,541141169,MDEyOklzc3VlQ29tbWVudDU0MTE0MTE2OQ==,9599,simonw,2019-10-11T16:51:29Z,2019-10-11T16:51:29Z,MEMBER,Documented here: https://github.com/dogsheep/twitter-to-sqlite/blob/master/README.md#retrieving-tweets-from-your-home-timeline,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505928530,Command to import home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-541248629,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19,541248629,MDEyOklzc3VlQ29tbWVudDU0MTI0ODYyOQ==,9599,simonw,2019-10-11T22:48:56Z,2019-10-11T22:48:56Z,MEMBER,"`since_id` documented here: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-home_timeline

> Returns results with an ID greater than (that is, more recent than) the specified ID. There are limits to the number of Tweets which can be accessed through the API. If the limit of Tweets has occured since the since_id, the since_id will be forced to the oldest ID available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506087267,since_id support for home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542832952,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19,542832952,MDEyOklzc3VlQ29tbWVudDU0MjgzMjk1Mg==,9599,simonw,2019-10-16T18:30:11Z,2019-10-16T18:30:11Z,MEMBER,"The `--since` option will derive the `since_id` from the max ID in the `timeline_tweets` table:

    $ twitter-to-sqlite home-timeline --since

The `--since_id=xxx` option lets you specify that ID directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506087267,since_id support for home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542849963,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19,542849963,MDEyOklzc3VlQ29tbWVudDU0Mjg0OTk2Mw==,9599,simonw,2019-10-16T19:13:06Z,2019-10-16T19:13:06Z,MEMBER,Updated documentation: https://github.com/dogsheep/twitter-to-sqlite/blob/fced2a9b67d2cbdf9817f1eb75f7c28e413c963b/README.md#retrieving-tweets-from-your-home-timeline,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506087267,since_id support for home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527954898,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2,527954898,MDEyOklzc3VlQ29tbWVudDUyNzk1NDg5OA==,9599,simonw,2019-09-04T15:31:46Z,2019-09-04T15:31:46Z,MEMBER,I'm going to call this `twitter-to-sqlite user-timeline` to reflect the language used to describe the API endpoint: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833698,"""twitter-to-sqlite user-timeline"" command for pulling tweets by a specific user",
https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527955302,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2,527955302,MDEyOklzc3VlQ29tbWVudDUyNzk1NTMwMg==,9599,simonw,2019-09-04T15:32:39Z,2019-09-04T15:32:39Z,MEMBER,Rate limit is 900 / 15 minutes which is 1 call per second.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833698,"""twitter-to-sqlite user-timeline"" command for pulling tweets by a specific user",
https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527990908,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2,527990908,MDEyOklzc3VlQ29tbWVudDUyNzk5MDkwOA==,9599,simonw,2019-09-04T16:57:24Z,2019-09-04T16:57:24Z,MEMBER,"I just tried this using `max_id=` pagination as described in [Working with timelines](https://developer.twitter.com/en/docs/tweets/timelines/guides/working-with-timelines) and I got back all 17,759 of my tweets.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833698,"""twitter-to-sqlite user-timeline"" command for pulling tweets by a specific user",
https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-541388038,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20,541388038,MDEyOklzc3VlQ29tbWVudDU0MTM4ODAzOA==,9599,simonw,2019-10-13T05:31:58Z,2019-10-13T05:31:58Z,MEMBER,For favourites a `--stop_after=200` option is probably good enough.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506268945,--since support for various commands for refresh-by-cron,
https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-542854749,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20,542854749,MDEyOklzc3VlQ29tbWVudDU0Mjg1NDc0OQ==,9599,simonw,2019-10-16T19:26:01Z,2019-10-16T19:26:01Z,MEMBER,"I'm not going to do this for ""accounts that have followed me"" and ""new accounts that I have followed"" - instead I will recommend running the `friend_ids` and `followers_ids` commands on a daily basis since that data doesn't really change much by the hour.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506268945,--since support for various commands for refresh-by-cron,
https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-544335363,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20,544335363,MDEyOklzc3VlQ29tbWVudDU0NDMzNTM2Mw==,9599,simonw,2019-10-21T03:32:04Z,2019-10-21T03:32:04Z,MEMBER,"In case anyone is interested, here's an extract from the crontab I'm running these under at the moment:
```
1,11,21,31,41,51 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite user-timeline /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --since
2,7,12,17,22,27,32,37,42,47,52,57 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite home-timeline /home/ubuntu/timeline.db -a /home/ubuntu/auth.json --since
6,16,26,36,46,56 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite favorites /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --stop_after=50
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506268945,--since support for various commands for refresh-by-cron,
https://github.com/dogsheep/twitter-to-sqlite/issues/21#issuecomment-542333836,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/21,542333836,MDEyOklzc3VlQ29tbWVudDU0MjMzMzgzNg==,9599,simonw,2019-10-15T18:00:48Z,2019-10-15T18:00:48Z,MEMBER,I'll use `html.unescape()` for this: https://docs.python.org/3/library/html.html#html.unescape,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506432572,Fix &amp; escapes in tweet text,
https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543217890,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23,543217890,MDEyOklzc3VlQ29tbWVudDU0MzIxNzg5MA==,9599,simonw,2019-10-17T15:03:10Z,2019-10-17T15:03:10Z,MEMBER,"Thinking about this further: the concept of migrations may end up being in direct conflict with the `sqlite-utils` concept of creating tables on demand the first time they are used - and of creating table schemas automatically to fit the shape of the JSON that is being inserted into them.

I'm going to forge ahead anyway and build this because I think it will be an interesting exploration, but it's very likely this turns out to be a bad idea in the long run!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508190730,Extremely simple migration system,
https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543222239,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23,543222239,MDEyOklzc3VlQ29tbWVudDU0MzIyMjIzOQ==,9599,simonw,2019-10-17T15:12:33Z,2019-10-17T15:12:33Z,MEMBER,"Migrations will run only if you open a database that previously existed (as opposed to opening a brand new empty database).

This means that the first time you run a command against a fresh database, migrations will not run and the `migrations` table will not be created. The _second_ time you run any command against that database the migrations will execute and populate the `migrations` table.

This also means that each migration needs to be able to sanity check the database to see if it should run or not. If it should NOT run, it will do nothing but still be marked as having executed by adding to the `migrations` table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508190730,Extremely simple migration system,
https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543265058,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25,543265058,MDEyOklzc3VlQ29tbWVudDU0MzI2NTA1OA==,9599,simonw,2019-10-17T16:51:12Z,2019-10-17T16:51:12Z,MEMBER,"This migration function only runs if there is a table called `tweets` and the migration has not run before.

I think this can happen if the database has just been freshly created (by a command that fetches the user's user timeline for example) and is then run a SECOND time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508578780,Ensure migrations don't accidentally create foreign key twice,
https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543266947,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25,543266947,MDEyOklzc3VlQ29tbWVudDU0MzI2Njk0Nw==,9599,simonw,2019-10-17T16:56:06Z,2019-10-17T16:56:06Z,MEMBER,I wrote a test that proves that this is a problem. Should be an easy fix though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508578780,Ensure migrations don't accidentally create foreign key twice,
https://github.com/dogsheep/twitter-to-sqlite/issues/26#issuecomment-547713287,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/26,547713287,MDEyOklzc3VlQ29tbWVudDU0NzcxMzI4Nw==,9599,simonw,2019-10-30T02:36:13Z,2019-10-30T02:36:13Z,MEMBER,"Shipped this in 0.13: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.13

See also this Twitter thread: https://twitter.com/simonw/status/1189369677509623809","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",513074501,Command for importing mentions timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095217,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27,549095217,MDEyOklzc3VlQ29tbWVudDU0OTA5NTIxNw==,9599,simonw,2019-11-03T01:06:25Z,2019-11-03T01:06:25Z,MEMBER,"Wow, that `retweets_of_me` endpoint is almost completely useless:
```
$ twitter-to-sqlite fetch https://api.twitter.com/1.1/statuses/retweets_of_me.json
```
It returns my own tweets that have been retweeted, but with no indication at all of who retweeted them.

It looks like this needs to be combined with this API - https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-retweets-id - to fetch the details of up to 100 recent users who actually DID retweet an individual status. But that has a one-every-12-seconds rate limit on it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",514459062,retweets-of-me command,
https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095317,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27,549095317,MDEyOklzc3VlQ29tbWVudDU0OTA5NTMxNw==,9599,simonw,2019-11-03T01:08:10Z,2019-11-03T01:08:10Z,MEMBER,"Hmm... one thing that could be useful is that `retweets_of_me` can support a `--since` parameter - so if run frequently it should hopefully let us know which tweets we would need to run `statuses/retweets/:id.json` against.

I'm not sure if the `--since` parameter would show me a tweet that was previously retweeted but has now been retweeted again. I'll have a bit of a test and see.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",514459062,retweets-of-me command,
https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095463,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27,549095463,MDEyOklzc3VlQ29tbWVudDU0OTA5NTQ2Mw==,9599,simonw,2019-11-03T01:10:52Z,2019-11-03T01:10:52Z,MEMBER,"I imagine it won't, since the data I would be recording and then passing to `since_id` would be the highest ID of my own tweets that have been retweeted at least once. So it won't be able to spot if I should check for fresh retweets of a given tweet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",514459062,retweets-of-me command,
https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095641,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27,549095641,MDEyOklzc3VlQ29tbWVudDU0OTA5NTY0MQ==,9599,simonw,2019-11-03T01:12:58Z,2019-11-03T01:12:58Z,MEMBER,"It looks like Twitter really want you to subscribe to a premium API for this kind of thing and consume retweets via webhooks: https://developer.twitter.com/en/docs/accounts-and-users/subscribe-account-activity/api-reference

I'm going to give up on this for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",514459062,retweets-of-me command,
https://github.com/dogsheep/twitter-to-sqlite/issues/28#issuecomment-552133488,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/28,552133488,MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ4OA==,9599,simonw,2019-11-09T20:15:42Z,2019-11-09T20:15:42Z,MEMBER,Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",515658861,Add indexes to followers table,
https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129686,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29,552129686,MDEyOklzc3VlQ29tbWVudDU1MjEyOTY4Ng==,9599,simonw,2019-11-09T19:27:39Z,2019-11-09T19:27:39Z,MEMBER,I think this is fixed by the latest version of `sqlite-utils` - https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-12-1 - I'll bump the dependency.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518725064,`import` command fails on empty files,
https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129921,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29,552129921,MDEyOklzc3VlQ29tbWVudDU1MjEyOTkyMQ==,9599,simonw,2019-11-09T19:30:42Z,2019-11-09T19:30:42Z,MEMBER,"Confirmed, that seems to fix it:
```
(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js 
Traceback (most recent call last):
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite"", line 11, in <module>
    load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')()
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 764, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 717, in main
    rv = self.invoke(ctx)
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 555, in invoke
    return callback(*args, **kwargs)
  File ""/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py"", line 633, in import_
    archive.import_from_file(db, path.name, open(path, ""rb"").read())
  File ""/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/archive.py"", line 224, in import_from_file
    db[table_name].upsert_all(rows, hash_id=""pk"")
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1094, in upsert_all
    extracts=extracts,
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py"", line 961, in insert_all
    first_record = next(records)
StopIteration
(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ pip install -U sqlite-utils
Collecting sqlite-utils
  Using cached https://files.pythonhosted.org/packages/ee/a2/1b135010c7ac8e2d7545f659e9e6c6ede0f406f20b52e08d5817e1e31a9a/sqlite_utils-1.12.1-py3-none-any.whl
Requirement already satisfied, skipping upgrade: click in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (7.0)
Requirement already satisfied, skipping upgrade: tabulate in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (0.8.5)
Requirement already satisfied, skipping upgrade: click-default-group in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (1.2.2)
Installing collected packages: sqlite-utils
  Found existing installation: sqlite-utils 1.11
    Uninstalling sqlite-utils-1.11:
      Successfully uninstalled sqlite-utils-1.11
Successfully installed sqlite-utils-1.12.1
(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js 
(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518725064,`import` command fails on empty files,
https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552133449,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29,552133449,MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ0OQ==,9599,simonw,2019-11-09T20:15:15Z,2019-11-09T20:15:15Z,MEMBER,Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518725064,`import` command fails on empty files,
https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552134876,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29,552134876,MDEyOklzc3VlQ29tbWVudDU1MjEzNDg3Ng==,21148,jacobian,2019-11-09T20:33:38Z,2019-11-09T20:33:38Z,CONTRIBUTOR,❤️ thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518725064,`import` command fails on empty files,
https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-541493242,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,541493242,MDEyOklzc3VlQ29tbWVudDU0MTQ5MzI0Mg==,9599,simonw,2019-10-14T03:35:36Z,2019-10-14T03:35:36Z,MEMBER,"https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search,
https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543273540,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,543273540,MDEyOklzc3VlQ29tbWVudDU0MzI3MzU0MA==,9599,simonw,2019-10-17T17:12:51Z,2019-10-17T17:12:51Z,MEMBER,"Just importing tweets here isn't enough - how are we supposed to know which tweets were imported by which search?

So I think the right thing to do here is to also create a `search_runs` table, which records each individual run of this tool (with a timestamp and the search terms used). Then have a `search_runs_tweets` m2m table which shows which Tweets were found by that search.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search,
https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543290744,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,543290744,MDEyOklzc3VlQ29tbWVudDU0MzI5MDc0NA==,9599,simonw,2019-10-17T17:57:14Z,2019-10-17T17:57:14Z,MEMBER,I have a working command now. I'm going to ship it early because it could do with some other people trying it out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search,
https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549096321,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,549096321,MDEyOklzc3VlQ29tbWVudDU0OTA5NjMyMQ==,9599,simonw,2019-11-03T01:27:55Z,2019-11-03T01:28:17Z,MEMBER,"It would be neat if this could support `--since`, with that argument automatically finding the maximum tweet ID from a previous search that used the same exact arguments (using the `search_runs` table).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search,
https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549226399,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,549226399,MDEyOklzc3VlQ29tbWVudDU0OTIyNjM5OQ==,9599,simonw,2019-11-04T05:11:57Z,2019-11-04T05:11:57Z,MEMBER,I'm going to add a `hash` column to `search_runs` to support that. It's going to be the sha1 hash of the key-ordered JSON of the search arguments used by that run. Then `--since` can look for an identical hash and use it to identify the highest last fetched tweet to use in `since_id`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search,
https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549228535,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,549228535,MDEyOklzc3VlQ29tbWVudDU0OTIyODUzNQ==,9599,simonw,2019-11-04T05:31:55Z,2019-11-04T05:31:55Z,MEMBER,Documented here: https://github.com/dogsheep/twitter-to-sqlite/blob/801c0c2daf17d8abce9dcb5d8d610410e7e25dbe/README.md#running-searches,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search,
https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552131798,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30,552131798,MDEyOklzc3VlQ29tbWVudDU1MjEzMTc5OA==,9599,simonw,2019-11-09T19:54:45Z,2019-11-09T19:54:45Z,MEMBER,Good catch - not sure how that bug crept in. Removing line 116 looks like the right fix to me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518739697,`followers` fails because `transform_user` is called twice,
https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552133468,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30,552133468,MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ2OA==,9599,simonw,2019-11-09T20:15:27Z,2019-11-09T20:15:27Z,MEMBER,Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518739697,`followers` fails because `transform_user` is called twice,
https://github.com/dogsheep/twitter-to-sqlite/issues/31#issuecomment-1251845216,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31,1251845216,IC_kwDODEm0Qs5KnaRg,150986,dckc,2022-09-20T05:05:03Z,2022-09-20T05:05:03Z,NONE,"yay! Thanks a bunch for the `twitter-to-sqlite friends` command!

The twitter ""Download an archive of your data"" feature doesn't include who I follow, so this is particularly handy.

The whole Dogsheep thing is great :) I've written about similar things under [cloud-services](https://www.madmode.com/search/label/cloud-services/):
 - 2021: [Closet Librarian Approach to Cloud Services](https://www.madmode.com/2021/closet-librarian-approach-cloud-services.html)
 - 2015: [jukekb \- Browse iTunes libraries and upload playlists to Google Music](https://www.madmode.com/2015/jukekb)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520508502,"""friends"" command (similar to ""followers"")",
https://github.com/dogsheep/twitter-to-sqlite/issues/31#issuecomment-552135263,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31,552135263,MDEyOklzc3VlQ29tbWVudDU1MjEzNTI2Mw==,9599,simonw,2019-11-09T20:38:35Z,2019-11-09T20:38:35Z,MEMBER,Command still needs documentation and a bit more testing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520508502,"""friends"" command (similar to ""followers"")",
https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601861908,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34,601861908,MDEyOklzc3VlQ29tbWVudDYwMTg2MTkwOA==,9599,simonw,2020-03-20T18:56:44Z,2020-03-20T18:56:44Z,MEMBER,"Could this be a bug in `sqlite-utils`? This table has a primary key, so why is it running a query on `rowid = ?`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585266763,IndexError running user-timeline command,
https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601871579,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34,601871579,MDEyOklzc3VlQ29tbWVudDYwMTg3MTU3OQ==,9599,simonw,2020-03-20T19:20:52Z,2020-03-20T19:20:52Z,MEMBER,Cause was the redefining of how `upsert()` works in sqlite-utils 2.x.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585266763,IndexError running user-timeline command,
https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875405,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35,601875405,MDEyOklzc3VlQ29tbWVudDYwMTg3NTQwNQ==,9599,simonw,2020-03-20T19:30:09Z,2020-03-20T19:32:33Z,MEMBER,"I'm going to redesign the command a little. Right now it defaults to fetching the authenticated user's timeline and allows `--screen_name` to be used to fetch for one other user instead.

I think the design of `users-lookup` is much better. I'm going to copy that:

```
twitter-to-sqlite user-timeline users.db simonw cleopaws
twitter-to-sqlite user-timeline users.db 12497 3166449535 --ids
```
It will also accept `--since` and `--sql` and `--alter` and `--since_id`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585282212,twitter-to-sqlite user-timeline [screen_names] --sql / --attach,
https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875524,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35,601875524,MDEyOklzc3VlQ29tbWVudDYwMTg3NTUyNA==,9599,simonw,2020-03-20T19:30:27Z,2020-03-20T19:30:27Z,MEMBER,"I can give it a snazzier progress bar to, as requested by #10.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585282212,twitter-to-sqlite user-timeline [screen_names] --sql / --attach,
https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601876051,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35,601876051,MDEyOklzc3VlQ29tbWVudDYwMTg3NjA1MQ==,9599,simonw,2020-03-20T19:31:43Z,2020-03-20T19:31:43Z,MEMBER,I'm going to keep `--screen_name` working for backwards compatibility. Using `hidden=True` on it should hide it from the help output.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585282212,twitter-to-sqlite user-timeline [screen_names] --sql / --attach,
https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601890795,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35,601890795,MDEyOklzc3VlQ29tbWVudDYwMTg5MDc5NQ==,9599,simonw,2020-03-20T20:13:08Z,2020-03-20T20:13:08Z,MEMBER,Needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585282212,twitter-to-sqlite user-timeline [screen_names] --sql / --attach,
https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601892338,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35,601892338,MDEyOklzc3VlQ29tbWVudDYwMTg5MjMzOA==,9599,simonw,2020-03-20T20:17:00Z,2020-03-20T20:17:00Z,MEMBER,Released as 0.18.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585282212,twitter-to-sqlite user-timeline [screen_names] --sql / --attach,
https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509578,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37,615509578,MDEyOklzc3VlQ29tbWVudDYxNTUwOTU3OA==,9599,simonw,2020-04-17T23:36:00Z,2020-04-17T23:36:00Z,MEMBER,"```
$ twitter-to-sqlite user-timeline doggo.db doggoenthuonetuh
Traceback (most recent call last):
...
  File ""/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py"", line 272, in transform_user
    user[""created_at""] = parser.parse(user[""created_at""])
KeyError: 'created_at'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585353598,"Handle ""User not found"" error",
https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509803,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37,615509803,MDEyOklzc3VlQ29tbWVudDYxNTUwOTgwMw==,9599,simonw,2020-04-17T23:36:40Z,2020-04-17T23:36:40Z,MEMBER,"I'm going to print a warning to stderr, skip and continue - because if you have 100 screen names and only one of them is invalid you should still execute for the other 99.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585353598,"Handle ""User not found"" error",
https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615510361,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37,615510361,MDEyOklzc3VlQ29tbWVudDYxNTUxMDM2MQ==,9599,simonw,2020-04-17T23:38:27Z,2020-04-17T23:38:27Z,MEMBER,That's a bit tricky since I'd have to rewrite the internals of a bunch of other commands. For the moment I'll exit the script with an error but at least it will be a decent error!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585353598,"Handle ""User not found"" error",
https://github.com/dogsheep/twitter-to-sqlite/issues/38#issuecomment-601939656,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/38,601939656,MDEyOklzc3VlQ29tbWVudDYwMTkzOTY1Ng==,9599,simonw,2020-03-20T22:37:04Z,2020-03-20T22:37:04Z,MEMBER,"New formatting:
```
@Alaska_DHSS      [####################################]  3205
@BWhippleKS       [####################################]  1
@CAPublicHealth   [####################################]  2
@CDPHE            [####################################]  67
@CHFSKy           [####################################]  3216
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585359363,Screen name display for user-timeline is uneven,
https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606304837,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606304837,MDEyOklzc3VlQ29tbWVudDYwNjMwNDgzNw==,9599,simonw,2020-03-30T23:27:50Z,2020-03-30T23:29:31Z,MEMBER,"One option would be something like this:
```sql
select max(id) from tweets
where user = ?
and not exists (select id from tweets where retweeted_status = id)
and not exists (select id from tweets where quoted_status = id)
and not exists (select id from tweets where in_reply_to_status_id = id)
```
Might be a good idea to index those columns (after confirming that doing so would indeed speed up the query).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets,
https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606305701,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606305701,MDEyOklzc3VlQ29tbWVudDYwNjMwNTcwMQ==,9599,simonw,2020-03-30T23:30:27Z,2020-03-30T23:30:27Z,MEMBER,A better alternative would be to maintain a separate table with the last seen since value for when we ran `user-timeline` for any specific user.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets,
https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606309165,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606309165,MDEyOklzc3VlQ29tbWVudDYwNjMwOTE2NQ==,9599,simonw,2020-03-30T23:41:31Z,2020-03-30T23:41:31Z,MEMBER,I like the separate `user_timeline_since` table solution.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets,
https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606824992,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606824992,MDEyOklzc3VlQ29tbWVudDYwNjgyNDk5Mg==,9599,simonw,2020-03-31T19:24:23Z,2020-03-31T19:24:23Z,MEMBER,"The `--since` option is actually used by four commands:

* `user-timeline`
* `home-timeline`
* `mentions-timeline`
* `search`

All of them use the same `fetch_timeline()` utility function under the hood. I should move the logic that looks up the last `since_id` into that shared function.

Question: should I have a table for each of those four methods or a single table that is used by them all? I'm leaning towards four separate tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets,
https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606843224,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606843224,MDEyOklzc3VlQ29tbWVudDYwNjg0MzIyNA==,9599,simonw,2020-03-31T19:59:11Z,2020-03-31T20:06:32Z,MEMBER,"Or... have a single `since_ids` table to track since values, and have its primary key be a string that looks something like this:

`user:123145`
`home:23441`
`mentions:23425`
`search:99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d`

That last example would use the hash generated here:

https://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/cli.py#L792-L808","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets,
https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606844521,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606844521,MDEyOklzc3VlQ29tbWVudDYwNjg0NDUyMQ==,9599,simonw,2020-03-31T20:01:39Z,2020-03-31T20:01:39Z,MEMBER,"I think `utils.fetch_timeline()` grows a new argument, `since_key`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets,
https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850008,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606850008,MDEyOklzc3VlQ29tbWVudDYwNjg1MDAwOA==,9599,simonw,2020-03-31T20:13:59Z,2020-04-01T00:23:00Z,MEMBER,"Table design for `since_ids` table:

type | key | since_id
--- | --- | ---
1 | 124324 | 2347239847293
2 | 99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d | 2125947927344

Primary compound key on `(category, key)`

`type` is also a foreign key to a `since_id_types` table with `id` and `name` columns (probably created using https://sqlite-utils.readthedocs.io/en/stable/python-api.html#working-with-lookup-tables )","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets,
https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850453,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606850453,MDEyOklzc3VlQ29tbWVudDYwNjg1MDQ1Mw==,9599,simonw,2020-03-31T20:14:58Z,2020-04-01T03:03:50Z,MEMBER,Actually I'll hard-code the population of `since_id_types` to get known ID constants.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets,
https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606998669,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606998669,MDEyOklzc3VlQ29tbWVudDYwNjk5ODY2OQ==,9599,simonw,2020-04-01T02:57:36Z,2020-04-01T02:57:36Z,MEMBER,"The tricky thing here is thinking about the interaction between the recorded since_id and a desire to run the initial import.

The first time you run `twitter-to-sqlite user-timeline db.db username` we want to fetch as many tweets from that user as possible - probably around 3,200 before the API limitations cut us off.

We need to record the maximum ID from those as the `since_id` - which we will see on the very first page we paginate through. That way next time we run the command with `--since` we will only fetch new tweets.

But what happens if our initial import is cancelled after only a few tweets? We risk never pulling in the rest of the tweets.

Not sure if I need to solve this at all or if I should instead trust users to run the command a second time without `--since` if they think they didn't retrieve anything the first time.

I had considered letting `--stop_after=` over-ride `--since` but that doesn't actually make sense - if you send a since_id to the Twitter API you'll never get back more tweets than exist after that ID, so the `--stop_after` would not make a meaningful difference.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets,
https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607003655,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,607003655,MDEyOklzc3VlQ29tbWVudDYwNzAwMzY1NQ==,9599,simonw,2020-04-01T03:18:00Z,2020-04-01T03:18:00Z,MEMBER,I've got this working for the `user-timeline` command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets,
https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607010634,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,607010634,MDEyOklzc3VlQ29tbWVudDYwNzAxMDYzNA==,9599,simonw,2020-04-01T03:45:16Z,2020-04-01T03:45:16Z,MEMBER,"OK, fix is applied to everything now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets,
https://github.com/dogsheep/twitter-to-sqlite/issues/4#issuecomment-527682713,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4,527682713,MDEyOklzc3VlQ29tbWVudDUyNzY4MjcxMw==,9599,simonw,2019-09-03T23:48:57Z,2019-09-03T23:48:57Z,MEMBER,"One interesting challenge here is that the JSON format for tweets in the archive is subtly different from the JSON format currently returned by the API.

If we want to keep the tweets in the same database table (which feels like the right thing to me) we'll need to handle this.

One thing we can do is have a column for `from_archive` which is set to 1 for tweets that were recovered from the archive.

We can also ensure that tweets from the API always over-write the version that came from the archive (using `.upsert()`) while tweets from the archive use `.insert(..., ignore=True)` to avoid over-writing a better version that came from the API.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488835586,Command for importing data from a Twitter Export file,
https://github.com/dogsheep/twitter-to-sqlite/issues/4#issuecomment-540879620,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4,540879620,MDEyOklzc3VlQ29tbWVudDU0MDg3OTYyMA==,9599,simonw,2019-10-11T02:59:16Z,2019-10-11T02:59:16Z,MEMBER,Also import ad preferences and all that other junk.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488835586,Command for importing data from a Twitter Export file,
https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307019,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40,606307019,MDEyOklzc3VlQ29tbWVudDYwNjMwNzAxOQ==,9599,simonw,2020-03-30T23:34:27Z,2020-03-30T23:34:27Z,MEMBER,"The count properties available for a user are:

* followers_count
* friends_count
* listed_count
* favourites_count
* statuses_count

May as well track history for all of them? Should be pretty cheap to store.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590669793,Feature: record history of follower counts,
https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307376,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40,606307376,MDEyOklzc3VlQ29tbWVudDYwNjMwNzM3Ng==,9599,simonw,2020-03-30T23:35:40Z,2020-03-30T23:39:15Z,MEMBER,"I think five separate tables:

* followers_count_history
* friends_count_history
* listed_count_history
* favourites_count_history
* statuses_count_history

Each with the following structure:

* datetime (ISO UTC)
* user (ID, foreign key to users)
* count (integer)

I'm tempted to have a compound primary key here - user, datetime
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590669793,Feature: record history of follower counts,
https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011421,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40,607011421,MDEyOklzc3VlQ29tbWVudDYwNzAxMTQyMQ==,9599,simonw,2020-04-01T03:47:37Z,2020-04-01T03:55:08Z,MEMBER,"Actually a single table with a `type` integer ID referencing a `count_history_types` table would better match the way I implemented the `since_ids` table:

https://github.com/dogsheep/twitter-to-sqlite/blob/4b6c8d8c1cc6fefdb566ec8506157133f47c569a/twitter_to_sqlite/utils.py#L331-L341

In which case the compound primary key would be `type`, `user`, `datetime`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590669793,Feature: record history of follower counts,
https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011972,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40,607011972,MDEyOklzc3VlQ29tbWVudDYwNzAxMTk3Mg==,9599,simonw,2020-04-01T03:49:02Z,2020-04-01T03:50:01Z,MEMBER,"I want the datetime value to look like `2020-04-01T03:34:58+00:00` (the format returned by the Twitter API which I am storing in other tables at the moment).
```
>>> datetime.utcnow().isoformat().split('.')[0] + '+00:00'
'2020-04-01T03:49:52+00:00'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590669793,Feature: record history of follower counts,
https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607019151,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40,607019151,MDEyOklzc3VlQ29tbWVudDYwNzAxOTE1MQ==,9599,simonw,2020-04-01T04:11:10Z,2020-04-01T04:11:10Z,MEMBER,"In testing this collects a LOT of data. I'm going to skip tracking favourites_count and statuses_count and just track followers, friends and listed instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590669793,Feature: record history of follower counts,
https://github.com/dogsheep/twitter-to-sqlite/issues/43#issuecomment-615513491,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/43,615513491,MDEyOklzc3VlQ29tbWVudDYxNTUxMzQ5MQ==,9599,simonw,2020-04-17T23:48:28Z,2020-04-17T23:48:28Z,MEMBER,Released in 0.21.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602176870,"""twitter-to-sqlite lists"" command for retrieving a user's owned lists",
https://github.com/dogsheep/twitter-to-sqlite/issues/45#issuecomment-616029262,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45,616029262,MDEyOklzc3VlQ29tbWVudDYxNjAyOTI2Mg==,9599,simonw,2020-04-19T04:39:21Z,2020-04-19T04:39:21Z,MEMBER,"![44714E00-8CC5-46CD-9E48-1F4DD148FCC8](https://user-images.githubusercontent.com/9599/79679696-09b6d300-81bd-11ea-80e4-0653d92e4f58.jpeg)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602619330,Use raise_for_status() everywhere,
https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47,645512127,MDEyOklzc3VlQ29tbWVudDY0NTUxMjEyNw==,9599,simonw,2020-06-17T17:24:22Z,2020-06-17T17:24:22Z,MEMBER,"That means your version of SQLite is old enough that it doesn't support the FTS5 extension.

Could you share what operating system you're running, and what the output is that you get from running this?

    python -c 'import sqlite3; print(sqlite3.connect("":memory:"").execute(""select sqlite_version()"").fetchone()[0])'

I can teach this tool to fall back on FTS4 if FTS5 isn't available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639542974,Fall back to FTS4 if FTS5 is not available,
https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645515103,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47,645515103,MDEyOklzc3VlQ29tbWVudDY0NTUxNTEwMw==,73579,hpk42,2020-06-17T17:30:01Z,2020-06-17T17:30:01Z,NONE,"It's the one with python3.7::

    >>> sqlite3.sqlite_version
    '3.11.0'

 
On Wed, Jun 17, 2020 at 10:24 -0700, Simon Willison wrote:

> That means your version of SQLite is old enough that it doesn't support the FTS5 extension.
> 
> Could you share what operating system you're running, and what the output is that you get from running this?
> 
>     python -c 'import sqlite3; print(sqlite3.connect("":memory:"").execute(""select sqlite_version()"").fetchone()[0])'
> 
> I can teach this tool to fall back on FTS4 if FTS5 isn't available.
> 
> -- 
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub:
> https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639542974,Fall back to FTS4 if FTS5 is not available,
https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645599881,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47,645599881,MDEyOklzc3VlQ29tbWVudDY0NTU5OTg4MQ==,9599,simonw,2020-06-17T20:13:48Z,2020-06-17T20:13:48Z,MEMBER,"I've now figured out how to compile specific SQLite versions to help replicate this problem: https://github.com/simonw/til/blob/master/sqlite/ld-preload.md

Next step: replicate the problem!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639542974,Fall back to FTS4 if FTS5 is not available,
https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662626901,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48,662626901,MDEyOklzc3VlQ29tbWVudDY2MjYyNjkwMQ==,9599,simonw,2020-07-22T18:54:53Z,2020-07-22T18:54:53Z,MEMBER,I'm going to use a GitHub Action to run `npx markdown-toc README.md -i`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663976976,Add a table of contents to the README,
https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662630868,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48,662630868,MDEyOklzc3VlQ29tbWVudDY2MjYzMDg2OA==,9599,simonw,2020-07-22T19:03:02Z,2020-07-22T19:03:02Z,MEMBER,Done!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663976976,Add a table of contents to the README,
https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-663143160,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48,663143160,MDEyOklzc3VlQ29tbWVudDY2MzE0MzE2MA==,9599,simonw,2020-07-23T17:46:07Z,2020-07-23T17:46:07Z,MEMBER,"Frustratingly, these links don't work on PyPI: https://pypi.org/project/twitter-to-sqlite/

There's an issue about that here: https://github.com/pypa/readme_renderer/issues/169","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663976976,Add a table of contents to the README,
https://github.com/dogsheep/twitter-to-sqlite/issues/5#issuecomment-527684202,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/5,527684202,MDEyOklzc3VlQ29tbWVudDUyNzY4NDIwMg==,9599,simonw,2019-09-03T23:56:28Z,2019-09-03T23:56:28Z,MEMBER,I previously used betamax here: https://github.com/simonw/github-contents/blob/master/test_github_contents.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488874815,Write tests that simulate the Twitter API,
https://github.com/dogsheep/twitter-to-sqlite/issues/50#issuecomment-690860653,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50,690860653,MDEyOklzc3VlQ29tbWVudDY5MDg2MDY1Mw==,370930,mikepqr,2020-09-11T04:04:08Z,2020-09-11T04:04:08Z,CONTRIBUTOR,"There's probably a nicer way of doing (hence this is a comment rather than a PR), but this appears to fix it:
```diff
--- a/twitter_to_sqlite/utils.py
+++ b/twitter_to_sqlite/utils.py
@@ -181,6 +181,7 @@ def fetch_timeline(
     args[""tweet_mode""] = ""extended""
     min_seen_id = None
     num_rate_limit_errors = 0
+    seen_count = 0
     while True:
         if min_seen_id is not None:
             args[""max_id""] = min_seen_id - 1
@@ -208,6 +209,7 @@ def fetch_timeline(
             yield tweet
         min_seen_id = min(t[""id""] for t in tweets)
         max_seen_id = max(t[""id""] for t in tweets)
+        seen_count += len(tweets)
         if last_since_id is not None:
             max_seen_id = max((last_since_id, max_seen_id))
             last_since_id = max_seen_id
@@ -217,7 +219,9 @@ def fetch_timeline(
                 replace=True,
             )
         if stop_after is not None:
-            break
+            if seen_count >= stop_after:
+                break
+            args[""count""] = min(args[""count""], stop_after - seen_count)
         time.sleep(sleep)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",698791218,"favorites --stop_after=N stops after min(N, 200)",
https://github.com/dogsheep/twitter-to-sqlite/issues/50#issuecomment-691501132,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50,691501132,MDEyOklzc3VlQ29tbWVudDY5MTUwMTEzMg==,706257,bcongdon,2020-09-12T14:48:10Z,2020-09-12T14:48:10Z,NONE,"This seems to be an issue even with larger values of `--stop_after`:

```
$ twitter-to-sqlite favorites twitter.db --stop_after=2000
Importing favorites  [####################################]  198
$
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",698791218,"favorites --stop_after=N stops after min(N, 200)",
https://github.com/dogsheep/twitter-to-sqlite/issues/52#issuecomment-729484478,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/52,729484478,MDEyOklzc3VlQ29tbWVudDcyOTQ4NDQ3OA==,4169772,fatihky,2020-11-18T07:12:45Z,2020-11-18T07:12:45Z,NONE,I'm so sorry that you already have `--since_id` option and that's enough for the case I've mentioned. Thank you for this excellent tool!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",745393298,Discussion: Adding support for fetching only fresh tweets,
https://github.com/dogsheep/twitter-to-sqlite/issues/53#issuecomment-748436453,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/53,748436453,MDEyOklzc3VlQ29tbWVudDc0ODQzNjQ1Mw==,27,anotherjesse,2020-12-19T07:47:01Z,2020-12-19T07:47:01Z,NONE,"I think this should probably be closed as won't fix.

Attempting to make a patch for this I realized that the since_id would limit to tweets posted since that since_id, not when it was favorited.  So favoriting something in the older would be missed if you used `--since` with a cron script

Better to just use `--stop_after` set to a couple hundred","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771324837,--since support for favorites,
https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-1370786026,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54,1370786026,IC_kwDODEm0Qs5RtIjq,6764957,swyxio,2023-01-04T11:06:44Z,2023-01-04T11:06:44Z,NONE,"as of 2023 it appears that `tweets: not yet implemented` is happening.. pretty important for a twitter export functionality!

this seems an impossible task with twitter liable to switch this around every other day","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779088071,Archive import appears to be broken on recent exports,
https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-754721153,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54,754721153,MDEyOklzc3VlQ29tbWVudDc1NDcyMTE1Mw==,21148,jacobian,2021-01-05T15:51:09Z,2021-01-05T15:51:09Z,CONTRIBUTOR,"Correction: the failure is on `lists-member.js` (I was thrown by the `block` variable name, but that's just a coincidence)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779088071,Archive import appears to be broken on recent exports,
https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-754729035,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54,754729035,MDEyOklzc3VlQ29tbWVudDc1NDcyOTAzNQ==,21148,jacobian,2021-01-05T16:03:29Z,2021-01-05T16:03:29Z,CONTRIBUTOR,"I was able to fix this, at least enough to get _my_ archive to import. Not sure if there's more work to be done here or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779088071,Archive import appears to be broken on recent exports,
https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-767888743,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54,767888743,MDEyOklzc3VlQ29tbWVudDc2Nzg4ODc0Mw==,19328961,henry501,2021-01-26T23:07:41Z,2021-01-26T23:07:41Z,NONE,"My import got much further with the applied fixes than 0.21.3, but not 100%. I do appear to have all of the tweets imported at least. 
Not sure when I'll have a chance to look further to try to fix or see what didn't make it into the import.

Here's my output:

```
direct-messages-group: not yet implemented
branch-links: not yet implemented
periscope-expired-broadcasts: not yet implemented
direct-messages: not yet implemented
mute: not yet implemented
periscope-comments-made-by-user: not yet implemented
periscope-ban-information: not yet implemented
periscope-profile-description: not yet implemented
screen-name-change: not yet implemented
manifest: not yet implemented
fleet: not yet implemented
user-link-clicks: not yet implemented
periscope-broadcast-metadata: not yet implemented
contact: not yet implemented
fleet-mute: not yet implemented
device-token: not yet implemented
protected-history: not yet implemented
direct-message-mute: not yet implemented
Traceback (most recent call last):
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/bin/twitter-to-sqlite"", line 33, in <module>
    sys.exit(load_entry_point('twitter-to-sqlite==0.21.3', 'console_scripts', 'twitter-to-sqlite')())
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/cli.py"", line 772, in import_
    archive.import_from_file(db, filepath.name, open(filepath, ""rb"").read())
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py"", line 233, in import_from_file
    to_insert = transformer(data)
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py"", line 21, in callback
    return {filename: [fn(item) for item in data]}
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py"", line 21, in <listcomp>
    return {filename: [fn(item) for item in data]}
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py"", line 88, in ageinfo
    return item[""ageMeta""][""ageInfo""]
KeyError: 'ageInfo'

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779088071,Archive import appears to be broken on recent exports,
https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-927312650,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54,927312650,IC_kwDODEm0Qs43RasK,2182,danp,2021-09-26T14:09:51Z,2021-09-26T14:09:51Z,NONE,"Similar trouble with ageinfo using 0.22. Here's what my ageinfo.js file looks like:

```
window.YTD.ageinfo.part0 = [
  {
    ""ageMeta"" : { }
  }
]
```

Commenting out the registration for ageinfo in archive.py gets my archive to import.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779088071,Archive import appears to be broken on recent exports,
https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769957751,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56,769957751,MDEyOklzc3VlQ29tbWVudDc2OTk1Nzc1MQ==,9599,simonw,2021-01-29T17:59:40Z,2021-01-29T17:59:40Z,MEMBER,"This is interesting - how did you create that initial table? Was this using the `twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip` command, or something else?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",796736607,Not all quoted statuses get fetched?,
https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769973212,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56,769973212,MDEyOklzc3VlQ29tbWVudDc2OTk3MzIxMg==,42315895,gsajko,2021-01-29T18:29:02Z,2021-01-29T18:31:55Z,NONE,"I think it was with `twitter-to-sqlite home-timeline home.db -a  auth.json --since`
and Im using only this command to grab tweets 

from cron tab
`2,7,12,17,22,27,32,37,42,47,52,57 * * * * run-one /home/gsajko/miniconda3/bin/twitter-to-sqlite home-timeline /home/gsajko/work/custom_twitter_feed/home.db -a /home/gsajko/work/custom_twitter_feed/auth/auth.json --since`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",796736607,Not all quoted statuses get fetched?,
https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-772408273,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56,772408273,MDEyOklzc3VlQ29tbWVudDc3MjQwODI3Mw==,42315895,gsajko,2021-02-03T10:36:36Z,2021-02-03T10:36:36Z,NONE,"I figured it out.
Those tweets are in database, because somebody quote tweeted them, or retweeted them.
And if you grab quoted tweet or reweeted tweet from other tweet json, It doesn't grab all of the details.

So if someone quote tweeted a quote tweet, the second quote tweet won't have `quoted_status`. 

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",796736607,Not all quoted statuses get fetched?,
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-860063190,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,860063190,MDEyOklzc3VlQ29tbWVudDg2MDA2MzE5MA==,232237,stiles,2021-06-12T14:46:44Z,2021-06-12T14:46:44Z,NONE,I'm having the same issue (same versions of python and twitter-to-sqlite). It's the `user-timeline` command. Other commands are working. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902327457,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,902327457,IC_kwDODEm0Qs41yGyh,9599,simonw,2021-08-19T23:53:25Z,2021-08-19T23:53:25Z,MEMBER,I'm getting this too. Looking into it now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328369,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,902328369,IC_kwDODEm0Qs41yHAx,9599,simonw,2021-08-19T23:56:26Z,2021-08-19T23:56:26Z,MEMBER,"https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-user_timeline says the API has been replaced by the new v2 one, but it should still work - and the `since_id` parameter is still documented on that page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328760,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,902328760,IC_kwDODEm0Qs41yHG4,9599,simonw,2021-08-19T23:57:41Z,2021-08-19T23:57:41Z,MEMBER,"Weird, added debug code and got this: `{'screen_name': 'simonw', 'count': 200, 'since_id': 'False', 'tweet_mode': 'extended'}` -  so maybe it's a `twitter-to-sqlite` bug where somehow the string `False` is being passed somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329455,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,902329455,IC_kwDODEm0Qs41yHRv,9599,simonw,2021-08-19T23:59:56Z,2021-08-19T23:59:56Z,MEMBER,"This looks like the bug to me:

https://github.com/dogsheep/twitter-to-sqlite/blob/197e69cec40052c423a5ed071feb5f7cccea41b9/twitter_to_sqlite/cli.py#L239-L241

`type=str, default=False`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329884,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,902329884,IC_kwDODEm0Qs41yHYc,9599,simonw,2021-08-20T00:01:05Z,2021-08-20T00:01:05Z,MEMBER,Maybe Click changed something which meant that this broke things when it didn't used to?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/dogsheep/twitter-to-sqlite/issues/58#issuecomment-910121331,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/58,910121331,IC_kwDODEm0Qs42P1lz,42904,rubenv,2021-09-01T09:49:33Z,2021-09-01T09:49:33Z,CONTRIBUTOR,"Found the cause, it's the other commands. PR #59 submitted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",984939366,"Error: Use either --since or --since_id, not both - still broken",
https://github.com/dogsheep/twitter-to-sqlite/issues/60#issuecomment-1279249898,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/60,1279249898,IC_kwDODEm0Qs5MP83q,7908073,chapmanjacobd,2022-10-14T16:58:26Z,2022-10-14T16:58:26Z,NONE,"You could try using `msys2`. I've had better luck running python CLIs within that system on Windows.

Here is a guide: https://github.com/chapmanjacobd/lb/blob/main/Windows.md#prep","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1063982712,Execution on Windows,
https://github.com/dogsheep/twitter-to-sqlite/issues/61#issuecomment-1297201971,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/61,1297201971,IC_kwDODEm0Qs5NUbsz,3153638,Profpatsch,2022-10-31T14:47:58Z,2022-10-31T14:47:58Z,NONE,There’s also a limit of 3200 tweets. I wonder if that can be circumvented somehow.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077560091,"Data Pull fails for ""Essential"" level access to the Twitter API (for Documentation)",
https://github.com/dogsheep/twitter-to-sqlite/issues/62#issuecomment-1001222213,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/62,1001222213,IC_kwDODEm0Qs47rXBF,6764957,swyxio,2021-12-26T17:59:25Z,2021-12-26T17:59:25Z,NONE,just confirmed that this error does not occur when i use my public main account. gets more interesting!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1088816961,KeyError: 'created_at' for private accounts?,
https://github.com/dogsheep/twitter-to-sqlite/issues/62#issuecomment-1049775451,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/62,1049775451,IC_kwDODEm0Qs4-kk1b,43036882,miuku,2022-02-24T11:43:31Z,2022-02-24T11:43:31Z,NONE,i seem to have fixed this issue by applying for [elevated API access](https://developer.twitter.com/en/portal/products/elevated),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1088816961,KeyError: 'created_at' for private accounts?,
https://github.com/dogsheep/twitter-to-sqlite/issues/62#issuecomment-1050123919,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/62,1050123919,IC_kwDODEm0Qs4-l56P,6764957,swyxio,2022-02-24T18:10:18Z,2022-02-24T18:10:18Z,NONE,gonna close this for now since i'm not actively working on it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1088816961,KeyError: 'created_at' for private accounts?,
https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-529239307,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8,529239307,MDEyOklzc3VlQ29tbWVudDUyOTIzOTMwNw==,9599,simonw,2019-09-08T20:36:49Z,2019-09-08T20:36:49Z,MEMBER,"`--attach` can optionally take a name for the database connection alias like this:

    $ twitter-to-sqlite users-lookup users.db --attach foo:attending.db ...

If you omit the `alias:` bit the stem of the database (without the file extension) will be used.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",490803176,--sql and --attach options for feeding commands from SQL queries,
https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-529240286,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8,529240286,MDEyOklzc3VlQ29tbWVudDUyOTI0MDI4Ng==,9599,simonw,2019-09-08T20:48:33Z,2019-09-08T20:48:33Z,MEMBER,"```ATTACH DATABASE ""file:blah.db?mode=ro"" AS foo```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",490803176,--sql and --attach options for feeding commands from SQL queries,
https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-530417631,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8,530417631,MDEyOklzc3VlQ29tbWVudDUzMDQxNzYzMQ==,9599,simonw,2019-09-11T14:52:44Z,2019-09-14T19:09:22Z,MEMBER,- [x] This needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",490803176,--sql and --attach options for feeding commands from SQL queries,
https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-531404891,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8,531404891,MDEyOklzc3VlQ29tbWVudDUzMTQwNDg5MQ==,9599,simonw,2019-09-13T22:01:57Z,2019-09-13T22:01:57Z,MEMBER,I also wrote about this in https://simonwillison.net/2019/Sep/13/weeknotestwitter-sqlite-datasette-rure/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",490803176,--sql and --attach options for feeding commands from SQL queries,
https://github.com/dogsheep/twitter-to-sqlite/issues/9#issuecomment-530028567,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/9,530028567,MDEyOklzc3VlQ29tbWVudDUzMDAyODU2Nw==,9599,simonw,2019-09-10T16:59:25Z,2019-09-10T16:59:25Z,MEMBER,"By default in SQLite foreign key constraints are not enforced (you need to run `PRAGMA foreign_keys = ON;` to enforce them).

We will take advantage of this - even though the `following` table has foreign keys against user we will allow IDs to populate that table without a corresponding user record.

In the future we may add a command that can backfill missing user records.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",491791152,followers-ids and friends-ids subcommands,
https://github.com/dogsheep/twitter-to-sqlite/pull/33#issuecomment-601871930,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/33,601871930,MDEyOklzc3VlQ29tbWVudDYwMTg3MTkzMA==,9599,simonw,2020-03-20T19:21:41Z,2020-03-20T19:21:41Z,MEMBER,Made obsolete by https://github.com/dogsheep/twitter-to-sqlite/commit/d3887d1535c3edd59d3ebf6aea4495bacd82e59b,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",561469252,Upgrade to sqlite-utils 2.2.1,
https://github.com/dogsheep/twitter-to-sqlite/pull/49#issuecomment-902330301,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49,902330301,IC_kwDODEm0Qs41yHe9,9599,simonw,2021-08-20T00:01:56Z,2021-08-20T00:01:56Z,MEMBER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681575714,"Document the use of --stop_after with favorites, refs #20",
https://github.com/dogsheep/twitter-to-sqlite/pull/55#issuecomment-754728696,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55,754728696,MDEyOklzc3VlQ29tbWVudDc1NDcyODY5Ng==,21148,jacobian,2021-01-05T16:02:55Z,2021-01-05T16:02:55Z,CONTRIBUTOR,"This now works for me, though I'm entirely ensure if it's a just-my-export thing or a wider issue. Also, this doesn't contain any tests. So I'm not sure if there's more work to be done here, or if this is good enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779211940,Fix archive imports,
https://github.com/dogsheep/twitter-to-sqlite/pull/55#issuecomment-760950128,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55,760950128,MDEyOklzc3VlQ29tbWVudDc2MDk1MDEyOA==,21148,jacobian,2021-01-15T13:44:52Z,2021-01-15T13:44:52Z,CONTRIBUTOR,"I found and fixed another bug, this one around importing the tweets table. @simonw let me know if you'd prefer this broken out into multiple PRs, happy to do that if it makes review/merging easier.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779211940,Fix archive imports,
https://github.com/dogsheep/twitter-to-sqlite/pull/59#issuecomment-924209583,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/59,924209583,IC_kwDODEm0Qs43FlGv,9599,simonw,2021-09-21T17:37:34Z,2021-09-21T17:37:34Z,MEMBER,Thanks for this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",984942782,"Fix for since_id bug, closes #58",
https://github.com/simonw/datasette/issues/1#issuecomment-338523957,https://api.github.com/repos/simonw/datasette/issues/1,338523957,MDEyOklzc3VlQ29tbWVudDMzODUyMzk1Nw==,9599,simonw,2017-10-23T01:09:05Z,2017-10-24T02:42:12Z,OWNER,"I also need to solve for weird primary keys. If it’s a single integer or a single char field that’s easy. But what if it is a compound key with more than one chat field? What delimiter can I use that will definitely be safe?

Let’s say I use hyphen. Now I need to find a durable encoding for any hyphens that might exist in the key fields themselves.

How about I use URLencoding for every non-alpha-numeric character? That will turn hyphens into (I think) %2D. It should also solve for unicode characters, but it means the vast majority of keys (integers) will display neatly, including a compound key of eg 5678-345

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267513424,Addressable pages for every row in a table,
https://github.com/simonw/datasette/issues/1#issuecomment-338524454,https://api.github.com/repos/simonw/datasette/issues/1,338524454,MDEyOklzc3VlQ29tbWVudDMzODUyNDQ1NA==,9599,simonw,2017-10-23T01:15:24Z,2017-10-23T01:15:24Z,OWNER,Table rendering logic needs to detect the primary key field and turn it into a hyperlink. If there is a compound primary key it should add an extra column at the start of the table which displays the compound key as a link,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267513424,Addressable pages for every row in a table,
https://github.com/simonw/datasette/issues/1#issuecomment-338857568,https://api.github.com/repos/simonw/datasette/issues/1,338857568,MDEyOklzc3VlQ29tbWVudDMzODg1NzU2OA==,9599,simonw,2017-10-24T02:57:12Z,2017-10-24T02:57:12Z,OWNER,"I can find the primary keys using:

    PRAGMA table_info(myTable)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267513424,Addressable pages for every row in a table,
https://github.com/simonw/datasette/issues/1#issuecomment-338861511,https://api.github.com/repos/simonw/datasette/issues/1,338861511,MDEyOklzc3VlQ29tbWVudDMzODg2MTUxMQ==,9599,simonw,2017-10-24T03:24:17Z,2017-10-24T03:24:17Z,OWNER,"Some tables won't have primary keys, in which case I won't generate pages for individual records.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267513424,Addressable pages for every row in a table,
https://github.com/simonw/datasette/issues/1#issuecomment-338872286,https://api.github.com/repos/simonw/datasette/issues/1,338872286,MDEyOklzc3VlQ29tbWVudDMzODg3MjI4Ng==,9599,simonw,2017-10-24T04:46:06Z,2017-10-24T04:46:06Z,OWNER,"I'm going to use `,` as the separator between elements of a compound primary key. If those elements themselves include a comma I will use `%2C` in its place.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267513424,Addressable pages for every row in a table,
https://github.com/simonw/datasette/issues/1#issuecomment-338882207,https://api.github.com/repos/simonw/datasette/issues/1,338882207,MDEyOklzc3VlQ29tbWVudDMzODg4MjIwNw==,9599,simonw,2017-10-24T05:56:04Z,2017-10-24T05:56:04Z,OWNER,Next step: generate links to these.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267513424,Addressable pages for every row in a table,
https://github.com/simonw/datasette/issues/10#issuecomment-341938424,https://api.github.com/repos/simonw/datasette/issues/10,341938424,MDEyOklzc3VlQ29tbWVudDM0MTkzODQyNA==,9599,simonw,2017-11-04T23:48:57Z,2017-11-04T23:48:57Z,OWNER,Done: https://github.com/simonw/stateless-datasets/commit/edaa10587e60946e0c1935333f6b79553db33798,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267517381,Set up Travis,
https://github.com/simonw/datasette/issues/100#issuecomment-344771130,https://api.github.com/repos/simonw/datasette/issues/100,344771130,MDEyOklzc3VlQ29tbWVudDM0NDc3MTEzMA==,9599,simonw,2017-11-16T00:06:00Z,2017-11-16T00:06:00Z,OWNER,"Aha... it looks like this is a Jinja version problem: https://github.com/ansible/ansible/issues/25381#issuecomment-306492389

Datasette depends on sanic-jinja2 - and that doesn't depend on a particular jinja2 version: https://github.com/lixxu/sanic-jinja2/blob/7e9520850d8c6bb66faf43b7f252593d7efe3452/setup.py#L22

So if you have an older version of Jinja installed, stuff breaks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274160723,TemplateAssertionError: no filter named 'tojson',
https://github.com/simonw/datasette/issues/100#issuecomment-344864254,https://api.github.com/repos/simonw/datasette/issues/100,344864254,MDEyOklzc3VlQ29tbWVudDM0NDg2NDI1NA==,13304454,coisnepe,2017-11-16T09:25:10Z,2017-11-16T09:25:10Z,NONE,@simonw I see. I upgraded sanic-jinja2 and jinja2: it now works flawlessly. Thank you!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274160723,TemplateAssertionError: no filter named 'tojson',
https://github.com/simonw/datasette/issues/1001#issuecomment-705904566,https://api.github.com/repos/simonw/datasette/issues/1001,705904566,MDEyOklzc3VlQ29tbWVudDcwNTkwNDU2Ng==,9599,simonw,2020-10-09T00:58:08Z,2020-10-09T00:58:08Z,OWNER,"To get a traceback:
```
datasette . -p 8009 --pdb
```
And then:
```
curl -XOPTIONS http://127.0.0.1:8009
```
This causes the server to open a debugging prompt:
```
INFO:     127.0.0.1:59514 - ""OPTIONS / HTTP/1.1"" 500 Internal Server Error
> /Users/simon/Dropbox/Development/datasette/datasette/views/base.py(115)dispatch_request()
-> return await handler(request, *args, **kwargs)
(Pdb) list
110  	    def database_color(self, database):
111  	        return ""ff0000""
112  	
113  	    async def dispatch_request(self, request, *args, **kwargs):
114  	        handler = getattr(self, request.method.lower(), None)
115  ->	        return await handler(request, *args, **kwargs)
116  	
117  	    async def render(self, templates, request, context=None):
118  	        context = context or {}
119  	        template = self.ds.jinja_env.select_template(templates)
120  	        template_context = {
(Pdb) 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705904679,https://api.github.com/repos/simonw/datasette/issues/1001,705904679,MDEyOklzc3VlQ29tbWVudDcwNTkwNDY3OQ==,9599,simonw,2020-10-09T00:58:32Z,2020-10-09T00:58:32Z,OWNER,So the bug is in this code here: https://github.com/simonw/datasette/blob/703439bdc37e724b01bc6d7a1fc1d955795132f2/datasette/views/base.py#L113-L115,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705904759,https://api.github.com/repos/simonw/datasette/issues/1001,705904759,MDEyOklzc3VlQ29tbWVudDcwNTkwNDc1OQ==,9599,simonw,2020-10-09T00:58:47Z,2020-10-09T00:58:47Z,OWNER,"What should an OPTIONS request return, anyway?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705904917,https://api.github.com/repos/simonw/datasette/issues/1001,705904917,MDEyOklzc3VlQ29tbWVudDcwNTkwNDkxNw==,9599,simonw,2020-10-09T00:59:25Z,2020-10-09T00:59:25Z,OWNER,"```
~ % curl -XOPTIONS https://www.google.com/
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content=""initial-scale=1, minimum-scale=1, width=device-width"">
  <title>Error 405 (Method Not Allowed)!!1</title>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705905121,https://api.github.com/repos/simonw/datasette/issues/1001,705905121,MDEyOklzc3VlQ29tbWVudDcwNTkwNTEyMQ==,9599,simonw,2020-10-09T01:00:07Z,2020-10-09T01:00:07Z,OWNER,"The www.djangoproject.com site returns an empty page:
```
~ % curl -vv -XOPTIONS https://www.djangoproject.com/
*   Trying 151.101.42.217:443...
* Connected to www.djangoproject.com (151.101.42.217) port 443 (#0)
* ALPN, offering http/1.1
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: osff2.map.fastly.net
* Server certificate: GlobalSign CloudSSL CA - SHA256 - G3
* Server certificate: GlobalSign Root CA
> OPTIONS / HTTP/1.1
> Host: www.djangoproject.com
> User-Agent: curl/7.70.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: keep-alive
< Content-Length: 0
< Server: nginx
< Content-Type: text/html; charset=utf-8
< Allow: GET, HEAD, OPTIONS
< Content-Language: en
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Access-Control-Allow-Origin: https://code.djangoproject.com
< Accept-Ranges: bytes
< Date: Fri, 09 Oct 2020 00:59:42 GMT
< Via: 1.1 varnish
< X-Served-By: cache-sjc10047-SJC
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Timer: S1602205182.833493,VS0,VE305
< Vary: Accept-Language, Accept-Encoding
< 
* Connection #0 to host www.djangoproject.com left intact
~ % 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705905418,https://api.github.com/repos/simonw/datasette/issues/1001,705905418,MDEyOklzc3VlQ29tbWVudDcwNTkwNTQxOA==,9599,simonw,2020-10-09T01:00:54Z,2020-10-09T01:00:54Z,OWNER,https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/OPTIONS,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705916614,https://api.github.com/repos/simonw/datasette/issues/1001,705916614,MDEyOklzc3VlQ29tbWVudDcwNTkxNjYxNA==,9599,simonw,2020-10-09T01:37:06Z,2020-10-09T01:37:06Z,OWNER,"I'm tempted to imitate Django Rest Framework here: https://github.com/encode/django-rest-framework/blob/2e721cdbc85a924d0b0f093b86fe1497b58fe287/rest_framework/views.py#L514-L521
```python
    def options(self, request, *args, **kwargs):
        """"""
        Handler method for HTTP 'OPTIONS' request.
        """"""
        if self.metadata_class is None:
            return self.http_method_not_allowed(request, *args, **kwargs)
        data = self.metadata_class().determine_metadata(request, self)
        return Response(data, status=status.HTTP_200_OK)
```
That `determine_metadata()` default method does this: https://github.com/encode/django-rest-framework/blob/335054a5d36b352a58286b303b608b6bf48152f8/rest_framework/metadata.py#L29

Note the comment at the top:
```
    There are not any formalized standards for `OPTIONS` responses
    for us to base this on.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705917015,https://api.github.com/repos/simonw/datasette/issues/1001,705917015,MDEyOklzc3VlQ29tbWVudDcwNTkxNzAxNQ==,9599,simonw,2020-10-09T01:38:49Z,2020-10-09T01:38:49Z,OWNER,"I actually have a sensible `OPTIONS` implementation here:

https://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/datasette/views/base.py#L154-L165

I'm going to set the default one to return a 405 (Method Not Allowed) like Google does.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1002#issuecomment-706306214,https://api.github.com/repos/simonw/datasette/issues/1002,706306214,MDEyOklzc3VlQ29tbWVudDcwNjMwNjIxNA==,9599,simonw,2020-10-09T17:23:51Z,2020-10-09T17:23:51Z,OWNER,I can start by combining the release notes for the 0.50 alphas.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717783692,Release notes for Datasette 0.50,
https://github.com/simonw/datasette/issues/1003#issuecomment-706272322,https://api.github.com/repos/simonw/datasette/issues/1003,706272322,MDEyOklzc3VlQ29tbWVudDcwNjI3MjMyMg==,9599,simonw,2020-10-09T16:14:56Z,2020-10-09T16:14:56Z,OWNER,Yes I think that makes sense. I added `json` to the template context in Dogsheep Beta just a few days ago because I needed that: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L176,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718238967,from_json jinja2 filter,
https://github.com/simonw/datasette/issues/1003#issuecomment-706273211,https://api.github.com/repos/simonw/datasette/issues/1003,706273211,MDEyOklzc3VlQ29tbWVudDcwNjI3MzIxMQ==,9599,simonw,2020-10-09T16:16:38Z,2020-10-09T16:16:38Z,OWNER,"I'm not a huge fan of `from_json` as the name for this. Some other options:

- Expose `json` directly so templates can do `json.loads()` and `json.dumps()` - this allows for outputting JSON too, which is useful. But is there anything else on the `json` module that shouldn't be exposed in templates?
- `json_dumps()` and `json_loads()` template functions. I quite like that.
- Something else?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718238967,from_json jinja2 filter,
https://github.com/simonw/datasette/issues/1003#issuecomment-706281451,https://api.github.com/repos/simonw/datasette/issues/1003,706281451,MDEyOklzc3VlQ29tbWVudDcwNjI4MTQ1MQ==,9599,simonw,2020-10-09T16:33:01Z,2020-10-09T16:33:01Z,OWNER,I think `json_dumps()` and `json_loads()` as aliases for `json.dumps()` and `json.loads()` is the way to go here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718238967,from_json jinja2 filter,
https://github.com/simonw/datasette/issues/1003#issuecomment-706302863,https://api.github.com/repos/simonw/datasette/issues/1003,706302863,MDEyOklzc3VlQ29tbWVudDcwNjMwMjg2Mw==,649467,mhalle,2020-10-09T17:17:06Z,2020-10-09T17:17:06Z,NONE,"I agree on the descriptive and python-consistent naming. There is already a tojson, but frankly i find the ""to"" and ""from"" confusing in a text templating language where what's a string and what's data isn't 100% transparent.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718238967,from_json jinja2 filter,
https://github.com/simonw/datasette/issues/1005#issuecomment-706473306,https://api.github.com/repos/simonw/datasette/issues/1005,706473306,MDEyOklzc3VlQ29tbWVudDcwNjQ3MzMwNg==,9599,simonw,2020-10-10T02:45:46Z,2020-10-10T02:45:46Z,OWNER,"https://github.com/simonw/datasette/blob/9f6dd985bc0eff70f8a9ce65c6578bc43d2e172b/tests/test_html.py#L145-L149

https://github.com/simonw/datasette/blob/9f6dd985bc0eff70f8a9ce65c6578bc43d2e172b/tests/test_html.py#L539-L564

https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/tests/test_api.py#L1190-L1194

https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/tests/test_api.py#L742-L746","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718259202,Remove xfail tests when new httpx is released,
https://github.com/simonw/datasette/issues/1005#issuecomment-747209115,https://api.github.com/repos/simonw/datasette/issues/1005,747209115,MDEyOklzc3VlQ29tbWVudDc0NzIwOTExNQ==,9599,simonw,2020-12-17T05:11:04Z,2020-12-17T05:11:04Z,OWNER,Tracking ticket for the next HTTPX release is https://github.com/encode/httpx/pull/1403,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718259202,Remove xfail tests when new httpx is released,
https://github.com/simonw/datasette/issues/1005#issuecomment-787536267,https://api.github.com/repos/simonw/datasette/issues/1005,787536267,MDEyOklzc3VlQ29tbWVudDc4NzUzNjI2Nw==,9599,simonw,2021-02-28T22:30:37Z,2021-02-28T22:30:37Z,OWNER,It's out! https://github.com/encode/httpx/releases/tag/0.17.0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718259202,Remove xfail tests when new httpx is released,
https://github.com/simonw/datasette/issues/1006#issuecomment-706270877,https://api.github.com/repos/simonw/datasette/issues/1006,706270877,MDEyOklzc3VlQ29tbWVudDcwNjI3MDg3Nw==,9599,simonw,2020-10-09T16:12:09Z,2020-10-09T16:12:09Z,OWNER,This can become a section on https://docs.datasette.io/en/stable/internals.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718264811,Documentation for datasette.client,
https://github.com/simonw/datasette/issues/1006#issuecomment-706305601,https://api.github.com/repos/simonw/datasette/issues/1006,706305601,MDEyOklzc3VlQ29tbWVudDcwNjMwNTYwMQ==,9599,simonw,2020-10-09T17:22:31Z,2020-10-09T17:22:31Z,OWNER,https://docs.datasette.io/en/latest/internals.html#client,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718264811,Documentation for datasette.client,
https://github.com/simonw/datasette/issues/1007#issuecomment-706276831,https://api.github.com/repos/simonw/datasette/issues/1007,706276831,MDEyOklzc3VlQ29tbWVudDcwNjI3NjgzMQ==,9599,simonw,2020-10-09T16:23:51Z,2020-10-09T16:23:51Z,OWNER,"I don't appear to be using these anywhere, not sure why I spotted a warning (which I now can't find).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718272593,set-env and add-path commands have been deprecated,
https://github.com/simonw/datasette/issues/101#issuecomment-344597274,https://api.github.com/repos/simonw/datasette/issues/101,344597274,MDEyOklzc3VlQ29tbWVudDM0NDU5NzI3NA==,450244,eaubin,2017-11-15T13:48:55Z,2017-11-15T13:48:55Z,NONE,This is a duplicate of https://github.com/simonw/datasette/issues/100,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274161964,TemplateAssertionError: no filter named 'tojson',
https://github.com/simonw/datasette/issues/1010#issuecomment-706450418,https://api.github.com/repos/simonw/datasette/issues/1010,706450418,MDEyOklzc3VlQ29tbWVudDcwNjQ1MDQxOA==,9599,simonw,2020-10-10T00:08:27Z,2020-10-10T00:09:13Z,OWNER,"This commit broke it: https://github.com/simonw/datasette/commit/8f97b9b58e77f82fef1f10e9c9f6754b993544b6#diff-ed4bb3a97b7efa4d45be6a6ea6f4636cL114-R165

<img width=""1106"" alt=""Banners_and_Alerts_and_datasette_client_internal_requests_mechanism_·_simonw_datasette_8f97b9b"" src=""https://user-images.githubusercontent.com/9599/95640559-25a37f00-0a52-11eb-838e-9f771ab5a413.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718484082,json / CSV links are broken in Datasette 0.50,
https://github.com/simonw/datasette/issues/1010#issuecomment-706455202,https://api.github.com/repos/simonw/datasette/issues/1010,706455202,MDEyOklzc3VlQ29tbWVudDcwNjQ1NTIwMg==,9599,simonw,2020-10-10T00:33:46Z,2020-10-10T00:33:46Z,OWNER,"Since this is an urgent fix that needs to go out I've marked the `base_url` tests as `xfail`. I need to solve those separately, see #900 and various other bug reports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718484082,json / CSV links are broken in Datasette 0.50,
https://github.com/simonw/datasette/issues/1010#issuecomment-706455505,https://api.github.com/repos/simonw/datasette/issues/1010,706455505,MDEyOklzc3VlQ29tbWVudDcwNjQ1NTUwNQ==,9599,simonw,2020-10-10T00:35:41Z,2020-10-10T00:35:41Z,OWNER,I missed the links on the query page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718484082,json / CSV links are broken in Datasette 0.50,
https://github.com/simonw/datasette/issues/1011#issuecomment-706480763,https://api.github.com/repos/simonw/datasette/issues/1011,706480763,MDEyOklzc3VlQ29tbWVudDcwNjQ4MDc2Mw==,9599,simonw,2020-10-10T03:42:58Z,2020-10-10T03:42:58Z,OWNER,"Oh no! It's the same bug as #1010. Thanks for the report, I shall push out 0.50.2 as quickly as possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718521469,column name links broken in 0.50.1,
https://github.com/simonw/datasette/issues/1011#issuecomment-706480791,https://api.github.com/repos/simonw/datasette/issues/1011,706480791,MDEyOklzc3VlQ29tbWVudDcwNjQ4MDc5MQ==,9599,simonw,2020-10-10T03:43:13Z,2020-10-10T03:43:13Z,OWNER,Demo: click column headers on https://latest.datasette.io/fixtures/compound_three_primary_keys,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718521469,column name links broken in 0.50.1,
https://github.com/simonw/datasette/issues/1011#issuecomment-706482542,https://api.github.com/repos/simonw/datasette/issues/1011,706482542,MDEyOklzc3VlQ29tbWVudDcwNjQ4MjU0Mg==,9599,simonw,2020-10-10T03:58:58Z,2020-10-10T03:58:58Z,OWNER,The fix is live on the demo: https://latest.datasette.io/fixtures/compound_three_primary_keys?_sort_desc=pk2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718521469,column name links broken in 0.50.1,
https://github.com/simonw/datasette/issues/1011#issuecomment-706483680,https://api.github.com/repos/simonw/datasette/issues/1011,706483680,MDEyOklzc3VlQ29tbWVudDcwNjQ4MzY4MA==,9599,simonw,2020-10-10T04:09:32Z,2020-10-10T04:09:32Z,OWNER,"OK, Datasette 0.50.2 is live on PyPI now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718521469,column name links broken in 0.50.1,
https://github.com/simonw/datasette/issues/1012#issuecomment-714908859,https://api.github.com/repos/simonw/datasette/issues/1012,714908859,MDEyOklzc3VlQ29tbWVudDcxNDkwODg1OQ==,45380,bollwyvl,2020-10-23T04:49:20Z,2020-10-23T04:49:20Z,CONTRIBUTOR,"Good luck on 1.0! It may also be worth lobbying for a `Framework::Datasette::1.0` classifier. This would be a nice way to allow the ecosystem to self-document a bit more [discoverably](https://pypi.org/search/?q=&o=&c=Framework+%3A%3A+Datasette%3A%3A+1.0).  

I was surprised to see the [PR for `Framework::Jupyter`](https://github.com/pypa/warehouse/pull/1905/files) is a... database migration! Of course, there may be more workflow to it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py,
https://github.com/simonw/datasette/issues/1012#issuecomment-753531657,https://api.github.com/repos/simonw/datasette/issues/1012,753531657,MDEyOklzc3VlQ29tbWVudDc1MzUzMTY1Nw==,45380,bollwyvl,2021-01-02T21:25:36Z,2021-01-02T21:25:36Z,CONTRIBUTOR,"Actually, on more research, I found out this is handled by the [trove-classifiers package](https://github.com/pypa/trove-classifiers/blob/master/src/trove_classifiers/__init__.py#L2) now, so it's just a one-liner pr instead of fire-up-a-docker-container-and-do-some-migrations","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py,
https://github.com/simonw/datasette/issues/1012#issuecomment-969602825,https://api.github.com/repos/simonw/datasette/issues/1012,969602825,IC_kwDOBm6k_c45yvcJ,9599,simonw,2021-11-16T01:21:14Z,2021-11-16T01:21:14Z,OWNER,"I'd been wondering how to get new classifiers into Trove - thanks, I'll give this a go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py,
https://github.com/simonw/datasette/issues/1012#issuecomment-969613166,https://api.github.com/repos/simonw/datasette/issues/1012,969613166,IC_kwDOBm6k_c45yx9u,9599,simonw,2021-11-16T01:27:25Z,2021-11-16T01:27:25Z,OWNER,"Requested here:

- https://github.com/pypa/trove-classifiers/pull/85","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py,
https://github.com/simonw/datasette/issues/1012#issuecomment-970266123,https://api.github.com/repos/simonw/datasette/issues/1012,970266123,IC_kwDOBm6k_c451RYL,45380,bollwyvl,2021-11-16T13:18:36Z,2021-11-16T13:18:36Z,CONTRIBUTOR,"Congratulations, looks like it went through! There was a bit of a hold-up
on the JupyterLab ones, but it's semi automated: a dependabot pr to
warehouse and a CI deploy, with a click in between.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py,
https://github.com/simonw/datasette/issues/1014#issuecomment-706626796,https://api.github.com/repos/simonw/datasette/issues/1014,706626796,MDEyOklzc3VlQ29tbWVudDcwNjYyNjc5Ng==,9599,simonw,2020-10-10T23:47:23Z,2020-10-10T23:48:26Z,OWNER,"The GitHub API has these:

`link: <https://api.github.com/repositories/107914493/commits?page=2>; rel=""next"", <https://api.github.com/repositories/107914493/commits?page=46>; rel=""last""`

WordPress API: https://css-tricks.com/wp-json/wp/v2/posts

`link: <https://css-tricks.com/wp-json/wp/v2/posts?page=2>; rel=""next""`

It also has these:

```
x-wp-total: 5162
x-wp-totalpages: 517
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706626913,https://api.github.com/repos/simonw/datasette/issues/1014,706626913,MDEyOklzc3VlQ29tbWVudDcwNjYyNjkxMw==,9599,simonw,2020-10-10T23:49:08Z,2020-10-10T23:49:08Z,OWNER,This should be documented on https://docs.datasette.io/en/stable/json_api.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706626934,https://api.github.com/repos/simonw/datasette/issues/1014,706626934,MDEyOklzc3VlQ29tbWVudDcwNjYyNjkzNA==,9599,simonw,2020-10-10T23:49:25Z,2020-10-10T23:49:25Z,OWNER,I'm just going to implement the 'next' one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706626971,https://api.github.com/repos/simonw/datasette/issues/1014,706626971,MDEyOklzc3VlQ29tbWVudDcwNjYyNjk3MQ==,9599,simonw,2020-10-10T23:49:57Z,2020-10-10T23:49:57Z,OWNER,I won't offer a total-number header. That's expensive to calculate - no need to calculate it unless it's explicitly asked for.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706627035,https://api.github.com/repos/simonw/datasette/issues/1014,706627035,MDEyOklzc3VlQ29tbWVudDcwNjYyNzAzNQ==,9599,simonw,2020-10-10T23:50:40Z,2020-10-10T23:50:40Z,OWNER,"First example page to add this to: `?_shape=array`, e.g. https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706631006,https://api.github.com/repos/simonw/datasette/issues/1014,706631006,MDEyOklzc3VlQ29tbWVudDcwNjYzMTAwNg==,9599,simonw,2020-10-11T00:36:43Z,2020-10-11T00:36:43Z,OWNER,"Demo using [paginate-json](https://github.com/simonw/paginate-json):
```
% paginate-json 'https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array' | jq '. | length' 
https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cd%2Cv
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ch%2Cr
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cl%2Cn
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cp%2Cj
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ct%2Cf
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cx%2Cb
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ca%2Cx
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ce%2Ct
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ci%2Cp
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Cm%2Cl
1001
```
New documentation: https://docs.datasette.io/en/latest/json_api.html#pagination","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1015#issuecomment-706756879,https://api.github.com/repos/simonw/datasette/issues/1015,706756879,MDEyOklzc3VlQ29tbWVudDcwNjc1Njg3OQ==,9599,simonw,2020-10-11T19:35:03Z,2020-10-11T19:35:03Z,OWNER,"Since plugins are installed via pip this would require Datasette to be restarted. This StackOverflow thread looks relevant to that: https://stackoverflow.com/questions/11329917/restart-python-script-from-within-itself

This recipe looks promising:
```python
import os
import sys
import psutil
import logging

def restart_program():
    """"""Restarts the current program, with file objects and descriptors
       cleanup
    """"""

    try:
        p = psutil.Process(os.getpid())
        for handler in p.get_open_files() + p.connections():
            os.close(handler.fd)
    except Exception, e:
        logging.error(e)

    python = sys.executable
    os.execl(python, python, *sys.argv)
```
https://docs.python.org/3/library/os.html#os.execl says about `os.execl`:

> These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718910318,Research: could Datasette install its own plugins?,
https://github.com/simonw/datasette/issues/1016#issuecomment-706788010,https://api.github.com/repos/simonw/datasette/issues/1016,706788010,MDEyOklzc3VlQ29tbWVudDcwNjc4ODAxMA==,9599,simonw,2020-10-11T23:50:39Z,2020-10-11T23:50:39Z,OWNER,"For consistency can reuse the icon used on selected facets:

<img width=""721"" alt=""evernote__notes__8_rows_where_where_application-data____ACD4B543-4663-40B2-BAF5-666FA54D4204__and_source-url_is_not_blank_sorted_by_id"" src=""https://user-images.githubusercontent.com/9599/95693564-e0ae5280-0be1-11eb-881b-fa0cec8397f5.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718953669,"Add a ""delete"" icon next to filters (in addition to ""remove filter"")",
https://github.com/simonw/datasette/issues/1016#issuecomment-706821561,https://api.github.com/repos/simonw/datasette/issues/1016,706821561,MDEyOklzc3VlQ29tbWVudDcwNjgyMTU2MQ==,9599,simonw,2020-10-12T02:33:52Z,2020-10-12T02:33:52Z,OWNER,"Should this do the equivalent of clicking Apply for you? I don't think so - that wouldn't fit with how the rest of that block of controls works.

But if it makes the row vanish it may not be clear to the user that they still need to click Apply.

Easiest thing would be for it to select the ""remove filter"" column and blank out the boxes. I will try that first.

The code can go in `table.js`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718953669,"Add a ""delete"" icon next to filters (in addition to ""remove filter"")",
https://github.com/simonw/datasette/issues/1016#issuecomment-706828375,https://api.github.com/repos/simonw/datasette/issues/1016,706828375,MDEyOklzc3VlQ29tbWVudDcwNjgyODM3NQ==,9599,simonw,2020-10-12T03:01:58Z,2020-10-12T03:01:58Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet=created&created=2019-01-17+08%3A00%3A00&_facet=state&state=MI

![x-button](https://user-images.githubusercontent.com/9599/95700744-9b4b4e80-0bfc-11eb-85a3-6da82d9e0cbf.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718953669,"Add a ""delete"" icon next to filters (in addition to ""remove filter"")",
https://github.com/simonw/datasette/issues/1019#issuecomment-708113989,https://api.github.com/repos/simonw/datasette/issues/1019,708113989,MDEyOklzc3VlQ29tbWVudDcwODExMzk4OQ==,9599,simonw,2020-10-14T02:22:40Z,2020-10-14T02:22:40Z,OWNER,"I've found myself wanting this too.

It should respect the allow-sql permission so it doesn't display if the user isn't able to run custom SQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708114636,https://api.github.com/repos/simonw/datasette/issues/1019,708114636,MDEyOklzc3VlQ29tbWVudDcwODExNDYzNg==,9599,simonw,2020-10-14T02:24:56Z,2020-10-14T02:24:56Z,OWNER,"So it would appear on this page: https://latest.datasette.io/fixtures/neighborhood_search#fragment-goes-here

Probably as a blue ""Edit SQL"" link next to those buttons.

![575FB160-4F75-43B2-B2AB-D75E60F10477](https://user-images.githubusercontent.com/9599/95936143-c2835680-0d89-11eb-9cf3-06f86d923103.jpeg)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708127294,https://api.github.com/repos/simonw/datasette/issues/1019,708127294,MDEyOklzc3VlQ29tbWVudDcwODEyNzI5NA==,9599,simonw,2020-10-14T03:08:13Z,2020-10-14T03:08:13Z,OWNER,"Maybe like this:

<img width=""1385"" alt=""fixtures__select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________order_by_neighborhood_"" src=""https://user-images.githubusercontent.com/9599/95938930-ce721700-0d8f-11eb-87c8-f67ea4e57d55.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708128286,https://api.github.com/repos/simonw/datasette/issues/1019,708128286,MDEyOklzc3VlQ29tbWVudDcwODEyODI4Ng==,9599,simonw,2020-10-14T03:11:33Z,2020-10-14T03:14:07Z,OWNER,"Another edge-case: https://latest.datasette.io/fixtures/pragma_cache_size (`PRAGMA cache_size;`) isn't an allowed query usually, so linking to ""Edit SQL"" for it would link to an error page: https://latest.datasette.io/fixtures?sql=PRAGMA+cache_size%3B

Can use `datasette.utils.validate_sql_select(sql)` to check for that - it raises `datasette.utils.InvalidSql` if there's a problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708130290,https://api.github.com/repos/simonw/datasette/issues/1019,708130290,MDEyOklzc3VlQ29tbWVudDcwODEzMDI5MA==,9599,simonw,2020-10-14T03:17:57Z,2020-10-14T03:17:57Z,OWNER,"One last edge-case (I think) - magic parameters aren't supported for non-canned-queries, so if a query includes those it shouldn't show an Edit SQL link: https://latest.datasette.io/fixtures/magic_parameters

I don't have a clean way of detecting if a query contains defined magic parameters, but I can instead do a dumb substring match for `:_` and skip showing the Edit SQL link if that is present.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708139822,https://api.github.com/repos/simonw/datasette/issues/1019,708139822,MDEyOklzc3VlQ29tbWVudDcwODEzOTgyMg==,9599,simonw,2020-10-14T03:53:21Z,2020-10-14T03:53:21Z,OWNER,"Demos:
- https://latest.datasette.io/fixtures/neighborhood_search has the link
- https://latest.datasette.io/fixtures/neighborhood_search?text=ber has the link, and the link passes through the parameter that has already been entered
- https://latest.datasette.io/fixtures/%F0%9D%90%9C%F0%9D%90%A2%F0%9D%90%AD%F0%9D%90%A2%F0%9D%90%9E%F0%9D%90%AC has the link
- https://latest.datasette.io/fixtures/pragma_cache_size does not have the link (`PRAGMA` is not allowed in regular editable queries)
- https://latest.datasette.io/fixtures/magic_parameters does not have the link (magic parameters)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708520800,https://api.github.com/repos/simonw/datasette/issues/1019,708520800,MDEyOklzc3VlQ29tbWVudDcwODUyMDgwMA==,639012,jsfenfen,2020-10-14T16:37:19Z,2020-10-14T16:37:19Z,CONTRIBUTOR,🎉   Thanks so much @simonw  ! 🎉 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/102#issuecomment-754192267,https://api.github.com/repos/simonw/datasette/issues/102,754192267,MDEyOklzc3VlQ29tbWVudDc1NDE5MjI2Nw==,9599,simonw,2021-01-04T20:13:19Z,2021-01-04T20:13:19Z,OWNER,"I'm more likely to do Lambda than Elastic Beanstalk, especially now the size limit for Lambdas has been increased as part of their support for Docker.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274264175,datasette publish elasticbeanstalk,
https://github.com/simonw/datasette/issues/1020#issuecomment-708669178,https://api.github.com/repos/simonw/datasette/issues/1020,708669178,MDEyOklzc3VlQ29tbWVudDcwODY2OTE3OA==,9599,simonw,2020-10-14T21:26:37Z,2020-10-14T21:26:37Z,OWNER,"One option: add an optional request=... parameter which can be passed the current request, and will use that to populate the mock request with the exception of the bits that are passed explicitly (like the path):

```python
response = await datasette.client.get(""/db/table.json"", request=request)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-708669778,https://api.github.com/repos/simonw/datasette/issues/1020,708669778,MDEyOklzc3VlQ29tbWVudDcwODY2OTc3OA==,9599,simonw,2020-10-14T21:27:58Z,2020-10-14T21:27:58Z,OWNER,"Maybe these internal requests should have some kind of flag that lets the underlying code tell that it's being called internally.

One option: add a `x-internal: 1` request header - and ensure that any requests from outside Datasette have that header stripped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-708670392,https://api.github.com/repos/simonw/datasette/issues/1020,708670392,MDEyOklzc3VlQ29tbWVudDcwODY3MDM5Mg==,9599,simonw,2020-10-14T21:29:22Z,2020-10-14T21:29:22Z,OWNER,"I should also verify (and probably unit-test) that things like the `?_trace=1` mechanism work across the internal request boundary.

`/-/permissions` appears to work across this boundary, but again a test would be useful confirmation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-712481568,https://api.github.com/repos/simonw/datasette/issues/1020,712481568,MDEyOklzc3VlQ29tbWVudDcxMjQ4MTU2OA==,9599,simonw,2020-10-19T22:41:59Z,2020-10-19T22:41:59Z,OWNER,"It turns out this works just fine:
```python
response = await datasette.client.get(path, cookies=request.cookies)
```
So I don't need a mechanism for this. I'm going to add this to the documentation instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-712482015,https://api.github.com/repos/simonw/datasette/issues/1020,712482015,MDEyOklzc3VlQ29tbWVudDcxMjQ4MjAxNQ==,9599,simonw,2020-10-19T22:43:24Z,2020-10-19T22:43:24Z,OWNER,"... unless I want to support authentication mechanisms that work based on incoming IP address instead, in which case there's an argument for copying more over from the incoming request.

Tailscale is a good example of a system where authentication based on IP address can actually work well, so this is worth doing. Also, there might be authentication mechanisms which work by setting a custom header on the incoming request (not to mention the `Authorization` header).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-712482504,https://api.github.com/repos/simonw/datasette/issues/1020,712482504,MDEyOklzc3VlQ29tbWVudDcxMjQ4MjUwNA==,9599,simonw,2020-10-19T22:45:01Z,2020-10-19T22:45:01Z,OWNER,"I'm having trouble coming up with the syntax for this. Here's one option:
```python
response = await datasette.client.get(path, request=request)
```
But this feels confusing to me. We're not using the `request=` argument as a request - we're using it as a source of some default request values (the cookies and incoming headers, but not the path).

We're essentially combining that request with the other arguments passed to `.get()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1023#issuecomment-712604541,https://api.github.com/repos/simonw/datasette/issues/1023,712604541,MDEyOklzc3VlQ29tbWVudDcxMjYwNDU0MQ==,9599,simonw,2020-10-20T05:39:44Z,2020-10-20T05:39:44Z,OWNER,Here's the alpha with most of this work ready for people to preview: https://github.com/simonw/datasette/releases/tag/0.51a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url,
https://github.com/simonw/datasette/issues/1023#issuecomment-712607608,https://api.github.com/repos/simonw/datasette/issues/1023,712607608,MDEyOklzc3VlQ29tbWVudDcxMjYwNzYwOA==,9599,simonw,2020-10-20T05:47:42Z,2020-10-20T05:47:42Z,OWNER,Requested alpha testers in https://github.com/simonw/datasette/issues/838#issuecomment-712604364,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url,
https://github.com/simonw/datasette/issues/1023#issuecomment-719986922,https://api.github.com/repos/simonw/datasette/issues/1023,719986922,MDEyOklzc3VlQ29tbWVudDcxOTk4NjkyMg==,9599,simonw,2020-10-31T20:51:01Z,2020-10-31T20:51:01Z,OWNER,This should all be working correctly now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url,
https://github.com/simonw/datasette/issues/1024#issuecomment-709589297,https://api.github.com/repos/simonw/datasette/issues/1024,709589297,MDEyOklzc3VlQ29tbWVudDcwOTU4OTI5Nw==,9599,simonw,2020-10-15T21:04:31Z,2020-10-15T21:04:31Z,OWNER,I think nginx or Apache would be the best tools for this. I'm inclined to try with nginx first since I know it better.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709590337,https://api.github.com/repos/simonw/datasette/issues/1024,709590337,MDEyOklzc3VlQ29tbWVudDcwOTU5MDMzNw==,9599,simonw,2020-10-15T21:06:24Z,2020-10-15T21:07:19Z,OWNER,"From https://stackoverflow.com/questions/32549684/nginx-proxy-and-remove-proxy-pass-prefix/32550251 it looks like the config I should use is:
```
server {
    listen 80;
    server_name example.com;

    location /datasette/ {
        proxy_pass              http://127.0.0.1:8001;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        proxy_read_timeout      90;
    }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709590941,https://api.github.com/repos/simonw/datasette/issues/1024,709590941,MDEyOklzc3VlQ29tbWVudDcwOTU5MDk0MQ==,9599,simonw,2020-10-15T21:07:47Z,2020-10-15T21:07:47Z,OWNER,On macOS I ran `brew install nginx`. I'm going to try running it on port 8000 so I don't have to run it as root.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709595960,https://api.github.com/repos/simonw/datasette/issues/1024,709595960,MDEyOklzc3VlQ29tbWVudDcwOTU5NTk2MA==,9599,simonw,2020-10-15T21:18:14Z,2020-10-15T21:18:14Z,OWNER,Typing `nginx` starts it running as a daemon listening on port `http-alt` aka 8080. It uses the config file from ` /usr/local/etc/nginx/nginx.conf`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709597589,https://api.github.com/repos/simonw/datasette/issues/1024,709597589,MDEyOklzc3VlQ29tbWVudDcwOTU5NzU4OQ==,9599,simonw,2020-10-15T21:21:53Z,2020-10-15T21:23:25Z,OWNER,"Here's a recipe for running nginx against a custom config file: https://gist.github.com/simonw/35f0ebf9c1d6df158759

```
daemon off;

events {
  worker_connections  1024;
}

http {
  access_log /dev/stdout;
  error_log /dev/stderr;

  types {
    text/html                             html htm shtml;
    text/css                              css;
    image/gif                             gif;
    image/jpeg                            jpeg jpg;
    application/javascript                js;
  }

  server {
    listen 8002;
    index index.html;
    root  app;
  }
}
```
```
nginx -p `pwd` -c `pwd`/nginx.conf
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709598324,https://api.github.com/repos/simonw/datasette/issues/1024,709598324,MDEyOklzc3VlQ29tbWVudDcwOTU5ODMyNA==,9599,simonw,2020-10-15T21:23:33Z,2020-10-15T21:26:55Z,OWNER,"Combining these two examples, here's the config file I am going to use for this. I'll save this as `nginx.conf`:
```
daemon off;

events {
  worker_connections  1024;
}

http {
  server {
    listen 8000;

    location /datasette {
        proxy_pass              http://127.0.0.1:8001;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    }
  }
}
```
Then start the server with:
```
nginx -p `pwd` -c `pwd`/nginx.conf
```
And start Datasette like this:
```
datasette fixtures.db --config base_url:/datasette/
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709600335,https://api.github.com/repos/simonw/datasette/issues/1024,709600335,MDEyOklzc3VlQ29tbWVudDcwOTYwMDMzNQ==,9599,simonw,2020-10-15T21:28:02Z,2020-10-15T22:25:43Z,OWNER,"This is working OK so far:

<img width=""897"" alt=""fixtures__compound_three_primary_keys__1_001_rows_where_sorted_by_pk2"" src=""https://user-images.githubusercontent.com/9599/96187878-96371980-0ef2-11eb-875e-5b1ae9b445fd.png"">

I'll try crawling it with `wget -r` to see if I get any errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709622973,https://api.github.com/repos/simonw/datasette/issues/1024,709622973,MDEyOklzc3VlQ29tbWVudDcwOTYyMjk3Mw==,9599,simonw,2020-10-15T22:27:31Z,2020-10-15T22:27:31Z,OWNER,"Here's how I tested it:
```
time wget -r 'http://localhost:8000/datasette/' 2>&1 | grep -i -C 5 ""failed\|error"" > /tmp/errors.txt
```
This wrote out any errors (plus context) to the `errors.txt` log - and reported that the full crawl took 33s.

Here's what I got in `errors.txt`:
```

     0K .                                                      71.6M=0s

2020-10-15 15:23:09 (71.6 MB/s) - ‘localhost:8000/datasette/index.html’ saved [1276]

Loading robots.txt; please ignore errors.
--2020-10-15 15:23:09--  http://localhost:8000/robots.txt
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
--
--2020-10-15 15:23:09--  http://localhost:8000/robots.txt
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:09 ERROR 404: Not Found.

--2020-10-15 15:23:09--  http://localhost:8000/datasette/-/static/app.css?b576be
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: 8563 (8.4K) [text/css]
--
--
2020-10-15 15:23:13 (7.90 MB/s) - ‘localhost:8000/datasette/fixtures/primary_key_multiple_columns_explicit_label.json?_shape=object’ saved [58]

--2020-10-15 15:23:13--  http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:13 ERROR 404: Not Found.

--2020-10-15 15:23:13--  http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+limit+101
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
--
--
2020-10-15 15:23:13 (84.3 MB/s) - ‘localhost:8000/datasette/fixtures/roadside_attractions.json?_shape=object’ saved [619]

--2020-10-15 15:23:13--  http://localhost:8000/datasette/fixtures/%5C%22https://www.openstreetmap.org/copyright%5C%22
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:13 ERROR 404: Not Found.

--2020-10-15 15:23:13--  http://localhost:8000/datasette/fixtures?sql=select+pk%2C+text1%2C+text2%2C+%5Bname+with+.+and+spaces%5D+from+searchable+order+by+pk+limit+101
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
--
--
2020-10-15 15:23:14 (28.6 MB/s) - ‘localhost:8000/datasette/fixtures/searchable_view_configured_by_metadata.json?_shape=array&_nl=on’ saved [180]

--2020-10-15 15:23:14--  http://localhost:8000/
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:14 ERROR 404: Not Found.

--2020-10-15 15:23:14--  http://localhost:8000/datasette/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101&_hide_sql=1
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
--
--
2020-10-15 15:23:21 (64.1 MB/s) - ‘localhost:8000/datasette/fixtures.csv?sql=select+pk,+name,+address,+latitude,+longitude+from+roadside_attractions+order+by+pk+limit+101&_size=max’ saved [403]

--2020-10-15 15:23:21--  http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:21 ERROR 404: Not Found.

--2020-10-15 15:23:21--  http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+desc+limit+101
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709625063,https://api.github.com/repos/simonw/datasette/issues/1024,709625063,MDEyOklzc3VlQ29tbWVudDcwOTYyNTA2Mw==,9599,simonw,2020-10-15T22:33:22Z,2020-10-15T22:33:22Z,OWNER,"Of those errors...

`http://localhost:8000/robots.txt` 404 is fine.

`http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22` looks to me like a `wget` parsing bug where it got confused by this JavaScript:

```
window.DATASETTE_CLUSTER_MAP_TILE_LAYER_OPTIONS = {""maxZoom"": 19, ""detectRetina"": true, ""attribution"": ""&copy; <a href=\""https://www.openstreetmap.org/copyright\"">OpenStreetMap</a> contributors""};</script>
```

`http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js` is a real bug. It's a bug in `datasette-cluster-map` but also requires me to solve #988 - mechanism for plugins to construct URLs that obey `base_url`.

I'm not sure why I'm getting a hit to `http://localhost:8000/` since I wouldn't expect to link to `/` anywhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1025#issuecomment-709629920,https://api.github.com/repos/simonw/datasette/issues/1025,709629920,MDEyOklzc3VlQ29tbWVudDcwOTYyOTkyMA==,9599,simonw,2020-10-15T22:48:20Z,2020-10-15T22:48:20Z,OWNER,"Also these:
```
datasette % git grep '""/' -- '*.html' ':(exclude)*/patterns.html'
datasette/templates/allow_debug.html:<form action=""/-/allow-debug"" method=""get"">
datasette/templates/base.html:        <form action=""/-/logout"" method=""post"">
datasette/templates/error.html:        <a href=""/"">home</a>
datasette/templates/logout.html:<form action=""/-/logout"" method=""post"">
datasette/templates/messages_debug.html:<form action=""/-/messages"" method=""post"">
datasette/templates/query.html:        <a href=""/"">home</a> /
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-709632136,https://api.github.com/repos/simonw/datasette/issues/1025,709632136,MDEyOklzc3VlQ29tbWVudDcwOTYzMjEzNg==,9599,simonw,2020-10-15T22:55:44Z,2020-10-15T22:55:44Z,OWNER,"It looks like there are also some generated redirect responses that don't take `base_url` into account:
```
datasette % git grep '""/' -- '*.py' ':(exclude)*test_*.py' ':(exclude)datasette/app.py'
datasette/_version.py:        for i in cfg.versionfile_source.split(""/""):
datasette/utils/asgi.py:        path=""/"",
datasette/views/base.py:            should_redirect = ""/{}-{}"".format(name, expected)
datasette/views/base.py:                should_redirect += ""/"" + urllib.parse.quote_plus(kwargs[""table""])
datasette/views/base.py:                should_redirect += ""/"" + kwargs[""pk_path""]
datasette/views/special.py:            response = Response.redirect(""/"")
datasette/views/special.py:            return Response.redirect(""/"")
datasette/views/special.py:        response = Response.redirect(""/"")
datasette/views/special.py:        return Response.redirect(""/"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-709632314,https://api.github.com/repos/simonw/datasette/issues/1025,709632314,MDEyOklzc3VlQ29tbWVudDcwOTYzMjMxNA==,9599,simonw,2020-10-15T22:56:25Z,2020-10-15T22:56:34Z,OWNER,"That `utils/asgi.py` line is the default path for setting cookies. That should likely take `base_url` into account too:
https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/utils/asgi.py#L331-L342","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-712481127,https://api.github.com/repos/simonw/datasette/issues/1025,712481127,MDEyOklzc3VlQ29tbWVudDcxMjQ4MTEyNw==,9599,simonw,2020-10-19T22:40:37Z,2020-10-20T01:21:36Z,OWNER,Was blocked on #904 - now unblocked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-712525557,https://api.github.com/repos/simonw/datasette/issues/1025,712525557,MDEyOklzc3VlQ29tbWVudDcxMjUyNTU1Nw==,9599,simonw,2020-10-20T01:07:02Z,2020-10-20T01:07:02Z,OWNER,"I fixed the `queries.html` one. I'm not going to fix these two:
```
datasette/templates/error.html:        <a href=""/"">home</a>
datasette/templates/patterns.html:        <a href=""/"">home</a> /
```
Because the `error.html` one does not get passed a context (which makes sense since an error has occurred) and the pattern portfolio doesn't need to link to anywhere in particular.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-712579674,https://api.github.com/repos/simonw/datasette/issues/1025,712579674,MDEyOklzc3VlQ29tbWVudDcxMjU3OTY3NA==,9599,simonw,2020-10-20T04:24:10Z,2020-10-20T04:24:10Z,OWNER,"Changed my mind, `error.html` needs access to `urls` in order to link to its CSS file. Passing it after all (it already got passed `ds.config(""base_url"")` so `ds` was available previously).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-712593790,https://api.github.com/repos/simonw/datasette/issues/1025,712593790,MDEyOklzc3VlQ29tbWVudDcxMjU5Mzc5MA==,9599,simonw,2020-10-20T05:12:36Z,2020-10-20T05:12:36Z,OWNER,"I'm going to leave the cookies code setting cookies to default to the `""/""` top level.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1026#issuecomment-709636372,https://api.github.com/repos/simonw/datasette/issues/1026,709636372,MDEyOklzc3VlQ29tbWVudDcwOTYzNjM3Mg==,9599,simonw,2020-10-15T23:09:34Z,2020-10-15T23:09:34Z,OWNER,"I'm inclined to say that internal requests should ignore `base_url` - since that seems like the right thing for plugins that need to access default Datasette APIs.

The one catch here is plugins that might want to proxy the current incoming URL for some reason - where that incoming `request.path` could include the `base_url`.

Actually those should be fine - because it will have been stripped off earlier:

https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/app.py#L963-L968","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1026#issuecomment-712607227,https://api.github.com/repos/simonw/datasette/issues/1026,712607227,MDEyOklzc3VlQ29tbWVudDcxMjYwNzIyNw==,9599,simonw,2020-10-20T05:46:44Z,2020-10-20T05:46:44Z,OWNER,We have a solution for this now: `datasette.urls` from #1033 can be used by plugins to assemble the correct URLs to pass to `.get()` and friends.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1026#issuecomment-712959034,https://api.github.com/repos/simonw/datasette/issues/1026,712959034,MDEyOklzc3VlQ29tbWVudDcxMjk1OTAzNA==,9599,simonw,2020-10-20T16:03:33Z,2020-10-20T16:03:55Z,OWNER,"Reconsidering this: I think the `.get()` etc methods should automatically add the `base_url` prefix for you, since these APIs are only intended to make internal calls.

The clincher on this is when I went to add a section to the `datasette.client` documentation recommending you use `datasette.urls.path()` for every call to them that you make.

But there's a problem: to handle table name escaping users are likely to want to use `datasette.urls.table()` anyway, like this:

    response = await datasette.client.get(datasette.urls.table(""db"", ""table"") + "".json"")

This risks adding the `base_url` prefix twice. Maybe the `.table()` method could return a string-like object that is marked as already having the `base_url` prefix added, so the `client.get()` method knows not to add it again.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1026#issuecomment-712962517,https://api.github.com/repos/simonw/datasette/issues/1026,712962517,MDEyOklzc3VlQ29tbWVudDcxMjk2MjUxNw==,9599,simonw,2020-10-20T16:09:12Z,2020-10-20T16:09:12Z,OWNER,"That `datasette.urls.table(""db"", ""table"") + "".json""` example is bad because if the table name contains a `.` it should be `?_format=json` instead.

Maybe `.table()` should have a `format=""json""` option that knows how to do this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1026#issuecomment-719959754,https://api.github.com/repos/simonw/datasette/issues/1026,719959754,MDEyOklzc3VlQ29tbWVudDcxOTk1OTc1NA==,9599,simonw,2020-10-31T16:56:35Z,2020-10-31T16:56:35Z,OWNER,#1041 can also benefit from the string subclass that shows that `base_url` has been added.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1027#issuecomment-709646865,https://api.github.com/repos/simonw/datasette/issues/1027,709646865,MDEyOklzc3VlQ29tbWVudDcwOTY0Njg2NQ==,9599,simonw,2020-10-15T23:47:08Z,2020-10-15T23:47:08Z,OWNER,It should cover both nginx and Apache. nginx config is here: https://github.com/simonw/datasette/issues/1024#issuecomment-709598324,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/1027#issuecomment-709647525,https://api.github.com/repos/simonw/datasette/issues/1027,709647525,MDEyOklzc3VlQ29tbWVudDcwOTY0NzUyNQ==,9599,simonw,2020-10-15T23:49:51Z,2020-10-15T23:51:39Z,OWNER,"I'll install Apache on macOS to figure this out using https://formulae.brew.sh/formula/httpd

`brew install httpd` output this at the end:

```
==> httpd
DocumentRoot is /usr/local/var/www.

The default ports have been set in /usr/local/etc/httpd/httpd.conf to 8080 and in
/usr/local/etc/httpd/extra/httpd-ssl.conf to 8443 so that httpd can run without sudo.

To have launchd start httpd now and restart at login:
  brew services start httpd
Or, if you don't want/need a background service you can just run:
  apachectl start
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/1027#issuecomment-710750038,https://api.github.com/repos/simonw/datasette/issues/1027,710750038,MDEyOklzc3VlQ29tbWVudDcxMDc1MDAzOA==,9599,simonw,2020-10-17T05:00:19Z,2020-10-17T05:00:19Z,OWNER,"Running `apachectl -X` appears to run Apache in the foreground so I can hit Ctrl+C to quit it again, using the config file at `/usr/local/etc/httpd/httpd.conf` and serving the `index.html` file from `/usr/local/var/www` (which confusingly says ""Welcome to nginx"", presumably because I installed nginx via Homebrew earlier).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/1027#issuecomment-712320103,https://api.github.com/repos/simonw/datasette/issues/1027,712320103,MDEyOklzc3VlQ29tbWVudDcxMjMyMDEwMw==,9599,simonw,2020-10-19T17:35:04Z,2020-10-19T17:35:04Z,OWNER,Still need to configure proxying though. https://www.netnea.com/cms/apache-tutorial-9_setting-up-a-reverse-proxy/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/1027#issuecomment-719988113,https://api.github.com/repos/simonw/datasette/issues/1027,719988113,MDEyOklzc3VlQ29tbWVudDcxOTk4ODExMw==,9599,simonw,2020-10-31T21:03:37Z,2020-10-31T21:03:37Z,OWNER,"On my Mac, I run:

    datasette . -p 8009 --config base_url:/datasette-prefix/

Then I edited `/usr/local/etc/httpd/httpd.conf` and add this section:
```
LoadModule proxy_module lib/httpd/modules/mod_proxy.so
LoadModule proxy_http_module lib/httpd/modules/mod_proxy_http.so
ProxyPass          /datasette-prefix/    http://localhost:8009/datasette-prefix/
```
I ran Apache in the foreground like so:

    apachectl -X

Now hitting http://localhost:8081/datasette-prefix/fixtures/compound_three_primary_keys worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/1028#issuecomment-712480866,https://api.github.com/repos/simonw/datasette/issues/1028,712480866,MDEyOklzc3VlQ29tbWVudDcxMjQ4MDg2Ng==,9599,simonw,2020-10-19T22:39:51Z,2020-10-19T22:39:51Z,OWNER,Documentation: https://docs.datasette.io/en/latest/spatialite.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723803777,--load-extension=spatialite shortcut,
https://github.com/simonw/datasette/issues/103#issuecomment-754188099,https://api.github.com/repos/simonw/datasette/issues/103,754188099,MDEyOklzc3VlQ29tbWVudDc1NDE4ODA5OQ==,9599,simonw,2021-01-04T20:05:14Z,2021-01-04T20:05:14Z,OWNER,"Wontfix, Cloud Run is already implemented and is a better fit for Datasette.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274265878,datasette publish appengine,
https://github.com/simonw/datasette/issues/1032#issuecomment-712363344,https://api.github.com/repos/simonw/datasette/issues/1032,712363344,MDEyOklzc3VlQ29tbWVudDcxMjM2MzM0NA==,9599,simonw,2020-10-19T18:31:46Z,2020-10-19T18:31:46Z,OWNER,"As always with dates, the challenge is America. `mm/dd/yy` is indistinguishable from the more sensible `dd/mm/yy`. It's crucially important to visibly tell people which format you assumed, otherwise all kinds of weird invisible bugs can manifest.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712363825,https://api.github.com/repos/simonw/datasette/issues/1032,712363825,MDEyOklzc3VlQ29tbWVudDcxMjM2MzgyNQ==,9599,simonw,2020-10-19T18:32:43Z,2020-10-19T18:32:43Z,OWNER,"Horrible thought: I bet there is data out there that uses more than one date format in the same table! So this needs to be exposed as a visible per-column setting.

Column action menu can help here, although it's not yet the full solution because it isn't yet visible on mobile.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712364317,https://api.github.com/repos/simonw/datasette/issues/1032,712364317,MDEyOklzc3VlQ29tbWVudDcxMjM2NDMxNw==,9599,simonw,2020-10-19T18:33:42Z,2020-10-19T18:33:42Z,OWNER,"Related challenge: timezones. I think I'll punt on those for the moment and just concentrate on dates, but I should keep them in mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712364532,https://api.github.com/repos/simonw/datasette/issues/1032,712364532,MDEyOklzc3VlQ29tbWVudDcxMjM2NDUzMg==,9599,simonw,2020-10-19T18:34:10Z,2020-10-19T18:34:10Z,OWNER,Lots of great example date data in https://biglocal.datasettes.com/COVID_WARN_Notices,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712365236,https://api.github.com/repos/simonw/datasette/issues/1032,712365236,MDEyOklzc3VlQ29tbWVudDcxMjM2NTIzNg==,9599,simonw,2020-10-19T18:35:25Z,2020-10-19T18:35:25Z,OWNER,"Since I'm dealing with tables of data I usually have a whole column of examples, so heuristics that check for numbers-greater-than-12 could actually work well for many cases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712365439,https://api.github.com/repos/simonw/datasette/issues/1032,712365439,MDEyOklzc3VlQ29tbWVudDcxMjM2NTQzOQ==,9599,simonw,2020-10-19T18:35:50Z,2020-10-19T18:37:57Z,OWNER,"Maybe I don't need to add `dateutil` as a dependency here?

`pendulum` and `arrow` both depend on it so it's pretty deeply embedded in the Python date ecosystem. Adding it as a dependency seems reasonable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712367285,https://api.github.com/repos/simonw/datasette/issues/1032,712367285,MDEyOklzc3VlQ29tbWVudDcxMjM2NzI4NQ==,9599,simonw,2020-10-19T18:39:32Z,2020-10-19T18:39:32Z,OWNER,https://github.com/digital-land/brownfield-land-collection/blob/a09ddf9960a6af59e72dc02448f7b645e59bf227/bin/harmonise.py#L217-L247 is a beautiful example of this problem.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712397537,https://api.github.com/repos/simonw/datasette/issues/1032,712397537,MDEyOklzc3VlQ29tbWVudDcxMjM5NzUzNw==,236498,saulpw,2020-10-19T19:37:55Z,2020-10-19T19:37:55Z,NONE,"python-dateutil is awesome, but it can only guess at one date at a time. So if you have a column of dates that are (presumably) in the same format, it can't use the full set of dates to deduce the format.  Also, once it has parsed a date, you can't get the format it used, whether to parse or render other dates.  These limitations prevent it from being a silver bullet for date parsing, though they're not enough for me to stop using it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1033#issuecomment-712529413,https://api.github.com/repos/simonw/datasette/issues/1033,712529413,MDEyOklzc3VlQ29tbWVudDcxMjUyOTQxMw==,9599,simonw,2020-10-20T01:21:12Z,2020-10-20T01:21:12Z,OWNER,Also refs #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method,
https://github.com/simonw/datasette/issues/1033#issuecomment-714657366,https://api.github.com/repos/simonw/datasette/issues/1033,714657366,MDEyOklzc3VlQ29tbWVudDcxNDY1NzM2Ng==,82988,psychemedia,2020-10-22T17:51:29Z,2020-10-22T17:51:29Z,CONTRIBUTOR,"How does `/-/static` relate to [current guidance docs around `static`](https://docs.datasette.io/en/latest/custom_templates.html?highlight=static#serving-static-files) regarding the `--static option` and metadata formulations such as `""extra_js_urls"": [ ""/static/app.js""]` (I've not managed to get this to work in a Jupyter server proxied set up; the [datasette / jupyter server proxy repo](https://github.com/simonw/jupyterserverproxy-datasette-demo) may provide a useful test example, eg via MyBinder, for folk to crib from?) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method,
https://github.com/simonw/datasette/issues/1033#issuecomment-714681365,https://api.github.com/repos/simonw/datasette/issues/1033,714681365,MDEyOklzc3VlQ29tbWVudDcxNDY4MTM2NQ==,9599,simonw,2020-10-22T18:33:48Z,2020-10-22T18:33:48Z,OWNER,"That's a good question - I hadn't considered that. I'm going to open a new issue to have `extra_js_urls` respect the `base_url` setting, since the static files will be served from a different location.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method,
https://github.com/simonw/datasette/issues/1033#issuecomment-716048564,https://api.github.com/repos/simonw/datasette/issues/1033,716048564,MDEyOklzc3VlQ29tbWVudDcxNjA0ODU2NA==,9599,simonw,2020-10-24T20:08:31Z,2020-10-24T20:08:31Z,OWNER,Documentation here: https://docs.datasette.io/en/latest/internals.html#datasette-urls,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method,
https://github.com/simonw/datasette/issues/1033#issuecomment-716066000,https://api.github.com/repos/simonw/datasette/issues/1033,716066000,MDEyOklzc3VlQ29tbWVudDcxNjA2NjAwMA==,82988,psychemedia,2020-10-24T22:58:33Z,2020-10-24T22:58:33Z,CONTRIBUTOR,"From [the docs](https://docs.datasette.io/en/latest/internals.html#datasette-urls), I note:

```
datasette.urls.instance()
Returns the URL to the Datasette instance root page. This is usually ""/""
```

What about the proxy case? Eg if I am using jupyter-server-proxy on a MyBinder or local Jupyter notebook server site, `https://example.com:PORT/weirdpath/datasette`, what does `datasette.urls.instance()` refer to?

- [ ] `https://example.com:PORT/weirdpath/datasette`
- [ ] `https://example.com:PORT/weirdpath/`
- [ ] `https://example.com:PORT/`
- [ ] `https://example.com`
- [ ] something else?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method,
https://github.com/simonw/datasette/issues/1034#issuecomment-712580976,https://api.github.com/repos/simonw/datasette/issues/1034,712580976,MDEyOklzc3VlQ29tbWVudDcxMjU4MDk3Ng==,9599,simonw,2020-10-20T04:29:23Z,2020-10-20T04:29:23Z,OWNER,Most obvious option is base64. Any other potential solutions I'm missing?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-712581994,https://api.github.com/repos/simonw/datasette/issues/1034,712581994,MDEyOklzc3VlQ29tbWVudDcxMjU4MTk5NA==,9599,simonw,2020-10-20T04:33:28Z,2020-10-20T04:33:28Z,OWNER,"The [datasette-render-binary](https://github.com/simonw/datasette-render-binary) plugin does this, which I really like - but without the different coloured fonts I'm not sure how readable it would be as just plain text:

![image](https://user-images.githubusercontent.com/9599/96540435-9c125f00-1252-11eb-85aa-5fc8d0e63728.png)

Really the goal here is to find the most human-friendly option, so that people looking at the output have a vague idea what's going on. That's why I'm not leaping at the chance to use base64.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-712582699,https://api.github.com/repos/simonw/datasette/issues/1034,712582699,MDEyOklzc3VlQ29tbWVudDcxMjU4MjY5OQ==,9599,simonw,2020-10-20T04:36:04Z,2020-10-20T04:36:14Z,OWNER,Asked for ideas on Twitter: https://twitter.com/simonw/status/1318409558805467136,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713172901,https://api.github.com/repos/simonw/datasette/issues/1034,713172901,MDEyOklzc3VlQ29tbWVudDcxMzE3MjkwMQ==,9599,simonw,2020-10-20T22:19:10Z,2020-10-20T22:20:28Z,OWNER,"I could go with the same format as `datasette-render-binary` but using `0x00` as the format for the hex bytes.

    0x15 0x1C 0x02 0xC7 JFIF 0x00 0x01

Problem with this is that it's ambiguous: if the ASCII characters `0x15` occur in the text they will be indistinguishable from those hex bytes.

But since representing binary data in CSV fundamentally doesn't make sense I'm not sure if that really matters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713174341,https://api.github.com/repos/simonw/datasette/issues/1034,713174341,MDEyOklzc3VlQ29tbWVudDcxMzE3NDM0MQ==,9599,simonw,2020-10-20T22:22:53Z,2020-10-20T22:23:14Z,OWNER,"An even easier option: do what the Datasette UI does and output `<Binary data: 7 bytes>` for that CSV cell, as seen on https://latest.datasette.io/fixtures/binary_data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713174690,https://api.github.com/repos/simonw/datasette/issues/1034,713174690,MDEyOklzc3VlQ29tbWVudDcxMzE3NDY5MA==,9599,simonw,2020-10-20T22:23:50Z,2020-10-20T22:23:50Z,OWNER,Or... default to `<Binary data: 7 bytes>` and support a `?_base64=1` option which outputs in base64 instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713175741,https://api.github.com/repos/simonw/datasette/issues/1034,713175741,MDEyOklzc3VlQ29tbWVudDcxMzE3NTc0MQ==,9599,simonw,2020-10-20T22:26:45Z,2020-10-20T22:26:45Z,OWNER,"> New idea: since binary in CSV doesn't make sense anyway, emulate Datasette's HTML UI default and output this:
> 
>     id,title,data
>     1,Some title,<Binary data: 14 bytes>
>     2,Other title,<Binary data: 57 bytes>
> 
> Then allow users to add ?_base64=1 to the URL to get base64 instead
> https://twitter.com/simonw/status/1318679950635888641","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713176082,https://api.github.com/repos/simonw/datasette/issues/1034,713176082,MDEyOklzc3VlQ29tbWVudDcxMzE3NjA4Mg==,9599,simonw,2020-10-20T22:27:33Z,2020-10-20T22:27:33Z,OWNER,"This feels good to me - it's consistent with how other features in Datasette work, and it means users who need the binary data in CSV (for whatever reason) can get it if they want to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713191819,https://api.github.com/repos/simonw/datasette/issues/1034,713191819,MDEyOklzc3VlQ29tbWVudDcxMzE5MTgxOQ==,9599,simonw,2020-10-20T23:12:58Z,2020-10-20T23:12:58Z,OWNER,"Enzo has a great solution here: https://twitter.com/enzo_mdd/status/1318685442976436226

> Or maybe an option for a url. This keeps the CSV small but allows scripts to download binary data as needed.

In #1036 I'm planning on adding a way for users to access BLOB data. I can include that URL in the CSV output.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713277810,https://api.github.com/repos/simonw/datasette/issues/1034,713277810,MDEyOklzc3VlQ29tbWVudDcxMzI3NzgxMA==,9599,simonw,2020-10-21T03:40:50Z,2020-10-25T01:01:23Z,OWNER,Blocked awaiting #1036 (update: now unblocked),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716077436,https://api.github.com/repos/simonw/datasette/issues/1034,716077436,MDEyOklzc3VlQ29tbWVudDcxNjA3NzQzNg==,9599,simonw,2020-10-25T01:08:35Z,2020-10-25T01:08:42Z,OWNER,"This is actually a bit tricky to implement, for a few reasons:

- Need to generate a full URL, including the `https://host/` bit. I've done this for `next_url` in the JSON output before, thankfully.
- This only makes sense for CSV output for tables. If it's the CSV output of an arbitrary query there's no `/db/table/-/blob/pk/column.blob` page for me to link to.
- Need to generate those `/.../-/blob/...` URLs for the data that is being output as CSV.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716077508,https://api.github.com/repos/simonw/datasette/issues/1034,716077508,MDEyOklzc3VlQ29tbWVudDcxNjA3NzUwOA==,9599,simonw,2020-10-25T01:09:17Z,2020-10-25T01:09:17Z,OWNER,Here's how those absolute `next_url` values are generated: https://github.com/simonw/datasette/blob/5db7ae3ce165ded57c7fb1cfbdb3258b1cf06c10/datasette/views/table.py#L774-L776,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716077541,https://api.github.com/repos/simonw/datasette/issues/1034,716077541,MDEyOklzc3VlQ29tbWVudDcxNjA3NzU0MQ==,9599,simonw,2020-10-25T01:09:38Z,2020-10-25T01:10:04Z,OWNER,I should turn `datasette.absolute_url(...)` into a documented internal API on https://docs.datasette.io/en/stable/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716078420,https://api.github.com/repos/simonw/datasette/issues/1034,716078420,MDEyOklzc3VlQ29tbWVudDcxNjA3ODQyMA==,9599,simonw,2020-10-25T01:20:00Z,2020-10-25T01:20:00Z,OWNER,That documentation: https://docs.datasette.io/en/latest/internals.html#absolute-url-request-path,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716078512,https://api.github.com/repos/simonw/datasette/issues/1034,716078512,MDEyOklzc3VlQ29tbWVudDcxNjA3ODUxMg==,9599,simonw,2020-10-25T01:21:11Z,2020-10-25T01:21:11Z,OWNER,"What should happen for CSV export of arbitrary SQL queries, where there's no obvious BLOB to link to?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716078605,https://api.github.com/repos/simonw/datasette/issues/1034,716078605,MDEyOklzc3VlQ29tbWVudDcxNjA3ODYwNQ==,9599,simonw,2020-10-25T01:22:22Z,2020-10-25T01:22:22Z,OWNER,For arbitrary CSV the only solution I can think of is to embed the base64 value.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716078777,https://api.github.com/repos/simonw/datasette/issues/1034,716078777,MDEyOklzc3VlQ29tbWVudDcxNjA3ODc3Nw==,9599,simonw,2020-10-25T01:25:11Z,2020-10-25T01:25:11Z,OWNER,"SQLite actually has APIs that could help here: https://www.sqlite.org/c3ref/column_database_name.html - for any given SQL query they identify the origin/table/column that is the source of each resulting column.

Those aren't exposed in the Python `sqlite3` module though, so using them could be extremely tricky.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-719050754,https://api.github.com/repos/simonw/datasette/issues/1034,719050754,MDEyOklzc3VlQ29tbWVudDcxOTA1MDc1NA==,9599,simonw,2020-10-29T22:04:52Z,2020-10-29T22:04:52Z,OWNER,I'm going to link to. the new `.blob` representation using the new `?_blob_hash=xxx` argument to ensure that the content served is the expected binary blob.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-719094027,https://api.github.com/repos/simonw/datasette/issues/1034,719094027,MDEyOklzc3VlQ29tbWVudDcxOTA5NDAyNw==,9599,simonw,2020-10-30T00:11:17Z,2020-10-30T00:11:17Z,OWNER,"Demos:

https://latest.datasette.io/fixtures/binary_data.csv?_size=max

```csv
rowid,data
1,http://latest.datasette.io/fixtures/binary_data/1.blob?_blob_column=data
2,http://latest.datasette.io/fixtures/binary_data/2.blob?_blob_column=data
3,
```

https://latest.datasette.io/fixtures.csv?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max

```csv
rowid,data
1,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=f3088978da8f9aea479ffc7f631370b968d2e855eeb172bea7f6c7a04262bb6d
2,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=b835b0483cedb86130b9a2c280880bf5fadc5318ddf8c18d0df5204d40df1724
3,
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1035#issuecomment-712963959,https://api.github.com/repos/simonw/datasette/issues/1035,712963959,MDEyOklzc3VlQ29tbWVudDcxMjk2Mzk1OQ==,9599,simonw,2020-10-20T16:11:25Z,2020-10-20T16:11:25Z,OWNER,"Relevant code:

https://github.com/simonw/datasette/blob/091441a4449beae559a8c0d007376dc85d3aa624/datasette/utils/__init__.py#L681-L696

Only used here:

https://github.com/simonw/datasette/blob/091441a4449beae559a8c0d007376dc85d3aa624/datasette/views/base.py#L498-L502","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument",
https://github.com/simonw/datasette/issues/1035#issuecomment-712965574,https://api.github.com/repos/simonw/datasette/issues/1035,712965574,MDEyOklzc3VlQ29tbWVudDcxMjk2NTU3NA==,9599,simonw,2020-10-20T16:13:57Z,2020-10-20T16:13:57Z,OWNER,"That `renderers[key] = path_with_format(` is in a base class which can be used for both arbitrary queries, canned queries and the table view. I think that's OK, but it means that the `format=""json""` argument on `datasette.urls.table()` won't be used by Datasette internally, it will just be available for plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument",
https://github.com/simonw/datasette/issues/1035#issuecomment-712976314,https://api.github.com/repos/simonw/datasette/issues/1035,712976314,MDEyOklzc3VlQ29tbWVudDcxMjk3NjMxNA==,9599,simonw,2020-10-20T16:21:42Z,2020-10-20T16:21:42Z,OWNER,"Makes me question if `datasette.urls` should grow functionality equivalent to the other path and querystring manipulation methods in `datasette.utils`:

https://github.com/simonw/datasette/blob/66120a7a1cb592e8a21164cf537f62a4d7ab1dfc/datasette/utils/__init__.py#L216-L279","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument",
https://github.com/simonw/datasette/issues/1036#issuecomment-713183306,https://api.github.com/repos/simonw/datasette/issues/1036,713183306,MDEyOklzc3VlQ29tbWVudDcxMzE4MzMwNg==,9599,simonw,2020-10-20T22:48:10Z,2020-10-20T22:48:10Z,OWNER,Twitter thread: https://twitter.com/dancow/status/1318681053347840005,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713184374,https://api.github.com/repos/simonw/datasette/issues/1036,713184374,MDEyOklzc3VlQ29tbWVudDcxMzE4NDM3NA==,9599,simonw,2020-10-20T22:51:22Z,2020-10-20T22:51:22Z,OWNER,"From https://hackerone.com/reports/126197:

> archive.uber.com mirrors pypi. When downloading `.tar.gz` files from archive.uber.com, the MIME type is `application/octet-stream`. Injecting `<html><script>alert(0)</script>` into the start of the `.tar.gz` causes an XSS in Internet Explorer due to MIME sniffing.

So you do have to be careful not to open accidental XSS holes with `application/octet-stream` thanks to (presumably older) versions of IE.

From that thread it looks like the solution is to add a `X-Content-Type-Options: nosniff` header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713185173,https://api.github.com/repos/simonw/datasette/issues/1036,713185173,MDEyOklzc3VlQ29tbWVudDcxMzE4NTE3Mw==,9599,simonw,2020-10-20T22:53:41Z,2020-10-20T22:53:41Z,OWNER,"https://security.stackexchange.com/questions/12896/does-x-content-type-options-really-prevent-content-sniffing-attacks says:

> In Tangled Web Michal Zalewski says:
> 
> > Refrain from using Content-Type: application/octet-stream and use application/binary instead, especially for unknown document types. Refrain from returning Content-Type: text/plain.
> > 
> > For example, any code-hosting platform must exercise caution when returning executables or source archives as application/octet-stream, because there is a risk they may be misinterpreted as HTML and displayed inline.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713185871,https://api.github.com/repos/simonw/datasette/issues/1036,713185871,MDEyOklzc3VlQ29tbWVudDcxMzE4NTg3MQ==,9599,simonw,2020-10-20T22:55:36Z,2020-10-20T22:55:36Z,OWNER,I can also use a `Content-Disposition` header to force a download. I'm reasonably confident that the combination of `Content-Disposition` and `X-Content-Type-Options: nosniff` and `application/binary` will let me allow users to download the contents of arbitrary BLOB columns without any XSS risk.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713186189,https://api.github.com/repos/simonw/datasette/issues/1036,713186189,MDEyOklzc3VlQ29tbWVudDcxMzE4NjE4OQ==,9599,simonw,2020-10-20T22:56:33Z,2020-10-20T22:56:33Z,OWNER,I think this plus the binary-CSV stuff in #1034 will justify a dedicated section of the documentation to talk about how Datasette handles binary BLOB columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713226726,https://api.github.com/repos/simonw/datasette/issues/1036,713226726,MDEyOklzc3VlQ29tbWVudDcxMzIyNjcyNg==,9599,simonw,2020-10-21T01:04:25Z,2020-10-21T01:04:25Z,OWNER,"Extra security idea: a `blob_download_host` setting which can be used to indicate a host that should be used for downloads - for example `datasettestatic.com`. If this setting is populated then binary downloads are served from paths on that host only, and no other Datasette URLs from that host will be served.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713278349,https://api.github.com/repos/simonw/datasette/issues/1036,713278349,MDEyOklzc3VlQ29tbWVudDcxMzI3ODM0OQ==,9599,simonw,2020-10-21T03:42:29Z,2020-10-21T03:42:29Z,OWNER,Possible URL for this: `/db/table/-/blob/primary-keys` - this would use the `/db/table/-/` namespace proposed in #296.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713818178,https://api.github.com/repos/simonw/datasette/issues/1036,713818178,MDEyOklzc3VlQ29tbWVudDcxMzgxODE3OA==,9599,simonw,2020-10-21T19:15:38Z,2020-10-21T19:16:34Z,OWNER,"What should the suggested filename be?

I think something that includes the table name, primary key and the name of the column would work.

How about a file extension? I guess `.binary`, then let the user rename it? Or `.raw`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713818817,https://api.github.com/repos/simonw/datasette/issues/1036,713818817,MDEyOklzc3VlQ29tbWVudDcxMzgxODgxNw==,9599,simonw,2020-10-21T19:17:01Z,2020-10-21T19:17:01Z,OWNER,Actually I like `.blob`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713821656,https://api.github.com/repos/simonw/datasette/issues/1036,713821656,MDEyOklzc3VlQ29tbWVudDcxMzgyMTY1Ng==,9599,simonw,2020-10-21T19:22:45Z,2020-10-21T19:41:48Z,OWNER,"So for https://latest.datasette.io/fixtures/binary_data the BLOB download URLs would be:

`https://latest.datasette.io/fixtures/-/blob/binary_data/1/data.blob` - that last bit after the primary key is to indicate the `data` column

With these headers:

- `Content-Disposition: attachment; filename=""binary_data-1-data.blob""`
- `X-Content-Type-Options: nosniff`
- `Content-Type: application/binary`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713829629,https://api.github.com/repos/simonw/datasette/issues/1036,713829629,MDEyOklzc3VlQ29tbWVudDcxMzgyOTYyOQ==,9599,simonw,2020-10-21T19:38:43Z,2020-10-21T19:38:43Z,OWNER,"Should this work just for BLOB columns, or should it work for other columns too?

For the moment I'm going to restrict it to BLOBs, since data from other columns is available through the UI whereas BLOB columns are not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713830842,https://api.github.com/repos/simonw/datasette/issues/1036,713830842,MDEyOklzc3VlQ29tbWVudDcxMzgzMDg0Mg==,9599,simonw,2020-10-21T19:41:20Z,2020-10-21T19:41:20Z,OWNER,Another useful demo database: https://datasette-render-images-demo.datasette.io/favicons/favicons - see https://datasette-render-images-demo.datasette.io/favicons/favicons.csv,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713899530,https://api.github.com/repos/simonw/datasette/issues/1036,713899530,MDEyOklzc3VlQ29tbWVudDcxMzg5OTUzMA==,9599,simonw,2020-10-21T21:55:00Z,2020-10-21T21:55:00Z,OWNER,"This code needs these permission checks:
https://github.com/simonw/datasette/blob/bf82b3d6a605c9ddadd5fb739249dfe6defaf635/datasette/views/table.py#L911-L913","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-762385981,https://api.github.com/repos/simonw/datasette/issues/1036,762385981,MDEyOklzc3VlQ29tbWVudDc2MjM4NTk4MQ==,4997607,philshem,2021-01-18T17:32:13Z,2021-01-18T17:34:50Z,NONE,"Hi Simon

Just finding this old issue regarding downloading blobs. Nice work!

<img width=""212"" alt=""image"" src=""https://user-images.githubusercontent.com/4997607/104946741-df4bad80-59ba-11eb-96e3-727c85cc4dc6.png"">

As a feature request, maybe it would be possible to assign a blob column as a certain data type (e.g. `.jpg`) and then each blob could be downloaded as that type of file (perhaps if the file types were constrained to normal blobs that people store in sqlite databases, this could avoid the execution stuff mentioned above).

I guess the column blob-type definition could fit into this dropdown selection:

<img width=""201"" alt=""image"" src=""https://user-images.githubusercontent.com/4997607/104947000-479a8f00-59bb-11eb-87d9-1644e5940894.png"">

Let me know if I should open a new issue with a feature request. (This could slowly go in the direction of displaying image blob-types in the browser.)

Thanks for the great tool!


---

edit: just reading the rest of the twitter thread: https://twitter.com/simonw/status/1318685933256855552

perhaps this is already possible in some form with the plugin datasette-media: https://github.com/simonw/datasette-media","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-762387875,https://api.github.com/repos/simonw/datasette/issues/1036,762387875,MDEyOklzc3VlQ29tbWVudDc2MjM4Nzg3NQ==,9599,simonw,2021-01-18T17:36:36Z,2021-01-18T17:36:36Z,OWNER,"As you can see, I'm pretty paranoid about serving content with `Content-Type` HTTP headers because I'm so worried about execution vulnerabilities. I'm much more comfortable exploring that kind of thing in plugins, since that way people can opt-in to riskier features.

You found `datasette-media` which is my most comprehensive exploration of that idea so far - but there's definitely lots of room for more plugins along those lines.

Maybe even an output plugin? `.jpg` as an export format which returns the `BLOB` column for a row as a JPEG image with the correct `content-type` header (but first verifies that the binary content does indeed look like a real JPEG) could be interesting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-762391426,https://api.github.com/repos/simonw/datasette/issues/1036,762391426,MDEyOklzc3VlQ29tbWVudDc2MjM5MTQyNg==,4997607,philshem,2021-01-18T17:45:00Z,2021-01-18T17:45:00Z,NONE,"It might be possible with this library: https://docs.python.org/3/library/imghdr.html

quick test of the downloaded blob:

```
>>> import imghdr
>>> imghdr.what('material_culture-1-image.blob')
'jpeg'
```

The output plugin would be cool. I'll look into making my first datasette plugin.  I'm also imagining displaying the image in the browser -- but that would be a step 2.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1037#issuecomment-713267989,https://api.github.com/repos/simonw/datasette/issues/1037,713267989,MDEyOklzc3VlQ29tbWVudDcxMzI2Nzk4OQ==,9599,simonw,2020-10-21T03:14:34Z,2020-10-21T03:14:34Z,OWNER,"This is particularly relevant to the `datasette-cluster-map` plugin - the map is much nicer to use if the table itself can be scrolled.

That plugin also makes this harder to build, because the plugin inserts the map as the direct predecessor of the `<table>` element and hence breaks if you try to wrap that in a `<div>`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726094754,Add horizontal scrollbar to tables,
https://github.com/simonw/datasette/issues/1037#issuecomment-713268498,https://api.github.com/repos/simonw/datasette/issues/1037,713268498,MDEyOklzc3VlQ29tbWVudDcxMzI2ODQ5OA==,9599,simonw,2020-10-21T03:15:44Z,2020-10-21T03:15:44Z,OWNER,"This may require updates to the column action menu JavaScript too, since it was not built with scrolling sideways in mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726094754,Add horizontal scrollbar to tables,
https://github.com/simonw/datasette/issues/1037#issuecomment-713268905,https://api.github.com/repos/simonw/datasette/issues/1037,713268905,MDEyOklzc3VlQ29tbWVudDcxMzI2ODkwNQ==,9599,simonw,2020-10-21T03:16:36Z,2020-10-21T03:16:36Z,OWNER,Dupe of #998.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726094754,Add horizontal scrollbar to tables,
https://github.com/simonw/datasette/issues/1039#issuecomment-713754844,https://api.github.com/repos/simonw/datasette/issues/1039,713754844,MDEyOklzc3VlQ29tbWVudDcxMzc1NDg0NA==,9599,simonw,2020-10-21T17:58:27Z,2020-10-21T17:58:27Z,OWNER,"Now live: https://latest.datasette.io/fixtures/roadside_attraction_characteristics

![anim](https://user-images.githubusercontent.com/9599/96759016-55288480-138c-11eb-8ba0-d8e0f6dd8b1f.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726687572,Add an animation to the column actions menu,
https://github.com/simonw/datasette/issues/1041#issuecomment-714682288,https://api.github.com/repos/simonw/datasette/issues/1041,714682288,MDEyOklzc3VlQ29tbWVudDcxNDY4MjI4OA==,9599,simonw,2020-10-22T18:35:15Z,2020-10-22T18:35:15Z,OWNER,"@psychemedia said: https://github.com/simonw/datasette/issues/1033#issuecomment-714657366

> How does `/-/static` relate to [current guidance docs around `static`](https://docs.datasette.io/en/latest/custom_templates.html?highlight=static#serving-static-files) regarding the `--static option` and metadata formulations such as `""extra_js_urls"": [ ""/static/app.js""]` (I've not managed to get this to work in a Jupyter server proxied set up; the [datasette / jupyter server proxy repo](https://github.com/simonw/jupyterserverproxy-datasette-demo) may provide a useful test example, eg via MyBinder, for folk to crib from?)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting,
https://github.com/simonw/datasette/issues/1041#issuecomment-714682825,https://api.github.com/repos/simonw/datasette/issues/1041,714682825,MDEyOklzc3VlQ29tbWVudDcxNDY4MjgyNQ==,9599,simonw,2020-10-22T18:36:10Z,2020-10-22T18:36:10Z,OWNER,I'll need to update these docs once there's a solution for this in place: https://docs.datasette.io/en/latest/custom_templates.html#serving-static-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting,
https://github.com/simonw/datasette/issues/1041#issuecomment-714683801,https://api.github.com/repos/simonw/datasette/issues/1041,714683801,MDEyOklzc3VlQ29tbWVudDcxNDY4MzgwMQ==,9599,simonw,2020-10-22T18:37:47Z,2020-10-22T18:37:47Z,OWNER,"I think I'll do this by looking for URLs that start with `/` - since it's also possible to have full `https://...` URLs in that setting.
```json
{
    ""extra_css_urls"": [
        ""/static/styles.css""
    ],
    ""extra_js_urls"": [
        ""/static/app.js""
    ]
}
```
I need to think about the `extra_css_urls` and `extra_js_urls` plugin hooks too: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-css-urls-template-database-table-columns-view-name-request-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting,
https://github.com/simonw/datasette/issues/1041#issuecomment-719986800,https://api.github.com/repos/simonw/datasette/issues/1041,719986800,MDEyOklzc3VlQ29tbWVudDcxOTk4NjgwMA==,9599,simonw,2020-10-31T20:49:28Z,2020-10-31T20:49:28Z,OWNER,Implemented in a4ca26a2659d21779adf625183061d8879954c15,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting,
https://github.com/simonw/datasette/issues/1042#issuecomment-714868207,https://api.github.com/repos/simonw/datasette/issues/1042,714868207,MDEyOklzc3VlQ29tbWVudDcxNDg2ODIwNw==,9599,simonw,2020-10-23T02:29:12Z,2020-10-23T02:29:12Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/d0cc6f4c32e1f89238ddec782086b3122f445bd4/datasette/app.py#L288-L311,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-714868624,https://api.github.com/repos/simonw/datasette/issues/1042,714868624,MDEyOklzc3VlQ29tbWVudDcxNDg2ODYyNA==,9599,simonw,2020-10-23T02:30:27Z,2020-10-23T02:30:37Z,OWNER,Maybe `register_template_loader(datasette)` which returns an object which is added in at the beginning of the list passed to `ChoiceLoader` here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-714868867,https://api.github.com/repos/simonw/datasette/issues/1042,714868867,MDEyOklzc3VlQ29tbWVudDcxNDg2ODg2Nw==,9599,simonw,2020-10-23T02:31:17Z,2020-10-23T02:31:17Z,OWNER,I'll build this in conjunction with a plugin that supports editing templates stored in SQLite.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715490532,https://api.github.com/repos/simonw/datasette/issues/1042,715490532,MDEyOklzc3VlQ29tbWVudDcxNTQ5MDUzMg==,9599,simonw,2020-10-23T17:57:34Z,2020-10-23T17:57:34Z,OWNER,"A better version of this hook would be passed the database, table and query name depending on what was being rendered.

This would require some re-thinking of how core templates are loaded, especially since I would want the templates considered comment to continue working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715496859,https://api.github.com/repos/simonw/datasette/issues/1042,715496859,MDEyOklzc3VlQ29tbWVudDcxNTQ5Njg1OQ==,9599,simonw,2020-10-23T18:11:27Z,2020-10-23T18:11:27Z,OWNER,"When loading a template the filename is required, but you can optionally also send a set of extra arguments which the template loader can take into consideration.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715497419,https://api.github.com/repos/simonw/datasette/issues/1042,715497419,MDEyOklzc3VlQ29tbWVudDcxNTQ5NzQxOQ==,9599,simonw,2020-10-23T18:12:40Z,2020-10-23T18:12:40Z,OWNER,Maybe the template loader can optionally return some extra context to pass to the template. That could be used to solve the templates considered comment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715614971,https://api.github.com/repos/simonw/datasette/issues/1042,715614971,MDEyOklzc3VlQ29tbWVudDcxNTYxNDk3MQ==,9599,simonw,2020-10-23T22:20:14Z,2020-10-23T22:23:51Z,OWNER,"Alternative plugin hook idea:
```python
@hookspec
def load_template(template, database, table, columns, view_name, request, datasette):
    ""Load the specified template, returning the template code as a string""
```
Imitating the existing `extra_template_vars` family of hooks: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-template-vars-template-database-table-columns-view-name-request-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715616757,https://api.github.com/repos/simonw/datasette/issues/1042,715616757,MDEyOklzc3VlQ29tbWVudDcxNTYxNjc1Nw==,9599,simonw,2020-10-23T22:27:28Z,2020-10-23T22:27:28Z,OWNER,"Almost all of the core template loading happens in the `BaseView.render` method: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/views/base.py#L114-L133

The one exception is the 404 handling code here: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L1034-L1042","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715617405,https://api.github.com/repos/simonw/datasette/issues/1042,715617405,MDEyOklzc3VlQ29tbWVudDcxNTYxNzQwNQ==,9599,simonw,2020-10-23T22:29:53Z,2020-10-23T22:29:53Z,OWNER,"Also consider that `DatasetteRouter` uses `.list_templates()` to gather together `{slug}.html` style templates for the custom page templates mechanism: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L949-L967

For that to work with the new plugin hook, custom template providing plugins will need a way to provide a list of templates that they know about.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715617830,https://api.github.com/repos/simonw/datasette/issues/1042,715617830,MDEyOklzc3VlQ29tbWVudDcxNTYxNzgzMA==,9599,simonw,2020-10-23T22:31:26Z,2020-10-23T22:31:26Z,OWNER,"So maybe this should be a `register_template_loader` mechanism that returns a Jinja loader after all? That would mean that only the template filename could be used as the input to the plugin, which doesn't seem as useful as emulating the `extra_template_vars()` interface.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715618077,https://api.github.com/repos/simonw/datasette/issues/1042,715618077,MDEyOklzc3VlQ29tbWVudDcxNTYxODA3Nw==,9599,simonw,2020-10-23T22:32:24Z,2020-10-23T22:32:24Z,OWNER,"Another option: the first version of the plugin hook could accept only the template filename. Subsequent releases could add more arguments, since Pluggy allows new arguments to be added without breaking backwards compatibility.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715618333,https://api.github.com/repos/simonw/datasette/issues/1042,715618333,MDEyOklzc3VlQ29tbWVudDcxNTYxODMzMw==,9599,simonw,2020-10-23T22:33:24Z,2020-10-23T22:33:24Z,OWNER,"It wouldn't be a disaster if template-loading plugins were unable to hook into the `/{slug1}/{slug2}.html` custom page mechanism, since plugins can define their own pages already using `register_routes()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715643646,https://api.github.com/repos/simonw/datasette/issues/1042,715643646,MDEyOklzc3VlQ29tbWVudDcxNTY0MzY0Ng==,9599,simonw,2020-10-24T00:33:46Z,2020-10-24T00:33:46Z,OWNER,"I'd like to do this all in the `datasette.render_template()` method to ensure it's available to plugins as well, not just core code that uses the `BaseView` class.

This code is the problem:
https://github.com/simonw/datasette/blob/d3e9b0aecb6f8e9b2befd9c654ccb7ce852db3e7/datasette/views/base.py#L114-L133

I think I'll fix this by moving the `select_templates` mechanism into `datasette.render_templates()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715643763,https://api.github.com/repos/simonw/datasette/issues/1042,715643763,MDEyOklzc3VlQ29tbWVudDcxNTY0Mzc2Mw==,9599,simonw,2020-10-24T00:34:31Z,2020-10-24T00:34:52Z,OWNER,I'm going to rename that to template variable from `select_templates` to `templates_considered` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1045#issuecomment-715641183,https://api.github.com/repos/simonw/datasette/issues/1045,715641183,MDEyOklzc3VlQ29tbWVudDcxNTY0MTE4Mw==,9599,simonw,2020-10-24T00:19:29Z,2020-10-24T00:19:29Z,OWNER,"It turns out it already does that:

https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L710-L720

But the documentation doesn't reflect that:

> `template` - string
> 
> > The template file to be rendered, e.g. `my_plugin.html`. Datasette will search for this file first in the `--template-dir=` location, if it was specified - then in the plugin's bundled templates and finally in Datasette's set of default templates.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728600048,"Document that datasette.render_template(template, ...) also accepts a list of templates",
https://github.com/simonw/datasette/issues/1046#issuecomment-716066342,https://api.github.com/repos/simonw/datasette/issues/1046,716066342,MDEyOklzc3VlQ29tbWVudDcxNjA2NjM0Mg==,9599,simonw,2020-10-24T23:02:07Z,2020-10-24T23:02:25Z,OWNER,"A download icon would be nice for the links in the table display. I like this one https://primer.style/octicons/download-24

```svg
<svg xmlns=""http://www.w3.org/2000/svg"" viewBox=""0 0 24 24"" width=""24"" height=""24"">
  <path d=""M4.97 11.03a.75.75 0 111.06-1.06L11 14.94V2.75a.75.75 0 011.5 0v12.19l4.97-4.97a.75.75 0 111.06 1.06l-6.25 6.25a.75.75 0 01-1.06 0l-6.25-6.25zm-.22 9.47a.75.75 0 000 1.5h14.5a.75.75 0 000-1.5H4.75z""></path>
</svg>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728895193,Link to blob downloads in the right places,
https://github.com/simonw/datasette/issues/1046#issuecomment-716071507,https://api.github.com/repos/simonw/datasette/issues/1046,716071507,MDEyOklzc3VlQ29tbWVudDcxNjA3MTUwNw==,9599,simonw,2020-10-25T00:06:47Z,2020-10-25T00:06:47Z,OWNER,"I used https://primer.style/octicons/download-16 instead.

<img width=""734"" alt=""favicons__favicons__71_rows"" src=""https://user-images.githubusercontent.com/9599/97096021-44417280-161b-11eb-9bc4-88def6afaca4.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728895193,Link to blob downloads in the right places,
https://github.com/simonw/datasette/issues/1047#issuecomment-719994676,https://api.github.com/repos/simonw/datasette/issues/1047,719994676,MDEyOklzc3VlQ29tbWVudDcxOTk5NDY3Ng==,9599,simonw,2020-10-31T22:11:25Z,2020-10-31T22:11:25Z,OWNER,https://docs.datasette.io/en/latest/binary_data.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728895233,A new section in the docs about how Datasette handles BLOB columns,
https://github.com/simonw/datasette/issues/1048#issuecomment-1179756391,https://api.github.com/repos/simonw/datasette/issues/1048,1179756391,IC_kwDOBm6k_c5GUadn,9599,simonw,2022-07-10T16:12:23Z,2022-07-10T16:12:23Z,OWNER,"Also related: `row` is now available to `render_cell()` hook as-of this issue:
- https://github.com/simonw/datasette/issues/1300","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728905098,Documentation and unit tests for urls.row() urls.row_blob() methods,
https://github.com/simonw/datasette/issues/1048#issuecomment-1179756689,https://api.github.com/repos/simonw/datasette/issues/1048,1179756689,IC_kwDOBm6k_c5GUaiR,9599,simonw,2022-07-10T16:14:03Z,2022-07-10T16:14:03Z,OWNER,"I'm considering changing these functions to accept the `row` object itself:

```python
 def row(self, database, table, row): 
     ...
  
 def row_blob(self, database, table, row, column): 
     ... 
```
Just one catch: in order to generate the correct `row_path` we need to know the primary keys for that table. We can look those up based on having access to `database` and `table`, but doing so requires an `await ...` operation - and these functions are not `async`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728905098,Documentation and unit tests for urls.row() urls.row_blob() methods,
https://github.com/simonw/datasette/issues/1048#issuecomment-1179757067,https://api.github.com/repos/simonw/datasette/issues/1048,1179757067,IC_kwDOBm6k_c5GUaoL,9599,simonw,2022-07-10T16:16:19Z,2022-07-10T16:16:19Z,OWNER,"None of the potential solutions for that problem are particularly appealing:

- Make these URL generation methods `async` - should the other ones be `async` too for consistency?
- Have some kind of mechanism that calculates and caches the `pks` for each table somewhere which can then be accessed here without an `async` call
- Require `pks` is passed to these methods, having been looked up elsewhere. This is a bit gross but may end up being the best solution","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728905098,Documentation and unit tests for urls.row() urls.row_blob() methods,
https://github.com/simonw/datasette/issues/1048#issuecomment-1179757605,https://api.github.com/repos/simonw/datasette/issues/1048,1179757605,IC_kwDOBm6k_c5GUawl,9599,simonw,2022-07-10T16:19:42Z,2022-07-10T16:21:03Z,OWNER,"If I do require `pks` to be passed here, maybe I could make those available to the `render_cell()` plugin hook to at least make this a bit more pleasant for plugin authors to use?

Current hook: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-row-value-column-table-database-datasette

https://github.com/simonw/datasette/blob/035dc5e7b95142d4a700819a8cc4ff64aefe4efe/datasette/hookspecs.py#L62-L64

The hook is called in two places in the codebase - when rendering a table (`pks` variable is already in scope here):

https://github.com/simonw/datasette/blob/6373bb341457e5becfd5b67792ac2c8b9ed7c384/datasette/views/table.py#L897-L904

And when rendering an arbitrary query:

https://github.com/simonw/datasette/blob/6373bb341457e5becfd5b67792ac2c8b9ed7c384/datasette/views/database.py#L377-L384

Note that in that second one `table` is `None` (which is also called out in the documentation) - `pks` would be `None` here too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728905098,Documentation and unit tests for urls.row() urls.row_blob() methods,
https://github.com/simonw/datasette/issues/1048#issuecomment-1179757912,https://api.github.com/repos/simonw/datasette/issues/1048,1179757912,IC_kwDOBm6k_c5GUa1Y,9599,simonw,2022-07-10T16:21:38Z,2022-07-10T16:21:38Z,OWNER,"I think the best way to do this is to change the method signatures to:
```python
 def row(self, database, table, row, pks): 
     ...
  
 def row_blob(self, database, table, row, pks, column): 
     ... 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728905098,Documentation and unit tests for urls.row() urls.row_blob() methods,
https://github.com/simonw/datasette/issues/1048#issuecomment-1179758180,https://api.github.com/repos/simonw/datasette/issues/1048,1179758180,IC_kwDOBm6k_c5GUa5k,9599,simonw,2022-07-10T16:23:34Z,2022-07-10T16:23:57Z,OWNER,"But do I need to pass the `use_rowid` boolean here as well, as used by `def path_from_row_pks(row, pks, use_rowid, quote=True)`? Or can I derive that from the fact that `pks` is an empty tuple?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728905098,Documentation and unit tests for urls.row() urls.row_blob() methods,
https://github.com/simonw/datasette/issues/1048#issuecomment-719996693,https://api.github.com/repos/simonw/datasette/issues/1048,719996693,MDEyOklzc3VlQ29tbWVudDcxOTk5NjY5Mw==,9599,simonw,2020-10-31T22:32:09Z,2022-07-10T16:22:48Z,OWNER,"The `row_path` part of these really isn't very user friendly, since you need to properly URL-encode the values. The safest way to do so is by calling this obscure, undocumented utility function: https://github.com/simonw/datasette/blob/f0bd2d05f5f7832df4879822afb99d2096c00d48/datasette/utils/__init__.py#L84-L98

This feels like it should be improved before I turn it into a documented API.

(Note that this API deals with a `row` that is a potentially-nested dictionary - not with a `sqlite3.Row` object.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728905098,Documentation and unit tests for urls.row() urls.row_blob() methods,
https://github.com/simonw/datasette/issues/105#issuecomment-345493344,https://api.github.com/repos/simonw/datasette/issues/105,345493344,MDEyOklzc3VlQ29tbWVudDM0NTQ5MzM0NA==,9599,simonw,2017-11-19T05:28:49Z,2017-11-19T05:28:49Z,OWNER,Looks like there are a ton of interesting datasets packaged in this way at http://datahub.io/docs/core-data - see also https://github.com/datasets,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274314940,Consider data-package as a format for metadata,
https://github.com/simonw/datasette/issues/105#issuecomment-345494052,https://api.github.com/repos/simonw/datasette/issues/105,345494052,MDEyOklzc3VlQ29tbWVudDM0NTQ5NDA1Mg==,9599,simonw,2017-11-19T05:49:53Z,2017-11-19T05:49:53Z,OWNER,https://github.com/rgieseke/pandas-datapackage-reader,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274314940,Consider data-package as a format for metadata,
https://github.com/simonw/datasette/issues/105#issuecomment-345503897,https://api.github.com/repos/simonw/datasette/issues/105,345503897,MDEyOklzc3VlQ29tbWVudDM0NTUwMzg5Nw==,198537,rgieseke,2017-11-19T09:38:08Z,2017-11-19T09:38:08Z,CONTRIBUTOR,"Thanks, I wrote this very simple reader because the default approach as described on the Datahub pages seemed to complicated. I had metadata from the `datapackage.json` attached to the returned DataFrames but removed this due to some attribute handling change in the latest Pandas version.

This could also be useful for getting from Data Package to SQL db: https://github.com/frictionlessdata/tableschema-sql-py

I maintain a few climate science related dataset at https://github.com/openclimatedata/

The Data Retriever (mainly ecological data) by @ethanwhite et al. is also using the Data Package format for metadata and has some tooling for different dbs: 

https://frictionlessdata.io/articles/the-data-retriever/
https://github.com/weecology/retriever

The Open Power System Data project also has a couple of datasets that show nicely how CSV is great for assembling and then already make SQLite files available. It's one of the first data sets I tried with Datasette, perfect for the use case of getting an API for putting power stations on a map ...

https://data.open-power-system-data.org/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274314940,Consider data-package as a format for metadata,
https://github.com/simonw/datasette/issues/105#issuecomment-345809808,https://api.github.com/repos/simonw/datasette/issues/105,345809808,MDEyOklzc3VlQ29tbWVudDM0NTgwOTgwOA==,9599,simonw,2017-11-20T19:50:53Z,2017-11-20T19:50:53Z,OWNER,"OK, https://github.com/openclimatedata/global-carbon-budget/blob/master/datapackage.json really does look like it covers all of the bases I need for #138. Closing this ticket in favour of that new one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274314940,Consider data-package as a format for metadata,
https://github.com/simonw/datasette/issues/1050#issuecomment-716174203,https://api.github.com/repos/simonw/datasette/issues/1050,716174203,MDEyOklzc3VlQ29tbWVudDcxNjE3NDIwMw==,9599,simonw,2020-10-25T16:27:39Z,2020-10-25T16:53:27Z,OWNER,"Idea: `.blob` output rendererer, where you tell it which column you want using `?_blob_column=x`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-716175236,https://api.github.com/repos/simonw/datasette/issues/1050,716175236,MDEyOklzc3VlQ29tbWVudDcxNjE3NTIzNg==,9599,simonw,2020-10-25T16:35:20Z,2020-10-25T16:35:20Z,OWNER,"This is clearly a better solution than the one I  implemented in #1040 - I don't have to add a new route, I don't have to implement permission checks, it reuses mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718317997,https://api.github.com/repos/simonw/datasette/issues/1050,718317997,MDEyOklzc3VlQ29tbWVudDcxODMxNzk5Nw==,283343,thadk,2020-10-29T02:24:50Z,2020-10-29T02:29:24Z,NONE,"Unsolicited feedback for an unreleased feature of the [current](https://github.com/simonw/datasette/commit/5e0b72247ecab4ce0fcec599b77a83d73a480872) unreleased GitHub version (I casually wanted to access a blob row) – the existing #1036 route doesn't support special characters in database or table names (e.g. `@()` ). Maybe this is motivation for your new idea here.

Also I got this error/crash with my blob and wasn't able to get the file: https://gist.github.com/thadk/28ac32af0e88747ce9056c90b0b19d34","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718342036,https://api.github.com/repos/simonw/datasette/issues/1050,718342036,MDEyOklzc3VlQ29tbWVudDcxODM0MjAzNg==,9599,simonw,2020-10-29T03:49:57Z,2020-10-29T03:49:57Z,OWNER,"@thadk from that error it looks like the problem may have been that you had a BLOB column containing a `null` value? If so that's definitely a bug, I'll fix that.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718346019,https://api.github.com/repos/simonw/datasette/issues/1050,718346019,MDEyOklzc3VlQ29tbWVudDcxODM0NjAxOQ==,9599,simonw,2020-10-29T04:05:07Z,2020-10-29T04:05:07Z,OWNER,"Yes, confirmed - this is a bug where if the `BLOB` column contains a `null` you get a nasty exception if you try to download it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718980944,https://api.github.com/repos/simonw/datasette/issues/1050,718980944,MDEyOklzc3VlQ29tbWVudDcxODk4MDk0NA==,9599,simonw,2020-10-29T19:46:19Z,2020-10-29T19:46:19Z,OWNER,"Had an idea in https://github.com/simonw/datasette/issues/1051#issuecomment-718980659

> OK, alternative idea. The `.blob` output renderer from #1050 gets to see multiple rows at once.
> 
> For an arbitrary SQL query, how about if I link to this?
> 
> `/db.blob?sql=...&_blob_column=data&_blob_hash=bc4c24181ed3ce666`
> 
> Then the output renderer loops through all of the `data` results that are available to it and, if one of them hashes to that value, serves up that data?
> 
> If no matches are found it can show an error message telling you that the link has expired (presumably because the underlying database has changed since the link was generated).
> 
> I think this might be the best solution to the problem.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718987852,https://api.github.com/repos/simonw/datasette/issues/1050,718987852,MDEyOklzc3VlQ29tbWVudDcxODk4Nzg1Mg==,9599,simonw,2020-10-29T20:00:32Z,2020-10-29T20:00:32Z,OWNER,"The reason I like the `?_blob_hash=` solution is that it feels really misleading to provide a link to ""download this binary"" which could conceivably download some other data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718989895,https://api.github.com/repos/simonw/datasette/issues/1050,718989895,MDEyOklzc3VlQ29tbWVudDcxODk4OTg5NQ==,9599,simonw,2020-10-29T20:04:15Z,2020-10-29T20:04:15Z,OWNER,I'll use `hashlib.sha256` for these hashes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-719001701,https://api.github.com/repos/simonw/datasette/issues/1050,719001701,MDEyOklzc3VlQ29tbWVudDcxOTAwMTcwMQ==,9599,simonw,2020-10-29T20:26:44Z,2020-10-29T20:26:44Z,OWNER,I'll do the rest of the work on this in the pull request #1061.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-719021514,https://api.github.com/repos/simonw/datasette/issues/1050,719021514,MDEyOklzc3VlQ29tbWVudDcxOTAyMTUxNA==,9599,simonw,2020-10-29T21:05:08Z,2020-10-29T21:05:08Z,OWNER,"Idea: what if Datasette had a custom SQLite function that could be used to generate URLs to the row-level BLOB download for a value? Then custom SQL query authors could use that function to link to the relevant content.

This could be expanded to exposing other `datasette.urls` functionality as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1051#issuecomment-716204090,https://api.github.com/repos/simonw/datasette/issues/1051,716204090,MDEyOklzc3VlQ29tbWVudDcxNjIwNDA5MA==,9599,simonw,2020-10-25T20:06:42Z,2020-10-25T20:06:42Z,OWNER,"Providing a binary download link here is actually extremely difficult.

The problem is that the SQL query itself represents data that can change from one moment to the next. It's no good showing a ""Binary: 55 bytes"" message that links to that same SQL query but with a `.blob` extension and arguments to select the particular result, because the data may change in a way that causes that query to return a different row - at which point the download link will give you the wrong data, not the 55 bytes you asked for.

So providing a download link risks being misleading.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1051#issuecomment-716204271,https://api.github.com/repos/simonw/datasette/issues/1051,716204271,MDEyOklzc3VlQ29tbWVudDcxNjIwNDI3MQ==,9599,simonw,2020-10-25T20:08:04Z,2020-10-25T20:08:04Z,OWNER,"This is bad though, because if I want to provide binary data in CSV as requested in #1034 I need some way of providing that data.

Which suggests to me that the base64 option is the only one that can make sense for arbitrary SQL queries represented as CSV. Download links won't work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1051#issuecomment-716681167,https://api.github.com/repos/simonw/datasette/issues/1051,716681167,MDEyOklzc3VlQ29tbWVudDcxNjY4MTE2Nw==,9599,simonw,2020-10-26T16:51:15Z,2020-10-26T16:51:15Z,OWNER,"Crazy idea: generate a signed URL containing a base64 of the gzip of the binary content (to try and reduce size).

No: this will blow through URL limits in various hosting providers and possibly even browsers. It could be made to work a little bit more reliably with some extra JavaScript that turns it into a download on the browser-side, but that would be hideously complicated.

Also the signed bit doesn't prevent people from generating SQL queries that generate nasty binary blobs for download.

I'm beginning to think that restricting this feature to just table view, not query view, is a better idea. Query view can still get at the binary using JSON and base64.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1051#issuecomment-716681602,https://api.github.com/repos/simonw/datasette/issues/1051,716681602,MDEyOklzc3VlQ29tbWVudDcxNjY4MTYwMg==,9599,simonw,2020-10-26T16:51:58Z,2020-10-26T16:51:58Z,OWNER,"I still need to improve the current binary display on the query page though, where it outputs a Python `b'...'` literal.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1051#issuecomment-718980659,https://api.github.com/repos/simonw/datasette/issues/1051,718980659,MDEyOklzc3VlQ29tbWVudDcxODk4MDY1OQ==,9599,simonw,2020-10-29T19:45:42Z,2020-10-29T19:45:42Z,OWNER,"OK, alternative idea. The `.blob` output renderer from #1050 gets to see multiple rows at once. 

For an arbitrary SQL query, how about if I link to this?

`/db.blob?sql=...&_blob_column=data&_blob_hash=bc4c24181ed3ce666`

Then the output renderer loops through all of the `data` results that are available to it and, if one of them hashes to that value, serves up that data?

If no matches are found it can show an error message telling you that the link has expired (presumably because the underlying database has changed since the link was generated).

I think this might be the best solution to the problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1051#issuecomment-719053669,https://api.github.com/repos/simonw/datasette/issues/1051,719053669,MDEyOklzc3VlQ29tbWVudDcxOTA1MzY2OQ==,9599,simonw,2020-10-29T22:12:16Z,2020-10-29T22:12:16Z,OWNER,"https://latest.datasette.io/fixtures?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+101 now looks like this:

<img width=""816"" alt=""fixtures__select_rowid__data_from_binary_data_order_by_rowid_limit_101_and_understanding_Google_auths_-_for_your_information"" src=""https://user-images.githubusercontent.com/9599/97638086-15e7dc80-19f9-11eb-90a9-686e012c4db9.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1052#issuecomment-716265360,https://api.github.com/repos/simonw/datasette/issues/1052,716265360,MDEyOklzc3VlQ29tbWVudDcxNjI2NTM2MA==,9599,simonw,2020-10-26T02:17:58Z,2020-10-26T02:17:58Z,OWNER,"The default z-index values for Leaflet are defined here: https://github.com/Leaflet/Leaflet/blob/b346bb8bf7bb80899baa1f4fc1536bae58e7e3e6/dist/leaflet.css#L81-L91

```css
.leaflet-pane         { z-index: 400; }

.leaflet-tile-pane    { z-index: 200; }
.leaflet-overlay-pane { z-index: 400; }
.leaflet-shadow-pane  { z-index: 500; }
.leaflet-marker-pane  { z-index: 600; }
.leaflet-tooltip-pane   { z-index: 650; }
.leaflet-popup-pane   { z-index: 700; }

.leaflet-map-pane canvas { z-index: 100; }
.leaflet-map-pane svg    { z-index: 200; }
```
So a `z-index` of 1000 on the menu should fix this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729183332,Column action menu overlapped by Leaflet maps,
https://github.com/simonw/datasette/issues/1053#issuecomment-718976679,https://api.github.com/repos/simonw/datasette/issues/1053,718976679,MDEyOklzc3VlQ29tbWVudDcxODk3NjY3OQ==,9599,simonw,2020-10-29T19:37:57Z,2020-10-29T19:37:57Z,OWNER,https://docs.datasette.io/en/latest/writing_plugins.html#designing-urls-for-your-plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729604838,Document recommendations for plugin authors to design URLs,
https://github.com/simonw/datasette/issues/1054#issuecomment-717050585,https://api.github.com/repos/simonw/datasette/issues/1054,717050585,MDEyOklzc3VlQ29tbWVudDcxNzA1MDU4NQ==,9599,simonw,2020-10-27T07:38:50Z,2020-10-27T07:38:50Z,OWNER,"Maybe imitate how Django does this, e.g. https://github.com/django/django/commit/6b9b2af7352908d40ca4d31bdb1b80c013cab29a","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730199464,Switch from versioneer to concrete version in setup.py,
https://github.com/simonw/datasette/issues/1054#issuecomment-717051707,https://api.github.com/repos/simonw/datasette/issues/1054,717051707,MDEyOklzc3VlQ29tbWVudDcxNzA1MTcwNw==,9599,simonw,2020-10-27T07:41:21Z,2020-10-27T07:41:21Z,OWNER,Essentially it's this problem: https://github.com/python-versioneer/python-versioneer/issues/140,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730199464,Switch from versioneer to concrete version in setup.py,
https://github.com/simonw/datasette/issues/1057#issuecomment-717531272,https://api.github.com/repos/simonw/datasette/issues/1057,717531272,MDEyOklzc3VlQ29tbWVudDcxNzUzMTI3Mg==,9599,simonw,2020-10-27T20:51:09Z,2020-10-27T20:51:09Z,OWNER,"That works!

<img width=""1370"" alt=""Banners_and_Alerts_and_SQLite___Mike_Bostock___Observable"" src=""https://user-images.githubusercontent.com/9599/97360439-717c6380-185b-11eb-9996-e0064e4b22d4.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730797787,--cors should enable /fixtures.db CORS access,
https://github.com/simonw/datasette/issues/1058#issuecomment-717527606,https://api.github.com/repos/simonw/datasette/issues/1058,717527606,MDEyOklzc3VlQ29tbWVudDcxNzUyNzYwNg==,9599,simonw,2020-10-27T20:44:06Z,2020-10-27T20:44:06Z,OWNER,Example: https://github.com/simonw/datasette/blob/5a1519796037105bc20bcf2f91a76e022926c204/datasette/views/database.py#L26-L32,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730802994,Database download should implement cascading permissions,
https://github.com/simonw/datasette/issues/106#issuecomment-504879510,https://api.github.com/repos/simonw/datasette/issues/106,504879510,MDEyOklzc3VlQ29tbWVudDUwNDg3OTUxMA==,9599,simonw,2019-06-24T06:42:33Z,2019-06-24T06:42:33Z,OWNER,https://datasette.readthedocs.io/en/stable/sql_queries.html?highlight=Pagination#pagination,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274315193,Document how pagination works,
https://github.com/simonw/datasette/issues/1062#issuecomment-1068327874,https://api.github.com/repos/simonw/datasette/issues/1062,1068327874,IC_kwDOBm6k_c4_rWPC,9599,simonw,2022-03-15T18:33:49Z,2022-03-15T18:33:49Z,OWNER,"I can get regular `.json` to stream too, using the pattern described in this TIL: https://til.simonwillison.net/python/output-json-array-streaming","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,
https://github.com/simonw/datasette/issues/1062#issuecomment-1260829829,https://api.github.com/repos/simonw/datasette/issues/1062,1260829829,IC_kwDOBm6k_c5LJryF,536941,fgregg,2022-09-28T12:27:19Z,2022-09-28T12:27:19Z,CONTRIBUTOR,"for teaching `register_output_renderer` to stream it seems like the two options are to

1. a [nested query technique ](https://github.com/simonw/datasette/issues/526#issuecomment-505162238)to paginate through
2. a fetching model that looks like something
```python
with sqlite_timelimit(conn, time_limit_ms):
     c.execute(query)
     for chunk in c.fetchmany(chunk_size):
         yield from chunk
```
currently `db.execute` is not a generator, so this would probably need a new method?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,
https://github.com/simonw/datasette/issues/1062#issuecomment-1260909128,https://api.github.com/repos/simonw/datasette/issues/1062,1260909128,IC_kwDOBm6k_c5LJ_JI,536941,fgregg,2022-09-28T13:22:53Z,2022-09-28T14:09:54Z,CONTRIBUTOR,"if you went this route:

```python
with sqlite_timelimit(conn, time_limit_ms):
     c.execute(query)
     for chunk in c.fetchmany(chunk_size):
         yield from chunk
```

then `time_limit_ms` would probably have to be greatly extended, because the time spent in the loop will depend on the downstream processing.

i wonder if this was why you were thinking this feature would need a dedicated connection?

---

reading more, there's no real limit i can find on the number of active cursors (or more precisely active prepared statements objects, because sqlite doesn't really have cursors). 

maybe something like this would be okay?

```python
with sqlite_timelimit(conn, time_limit_ms):
     c.execute(query)
     # step through at least one to evaluate the statement, not sure if this is necessary
     yield c.execute.fetchone()
for chunk in c.fetchmany(chunk_size):
    yield from chunk
```

this seems quite weird that there's not more of limit of the number of active prepared statements, but i haven't been able to find one.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,
https://github.com/simonw/datasette/issues/1062#issuecomment-719031901,https://api.github.com/repos/simonw/datasette/issues/1062,719031901,MDEyOklzc3VlQ29tbWVudDcxOTAzMTkwMQ==,9599,simonw,2020-10-29T21:25:54Z,2020-10-29T21:25:54Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/views/base.py#L258-L345,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,
https://github.com/simonw/datasette/issues/1062#issuecomment-853566011,https://api.github.com/repos/simonw/datasette/issues/1062,853566011,MDEyOklzc3VlQ29tbWVudDg1MzU2NjAxMQ==,9599,simonw,2021-06-03T05:07:42Z,2021-06-03T05:07:42Z,OWNER,Implementing this would make #1356 a whole lot more interesting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,
https://github.com/simonw/datasette/issues/1063#issuecomment-719043108,https://api.github.com/repos/simonw/datasette/issues/1063,719043108,MDEyOklzc3VlQ29tbWVudDcxOTA0MzEwOA==,9599,simonw,2020-10-29T21:46:48Z,2020-10-29T21:46:48Z,OWNER,Remove this `xfail` and `import pytest`: https://github.com/simonw/datasette/blob/503a5b7b4080a26ef9ceb1ecd1a4a6f4ef4ffc59/tests/test_csv.py#L83-L96,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732685643,.csv should link to .blob downloads,
https://github.com/simonw/datasette/issues/1063#issuecomment-719050390,https://api.github.com/repos/simonw/datasette/issues/1063,719050390,MDEyOklzc3VlQ29tbWVudDcxOTA1MDM5MA==,9599,simonw,2020-10-29T22:04:00Z,2020-10-29T22:04:00Z,OWNER,This will close #1034.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732685643,.csv should link to .blob downloads,
https://github.com/simonw/datasette/issues/1063#issuecomment-719066706,https://api.github.com/repos/simonw/datasette/issues/1063,719066706,MDEyOklzc3VlQ29tbWVudDcxOTA2NjcwNg==,9599,simonw,2020-10-29T22:46:28Z,2020-10-29T22:46:28Z,OWNER,I'm not going to do the base64 thing unless someone asks for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732685643,.csv should link to .blob downloads,
https://github.com/simonw/datasette/issues/1064#issuecomment-719104883,https://api.github.com/repos/simonw/datasette/issues/1064,719104883,MDEyOklzc3VlQ29tbWVudDcxOTEwNDg4Mw==,9599,simonw,2020-10-30T00:50:01Z,2020-10-30T00:52:29Z,OWNER,"Here's what the prototype looks like so far:

![menu](https://user-images.githubusercontent.com/9599/97647443-7eda4f00-1a0f-11eb-8d78-b703b7a13616.gif)

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719105197,https://api.github.com/repos/simonw/datasette/issues/1064,719105197,MDEyOklzc3VlQ29tbWVudDcxOTEwNTE5Nw==,9599,simonw,2020-10-30T00:51:16Z,2020-10-30T00:51:16Z,OWNER,"I used a `<details><summary>` for this: https://github.com/simonw/datasette/blob/0d7ac764861d84be24d661cf4104ce61ea11a82a/datasette/templates/base.html#L16-L36

I added a bit of JavaScript so that clicking outside the menu would close it: https://github.com/simonw/datasette/blob/0d7ac764861d84be24d661cf4104ce61ea11a82a/datasette/templates/base.html#L59-L74","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719105641,https://api.github.com/repos/simonw/datasette/issues/1064,719105641,MDEyOklzc3VlQ29tbWVudDcxOTEwNTY0MQ==,9599,simonw,2020-10-30T00:53:00Z,2020-10-30T00:53:00Z,OWNER,Tips for making this accessible: https://css-tricks.com/accessible-svgs/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719106174,https://api.github.com/repos/simonw/datasette/issues/1064,719106174,MDEyOklzc3VlQ29tbWVudDcxOTEwNjE3NA==,9599,simonw,2020-10-30T00:55:12Z,2020-10-30T00:55:12Z,OWNER,"So what should go in this menu?

If the user is logged in as root, I'll link to the various debug pages.

If they're not logged in at all I don't think the menu should appear.

If they are logged in as anyone, it should display to give them access to the ""log out"" button.

Plugins can add links to it. If those plugins add links, the menu will display.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719109770,https://api.github.com/repos/simonw/datasette/issues/1064,719109770,MDEyOklzc3VlQ29tbWVudDcxOTEwOTc3MA==,9599,simonw,2020-10-30T01:08:14Z,2020-10-30T01:08:14Z,OWNER,"How should the plugin hook work?

Here's the first version of the HTML:
```html
        <div class=""nav-menu-inner"">
            <ul>
                <li><a href=""{{ urls.instance() }}"">Home</a></li>
                <li><a href=""{{ urls.path('/-/plugins') }}"">Installed plugins</a></li>
                <li><a href=""{{ urls.path('/-/versions') }}"">Software versions</a></li>
                <li><a href=""{{ urls.path('/-/metadata') }}"">Metadata</a></li>
                {% if show_logout %}
                <form action=""{{ urls.logout() }}"" method=""post"">
                    <input type=""hidden"" name=""csrftoken"" value=""{{ csrftoken() }}"">
                    <button class=""button-as-link"">Log out</button>
                </form>{% endif %}
            </ul>
        </div>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719110582,https://api.github.com/repos/simonw/datasette/issues/1064,719110582,MDEyOklzc3VlQ29tbWVudDcxOTExMDU4Mg==,9599,simonw,2020-10-30T01:11:13Z,2020-10-30T01:11:13Z,OWNER,"Should plugins be able to add forms like the logout form here, or should they be restricted to adding navigation links?

I can't think of a reason a plugin would need to add a form. The logout form is a special case to protect against logout-csrf attacks.

So I think plugins get to return a list of dictionaries, each with a `label` and an `href`:

```python
return [{
  ""label"": ""Upload CSVs"",
  ""href"": datasette.urls.path(""/-/upload-csvs"")
}]
```
But... is there an argument for returning headings, to divide up the menu?

I think so. I also like the idea that a default plugin checks for the `root` user and outputs links to the different debugging tools - maybe those should be wrapped in a section heading.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719110808,https://api.github.com/repos/simonw/datasette/issues/1064,719110808,MDEyOklzc3VlQ29tbWVudDcxOTExMDgwOA==,9599,simonw,2020-10-30T01:12:09Z,2020-10-30T01:12:19Z,OWNER,Or... plugins could return HTML - maybe optionally using helper functions to generate common HTML such that plugins which use the helpers can have their HTML modified in the future.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719111373,https://api.github.com/repos/simonw/datasette/issues/1064,719111373,MDEyOklzc3VlQ29tbWVudDcxOTExMTM3Mw==,9599,simonw,2020-10-30T01:14:13Z,2020-10-30T01:14:13Z,OWNER,"Plugins returning HTML makes more sense for some of the other areas that plugins will be able to inject content - e.g. injecting content on the table or row page above the table.

If I'm going to have that as a pattern though it may make sense to use HTML here, since that will be consistent with other places that plugins can inject additional content.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719111597,https://api.github.com/repos/simonw/datasette/issues/1064,719111597,MDEyOklzc3VlQ29tbWVudDcxOTExMTU5Nw==,9599,simonw,2020-10-30T01:15:05Z,2020-10-30T01:15:05Z,OWNER,I'm torn on this one. I think I have a very slight preference for plugins returning structured objects as opposed to HTML. Less likely to regret that choice in the future?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719117185,https://api.github.com/repos/simonw/datasette/issues/1064,719117185,MDEyOklzc3VlQ29tbWVudDcxOTExNzE4NQ==,9599,simonw,2020-10-30T01:35:17Z,2020-10-30T01:35:17Z,OWNER,"I'm going to go with a list of `{""label"": ..., ""href"": ...}` as the first iteration of this. The logout link will not be returned as part of the plugin output. A default plugin will provide the debug tools if the user is logged in as root.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1066#issuecomment-719154646,https://api.github.com/repos/simonw/datasette/issues/1066,719154646,MDEyOklzc3VlQ29tbWVudDcxOTE1NDY0Ng==,9599,simonw,2020-10-30T03:48:15Z,2020-10-30T03:48:15Z,OWNER,This will use a very similar implementation to the navigation menu in #1064 - similar plugin hook and I'll use a `<details><summary>` to implement it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732859030,Table actions menu plus plugin hook,
https://github.com/simonw/datasette/issues/1066#issuecomment-719194619,https://api.github.com/repos/simonw/datasette/issues/1066,719194619,MDEyOklzc3VlQ29tbWVudDcxOTE5NDYxOQ==,9599,simonw,2020-10-30T05:18:04Z,2020-10-30T05:18:04Z,OWNER,"The cog only appears if at least one table action has been registered by a plugin. It looks like this:

![table-actions](https://user-images.githubusercontent.com/9599/97662535-9bd54900-1a34-11eb-8e0f-56d159c8834e.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732859030,Table actions menu plus plugin hook,
https://github.com/simonw/datasette/issues/1066#issuecomment-719194756,https://api.github.com/repos/simonw/datasette/issues/1066,719194756,MDEyOklzc3VlQ29tbWVudDcxOTE5NDc1Ng==,9599,simonw,2020-10-30T05:18:35Z,2020-10-30T05:18:35Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#table-actions-datasette-actor-database-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732859030,Table actions menu plus plugin hook,
https://github.com/simonw/datasette/issues/1067#issuecomment-719320948,https://api.github.com/repos/simonw/datasette/issues/1067,719320948,MDEyOklzc3VlQ29tbWVudDcxOTMyMDk0OA==,9599,simonw,2020-10-30T07:02:37Z,2020-10-30T07:02:37Z,OWNER,"Yes, this should be possible - no point restricting what plugin authors can do with the feature. Will need to add some extra arguments to the plugin hook for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1067#issuecomment-719322666,https://api.github.com/repos/simonw/datasette/issues/1067,719322666,MDEyOklzc3VlQ29tbWVudDcxOTMyMjY2Ng==,9599,simonw,2020-10-30T07:04:02Z,2020-10-30T07:04:02Z,OWNER,"Maybe rename it to `actions_menu` and have it work for database, view, table and query pages using different arguments on each.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1067#issuecomment-719956184,https://api.github.com/repos/simonw/datasette/issues/1067,719956184,MDEyOklzc3VlQ29tbWVudDcxOTk1NjE4NA==,9599,simonw,2020-10-31T16:26:09Z,2020-10-31T16:26:09Z,OWNER,"Should the hook provide an indication that it's running on a different type of page? I think yes for queries. Not sure about views  - they behave very much like tables, and the plugin can always introspect to see if something is a view if it needs to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1067#issuecomment-719959419,https://api.github.com/repos/simonw/datasette/issues/1067,719959419,MDEyOklzc3VlQ29tbWVudDcxOTk1OTQxOQ==,9599,simonw,2020-10-31T16:53:42Z,2020-10-31T16:53:42Z,OWNER,For the 0.51 release I'm going to add tests that show this works on view pages. I won't implement it for query pages.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1067#issuecomment-719961701,https://api.github.com/repos/simonw/datasette/issues/1067,719961701,MDEyOklzc3VlQ29tbWVudDcxOTk2MTcwMQ==,9599,simonw,2020-10-31T17:11:59Z,2020-10-31T17:11:59Z,OWNER,It bothers me that these aren't visible in any public demos. Maybe `latest.datasette.io` should include the `my_plugins.py` and `my_plugins2.py` plugins?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1067#issuecomment-719966176,https://api.github.com/repos/simonw/datasette/issues/1067,719966176,MDEyOklzc3VlQ29tbWVudDcxOTk2NjE3Ng==,9599,simonw,2020-10-31T17:51:31Z,2020-10-31T17:51:31Z,OWNER,"Demo:

- https://latest.datasette.io/fixtures/facetable?_bot=1
- https://latest.datasette.io/fixtures/simple_view?_bot=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1068#issuecomment-719328661,https://api.github.com/repos/simonw/datasette/issues/1068,719328661,MDEyOklzc3VlQ29tbWVudDcxOTMyODY2MQ==,9599,simonw,2020-10-30T07:09:30Z,2020-10-30T07:09:30Z,OWNER,Then this can make it available to root: https://github.com/simonw/datasette/blob/18a64fbb29271ce607937110bbdb55488c43f4e0/datasette/default_permissions.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1068#issuecomment-719329219,https://api.github.com/repos/simonw/datasette/issues/1068,719329219,MDEyOklzc3VlQ29tbWVudDcxOTMyOTIxOQ==,9599,simonw,2020-10-30T07:09:59Z,2020-10-30T07:09:59Z,OWNER,Permission idea: `debug-menu`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1068#issuecomment-719331236,https://api.github.com/repos/simonw/datasette/issues/1068,719331236,MDEyOklzc3VlQ29tbWVudDcxOTMzMTIzNg==,9599,simonw,2020-10-30T07:11:58Z,2020-10-30T07:11:58Z,OWNER,Document the new permission here: https://docs.datasette.io/en/stable/authentication.html#built-in-permissions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1068#issuecomment-719332460,https://api.github.com/repos/simonw/datasette/issues/1068,719332460,MDEyOklzc3VlQ29tbWVudDcxOTMzMjQ2MA==,9599,simonw,2020-10-30T07:13:10Z,2020-10-30T07:13:10Z,OWNER,I mainly want this so I can add that debug menu to my Dogsheep.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1068#issuecomment-719630745,https://api.github.com/repos/simonw/datasette/issues/1068,719630745,MDEyOklzc3VlQ29tbWVudDcxOTYzMDc0NQ==,9599,simonw,2020-10-30T15:44:13Z,2020-10-30T15:44:13Z,OWNER,Documentation: https://docs.datasette.io/en/latest/authentication.html#debug-menu,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1070#issuecomment-719955724,https://api.github.com/repos/simonw/datasette/issues/1070,719955724,MDEyOklzc3VlQ29tbWVudDcxOTk1NTcyNA==,9599,simonw,2020-10-31T16:22:45Z,2020-10-31T16:22:45Z,OWNER,I've removed this plugin hook in #1073.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733390884,load_template() example in documentation showing loading from a database,
https://github.com/simonw/datasette/issues/1071#issuecomment-719777499,https://api.github.com/repos/simonw/datasette/issues/1071,719777499,MDEyOklzc3VlQ29tbWVudDcxOTc3NzQ5OQ==,9599,simonw,2020-10-30T20:20:01Z,2020-10-30T20:20:01Z,OWNER,"Fixed: https://latest.datasette.io/-/messages

![demo-message](https://user-images.githubusercontent.com/9599/97753199-9c142980-1ab2-11eb-9be9-c0be41acc68e.gif)


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733485423,Messages should be displayed full width,
https://github.com/simonw/datasette/issues/1072#issuecomment-719784606,https://api.github.com/repos/simonw/datasette/issues/1072,719784606,MDEyOklzc3VlQ29tbWVudDcxOTc4NDYwNg==,9599,simonw,2020-10-30T20:35:33Z,2020-10-30T20:35:33Z,OWNER,"To fix this I think I need to move the `load_template` implementation into a Jinja template loader.

I'm not sure I'll be able to keep the `Templates considered` comment working though:

https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L745-L750","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719785005,https://api.github.com/repos/simonw/datasette/issues/1072,719785005,MDEyOklzc3VlQ29tbWVudDcxOTc4NTAwNQ==,9599,simonw,2020-10-30T20:36:22Z,2020-10-30T20:36:22Z,OWNER,"It should be easy enough to show a comment that says which original template names were considered, but I may not be able to show which one was actually used (or which ones came from plugins).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719803880,https://api.github.com/repos/simonw/datasette/issues/1072,719803880,MDEyOklzc3VlQ29tbWVudDcxOTgwMzg4MA==,9599,simonw,2020-10-30T21:17:11Z,2020-10-30T21:17:11Z,OWNER,"Example from the Jinja docs: https://jinja.palletsprojects.com/en/2.11.x/api/#jinja2.BaseLoader

```python
from jinja2 import BaseLoader, TemplateNotFound
from os.path import join, exists, getmtime

class MyLoader(BaseLoader):

    def __init__(self, path):
        self.path = path

    def get_source(self, environment, template):
        path = join(self.path, template)
        if not exists(path):
            raise TemplateNotFound(template)
        mtime = getmtime(path)
        with file(path) as f:
            source = f.read().decode('utf-8')
        return source, path, lambda: mtime == getmtime(path)
```
Also available: `jinja2.FunctionLoader(load_func)` which lets me pass it a function like this one:
```
>>> def load_template(name):
...     if name == 'index.html':
...         return '...'
...
>>> loader = FunctionLoader(load_template)
```

Just one catch: I need to be able to load templates asynchronously, because they live in the database. Let's hope Jinja has a mechanism for that!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719807502,https://api.github.com/repos/simonw/datasette/issues/1072,719807502,MDEyOklzc3VlQ29tbWVudDcxOTgwNzUwMg==,9599,simonw,2020-10-30T21:26:49Z,2020-10-30T21:26:49Z,OWNER,"It looks like Jinja does not have a mechanism for asynchronous template loading - the loader API is synchronous.

One option may be to figure out which templates are needed (including inherited templates and includes) before rendering the template. Then async load those templates from the database into a `DictLoader`, then pass that `DictLoader` to Jinja.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719809259,https://api.github.com/repos/simonw/datasette/issues/1072,719809259,MDEyOklzc3VlQ29tbWVudDcxOTgwOTI1OQ==,9599,simonw,2020-10-30T21:31:10Z,2020-10-30T21:31:10Z,OWNER,"How can we tell what template Jinja will need to render?

One approach that could work:

1. Set up a dummy template loader which records the name of the template that was requested
2. Load the template
3. Now we know the list of templates that were requested. Async load those
4. The dummy template loader can now return the ones we have loaded. Load the template again.
5. Did it request any more templates? If so, load those, and repeat.
6. Keep on with this loop until a template load (which might even have to be a render) fails to request any templates that we have not yet loaded.
7. Render the template.

This is GROSS. It feels like a huge waste of CPU, and it could lead to very weird behaviour if any template variables have side effects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719809780,https://api.github.com/repos/simonw/datasette/issues/1072,719809780,MDEyOklzc3VlQ29tbWVudDcxOTgwOTc4MA==,9599,simonw,2020-10-30T21:32:28Z,2020-10-30T21:32:28Z,OWNER,"Here's an alternative that would definitely work and would be a lot simpler, at the cost of a fair amount of RAM:

1. Before rendering the template, load ALL of the most-recent-versions of the templates that are stored in the DB. Use those to populate a `DictLoader`.
2. Render the template.

This does mean loading template bodies that we won't use. Provided an instance has less than 100 templates I imagine this will work just fine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719810023,https://api.github.com/repos/simonw/datasette/issues/1072,719810023,MDEyOklzc3VlQ29tbWVudDcxOTgxMDAyMw==,9599,simonw,2020-10-30T21:33:06Z,2020-10-30T21:33:06Z,OWNER,"The ideal solution is for Jinja to offer `async` template loading. I'll file a feature request, then I'll implement the second option above (async load all templates from the DB before each render).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719810533,https://api.github.com/repos/simonw/datasette/issues/1072,719810533,MDEyOklzc3VlQ29tbWVudDcxOTgxMDUzMw==,9599,simonw,2020-10-30T21:34:38Z,2020-10-30T21:34:38Z,OWNER,"... no wait, my comments above assume that I'm just building the `datasette-edit-templates` plugin. Does this work as a general solution for all of Datasette? I don't think it does.

This may mean I need to delay the whole feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719811312,https://api.github.com/repos/simonw/datasette/issues/1072,719811312,MDEyOklzc3VlQ29tbWVudDcxOTgxMTMxMg==,9599,simonw,2020-10-30T21:36:49Z,2020-10-30T21:36:49Z,OWNER,"There's one other option: in `datasette-edit-templates` I could maybe use `asyncio.get_event_loop().run_in_executor(...)` to load the templates asynchronously within the Jinja template loader mechanism.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719813212,https://api.github.com/repos/simonw/datasette/issues/1072,719813212,MDEyOklzc3VlQ29tbWVudDcxOTgxMzIxMg==,9599,simonw,2020-10-30T21:42:35Z,2020-10-30T21:42:35Z,OWNER,Filed a feature request here: https://github.com/pallets/jinja/issues/1304,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719813970,https://api.github.com/repos/simonw/datasette/issues/1072,719813970,MDEyOklzc3VlQ29tbWVudDcxOTgxMzk3MA==,9599,simonw,2020-10-30T21:44:40Z,2020-10-30T21:44:40Z,OWNER,"I'm pretty sure that `run_in_executor()` workaround won't work. https://github.com/django/asgiref/blob/7becc9daca2628c46af1cb7e46b4c47c1ea27adf/asgiref/sync.py#L83 for example says ""You cannot use AsyncToSync in the same thread as an async event loop"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719814279,https://api.github.com/repos/simonw/datasette/issues/1072,719814279,MDEyOklzc3VlQ29tbWVudDcxOTgxNDI3OQ==,9599,simonw,2020-10-30T21:45:33Z,2020-10-30T21:45:33Z,OWNER,Sadly I'm going to bump `load_template` from Datasette 0.51 - I don't think I should block the release on resolving this issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719819234,https://api.github.com/repos/simonw/datasette/issues/1072,719819234,MDEyOklzc3VlQ29tbWVudDcxOTgxOTIzNA==,9599,simonw,2020-10-30T22:00:21Z,2020-10-30T22:00:21Z,OWNER,"There might be a way to save this. Async template loading can't be supported, but what if you could define a `load_template()` hook which returned a sync function that returned templates...

Then the `datasette-edit-templates` plugin could reply to `load_template` by loading all DB templates into memory and returning a `load_template` sync function that looked up the values in those already-loaded templates.

It could even maintain an in-memory cache that gets updated when a template is edited.

If I do this, I could remove the ability to return an `async` function from `load_template()` but add that in the future should Jinja implement a mechanism for async template loading.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719819331,https://api.github.com/repos/simonw/datasette/issues/1072,719819331,MDEyOklzc3VlQ29tbWVudDcxOTgxOTMzMQ==,9599,simonw,2020-10-30T22:00:43Z,2020-10-30T22:00:43Z,OWNER,I'll try getting that to work. If I can't get it to work I'll drop the plugin hook for the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719832651,https://api.github.com/repos/simonw/datasette/issues/1072,719832651,MDEyOklzc3VlQ29tbWVudDcxOTgzMjY1MQ==,9599,simonw,2020-10-30T22:46:25Z,2020-10-30T22:46:25Z,OWNER,"I tried using a `FunctionLoader` and got this error on startup:
```python
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 989, in __init__
    for filepath in self.ds.jinja_env.list_templates()
  File ""/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/environment.py"", line 810, in list_templates
    names = self.loader.list_templates()
  File ""/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/loaders.py"", line 434, in list_templates
    found.update(loader.list_templates())
  File ""/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/loaders.py"", line 99, in list_templates
    raise TypeError(""this loader cannot iterate over all templates"")
TypeError: this loader cannot iterate over all templates
```
So if I'm going to define a custom Jinja loader I'll need to teach plugins to answer the ""list templates"" query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719832853,https://api.github.com/repos/simonw/datasette/issues/1072,719832853,MDEyOklzc3VlQ29tbWVudDcxOTgzMjg1Mw==,9599,simonw,2020-10-30T22:47:12Z,2020-10-30T22:47:12Z,OWNER,Maybe I should ditch this hook entirely in favour of the existing `prepare_jinja2_environment` hook. Could that add new template loaders?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719833070,https://api.github.com/repos/simonw/datasette/issues/1072,719833070,MDEyOklzc3VlQ29tbWVudDcxOTgzMzA3MA==,9599,simonw,2020-10-30T22:48:04Z,2020-10-30T22:48:04Z,OWNER,"https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L310-L318

So yeah that plugin hook can probably modify the list of loaders available to the `Environment`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719833744,https://api.github.com/repos/simonw/datasette/issues/1072,719833744,MDEyOklzc3VlQ29tbWVudDcxOTgzMzc0NA==,9599,simonw,2020-10-30T22:50:57Z,2020-10-30T22:50:57Z,OWNER,Yeah I'm going to remove the `load_template` plugin hook and see if it's possible to build the edit templates extension against `prepare_jinja2_environment` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719955491,https://api.github.com/repos/simonw/datasette/issues/1072,719955491,MDEyOklzc3VlQ29tbWVudDcxOTk1NTQ5MQ==,9599,simonw,2020-10-31T16:20:58Z,2020-10-31T16:20:58Z,OWNER,"Here's the proof of concept `FunctionLoader` that showed me that this wasn't going to work:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 4b28e71..b076be7 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -21,7 +21,7 @@ from pathlib import Path
 from markupsafe import Markup
 from itsdangerous import URLSafeSerializer
 import jinja2
-from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
+from jinja2 import ChoiceLoader, Environment, FileSystemLoader, FunctionLoader, PrefixLoader
 from jinja2.environment import Template
 from jinja2.exceptions import TemplateNotFound
 import uvicorn
@@ -300,6 +300,7 @@ class Datasette:
         template_paths.append(default_templates)
         template_loader = ChoiceLoader(
             [
+                FunctionLoader(self._load_template_from_plugins),
                 FileSystemLoader(template_paths),
                 # Support {% extends ""default:table.html"" %}:
                 PrefixLoader(
@@ -322,6 +323,17 @@ class Datasette:
         self._root_token = secrets.token_hex(32)
         self.client = DatasetteClient(self)
 
+    def _load_template_from_plugins(self, template):
+        # ""If auto reloading is enabled it’s called to check if the template changed""
+        uptodatefunc = lambda: True
+        source = pm.hook.load_template(
+            template=template,
+            datasette=self,
+        )
+        if source is None:
+            return None
+        return source, template, uptodatefunc
+
     @property
     def urls(self):
         return Urls(self)
@@ -719,35 +731,7 @@ class Datasette:
         else:
             if isinstance(templates, str):
                 templates = [templates]
-
-            # Give plugins first chance at loading the template
-            break_outer = False
-            plugin_template_source = None
-            plugin_template_name = None
-            template_name = None
-            for template_name in templates:
-                if break_outer:
-                    break
-                plugin_template_source = pm.hook.load_template(
-                    template=template_name,
-                    request=request,
-                    datasette=self,
-                )
-                plugin_template_source = await await_me_maybe(plugin_template_source)
-                if plugin_template_source:
-                    break_outer = True
-                    plugin_template_name = template_name
-                    break
-            if plugin_template_source is not None:
-                template = self.jinja_env.from_string(plugin_template_source)
-            else:
-                template = self.jinja_env.select_template(templates)
-            for template_name in templates:
-                from_plugin = template_name == plugin_template_name
-                used = from_plugin or template_name == template.name
-                templates_considered.append(
-                    {""name"": template_name, ""used"": used, ""from_plugin"": from_plugin}
-                )
+            template = self.jinja_env.select_template(templates)
         body_scripts = []
         # pylint: disable=no-member
         for extra_script in pm.hook.extra_body_script(
diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py
index ca84b35..7804def 100644
--- a/datasette/hookspecs.py
+++ b/datasette/hookspecs.py
@@ -50,7 +50,7 @@ def extra_template_vars(
 
 
 @hookspec(firstresult=True)
-def load_template(template, request, datasette):
+def load_template(template, datasette):
     ""Load the specified template, returning the template code as a string""
 
 
diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst
index 3c57b6a..8f2704e 100644
--- a/docs/plugin_hooks.rst
+++ b/docs/plugin_hooks.rst
@@ -273,15 +273,12 @@ Example: `datasette-cluster-map <https://github.com/simonw/datasette-cluster-map
 
 .. _plugin_hook_load_template:
 
-load_template(template, request, datasette)
--------------------------------------------
+load_template(template, datasette)
+----------------------------------
 
 ``template`` - string
     The template that is being rendered, e.g. ``database.html``
 
-``request`` - object or None
-    The current HTTP :ref:`internals_request`. This can be ``None`` if the request object is not available.
-
 ``datasette`` - :ref:`internals_datasette`
     You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``
 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719986698,https://api.github.com/repos/simonw/datasette/issues/1072,719986698,MDEyOklzc3VlQ29tbWVudDcxOTk4NjY5OA==,9599,simonw,2020-10-31T20:48:17Z,2020-10-31T20:48:17Z,OWNER,Here's the `datasette-edit-templates` plugin WIP I had before removing the hook: https://github.com/simonw/datasette-edit-templates/tree/82855c2612b84bc09c48fca885f831633a0d1552,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1073#issuecomment-719834200,https://api.github.com/repos/simonw/datasette/issues/1073,719834200,MDEyOklzc3VlQ29tbWVudDcxOTgzNDIwMA==,9599,simonw,2020-10-30T22:52:48Z,2020-10-30T22:52:48Z,OWNER,Should mostly be a case of backing out the changes from this commit: 81dea4b07ab2b6f4eaaf248307d2b588472054a1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733560417,Remove load_template plugin hook,
https://github.com/simonw/datasette/issues/1074#issuecomment-719963074,https://api.github.com/repos/simonw/datasette/issues/1074,719963074,MDEyOklzc3VlQ29tbWVudDcxOTk2MzA3NA==,9599,simonw,2020-10-31T17:23:48Z,2020-10-31T17:23:48Z,OWNER,"Needs a way to login as root, seeing as several plugins only show extra content if the user is logged in as root.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733768037,latest.datasette.io should include plugins from fixtures,
https://github.com/simonw/datasette/issues/1074#issuecomment-719965426,https://api.github.com/repos/simonw/datasette/issues/1074,719965426,MDEyOklzc3VlQ29tbWVudDcxOTk2NTQyNg==,9599,simonw,2020-10-31T17:45:00Z,2020-10-31T17:45:00Z,OWNER,"This is working. Go to https://latest.datasette.io/login-as-root and click the button, then visit this page to see extra content added by plugins: https://latest.datasette.io/fixtures/compound_three_primary_keys

![latest-plugins](https://user-images.githubusercontent.com/9599/97786052-19e53d00-1b66-11eb-8268-b452e08965e3.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733768037,latest.datasette.io should include plugins from fixtures,
https://github.com/simonw/datasette/issues/1074#issuecomment-719977864,https://api.github.com/repos/simonw/datasette/issues/1074,719977864,MDEyOklzc3VlQ29tbWVudDcxOTk3Nzg2NA==,9599,simonw,2020-10-31T19:35:01Z,2020-10-31T19:35:01Z,OWNER,"These plugins were not designed to be actually hosted online, so they do some nasty things like linking to the made-up `plugin-example.com` domain. I should fix that.

<img width=""853"" alt=""Datasette_Fixtures__fixtures"" src=""https://user-images.githubusercontent.com/9599/97788348-7b60d800-1b75-11eb-8519-a58e52108841.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733768037,latest.datasette.io should include plugins from fixtures,
https://github.com/simonw/datasette/issues/1075#issuecomment-719980742,https://api.github.com/repos/simonw/datasette/issues/1075,719980742,MDEyOklzc3VlQ29tbWVudDcxOTk4MDc0Mg==,9599,simonw,2020-10-31T19:58:57Z,2020-10-31T19:58:57Z,OWNER,"Sample traceback:
```
    <link rel=""stylesheet"" href=""{{ urls.static('app.css') }}?{{ app_css_hash }}"">
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/jinja2/asyncsupport.py:173: in auto_await
    if inspect.isawaitable(value):
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/inspect.py:226: in isawaitable
    isinstance(object, collections.abc.Awaitable))
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/abc.py:139: in __instancecheck__
    return _abc_instancecheck(cls, instance)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'collections.abc.Awaitable'>
subclass = <bound method PrefixedUrlString.__getattribute__.<locals>.method of '/-/static/app.css'>

    def __subclasscheck__(cls, subclass):
        """"""Override for issubclass(subclass, cls).""""""
>       return _abc_subclasscheck(cls, subclass)
E       TypeError: issubclass() arg 1 must be a class
```
This is within Jinja. It looks like Jinja really doesn't like methods that return non-string objects like `PrefixedUrlString`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719981173,https://api.github.com/repos/simonw/datasette/issues/1075,719981173,MDEyOklzc3VlQ29tbWVudDcxOTk4MTE3Mw==,9599,simonw,2020-10-31T20:02:30Z,2020-10-31T20:03:45Z,OWNER,"I wonder how Jinja's `Markup()` class works? It uses https://pypi.org/project/MarkupSafe/

It's a subclass of `str`, defined here: https://github.com/pallets/markupsafe/blob/master/src/markupsafe/__init__.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719983240,https://api.github.com/repos/simonw/datasette/issues/1075,719983240,MDEyOklzc3VlQ29tbWVudDcxOTk4MzI0MA==,9599,simonw,2020-10-31T20:18:49Z,2020-10-31T20:18:49Z,OWNER,"Here's the core problem:
```
(Pdb) isinstance('bob', collections.abc.Awaitable)
False
(Pdb) isinstance(PrefixedUrlString('bob'), collections.abc.Awaitable)
*** TypeError: issubclass() arg 1 must be a class
```
For some reason `isinstance()` does not like being handed an instance of PrefixedUrlString.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719983484,https://api.github.com/repos/simonw/datasette/issues/1075,719983484,MDEyOklzc3VlQ29tbWVudDcxOTk4MzQ4NA==,9599,simonw,2020-10-31T20:20:28Z,2020-10-31T20:20:28Z,OWNER,"It looks like this is specific to the way `PrefixedUrlString` is built.
```
(Pdb) class Weird(str): pass
(Pdb) isinstance(Weird('bob'), collections.abc.Awaitable)
False
```
So subclassing strings doesn't trigger this bug, but something about `PrefixedUrlString` causes the problem.

Here's the current `PrefixedUrlString` implementation:

https://github.com/simonw/datasette/blob/bf18b9ba175a7b25fb8b765847397dd6efb8bb7b/datasette/utils/__init__.py#L1015-L1035","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719983565,https://api.github.com/repos/simonw/datasette/issues/1075,719983565,MDEyOklzc3VlQ29tbWVudDcxOTk4MzU2NQ==,9599,simonw,2020-10-31T20:21:03Z,2020-10-31T20:21:03Z,OWNER,"Here's the output of `dir(str)`:

`['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'capitalize', 'casefold', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index', 'isalnum', 'isalpha', 'isascii', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719983750,https://api.github.com/repos/simonw/datasette/issues/1075,719983750,MDEyOklzc3VlQ29tbWVudDcxOTk4Mzc1MA==,9599,simonw,2020-10-31T20:22:29Z,2020-10-31T20:22:29Z,OWNER,I bet this is because I'm mucking around with one of those `__` methods. I'll try just doing the non-underscore methods instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1077#issuecomment-720003026,https://api.github.com/repos/simonw/datasette/issues/1077,720003026,MDEyOklzc3VlQ29tbWVudDcyMDAwMzAyNg==,9599,simonw,2020-10-31T23:48:21Z,2020-10-31T23:50:07Z,OWNER,Needed by https://github.com/simonw/datasette-backup/issues/6,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733829385,database_actions plugin hook,
https://github.com/simonw/datasette/issues/1077#issuecomment-720637322,https://api.github.com/repos/simonw/datasette/issues/1077,720637322,MDEyOklzc3VlQ29tbWVudDcyMDYzNzMyMg==,9599,simonw,2020-11-02T18:09:17Z,2020-11-02T18:09:17Z,OWNER,Here's the `table_actions` implementation: 2f7731e9e5ff9b324beb5039fbe2be55d704a184,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733829385,database_actions plugin hook,
https://github.com/simonw/datasette/issues/1077#issuecomment-720654925,https://api.github.com/repos/simonw/datasette/issues/1077,720654925,MDEyOklzc3VlQ29tbWVudDcyMDY1NDkyNQ==,9599,simonw,2020-11-02T18:43:25Z,2020-11-02T18:43:25Z,OWNER,Demo: https://latest.datasette.io/fixtures?_bot=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733829385,database_actions plugin hook,
https://github.com/simonw/datasette/issues/1079#issuecomment-720110298,https://api.github.com/repos/simonw/datasette/issues/1079,720110298,MDEyOklzc3VlQ29tbWVudDcyMDExMDI5OA==,9599,simonw,2020-11-01T15:58:22Z,2020-11-01T15:58:22Z,OWNER,Might try a drop shadow on that menu too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733999615,Handle long breadcrumbs better with new menu,
https://github.com/simonw/datasette/issues/1080#issuecomment-1029691693,https://api.github.com/repos/simonw/datasette/issues/1080,1029691693,IC_kwDOBm6k_c49X9kt,9599,simonw,2022-02-04T06:16:45Z,2022-02-04T06:16:45Z,OWNER,"Had a new, different idea for how this could work: support a `?_group_count=colname` parameter to the table view, which turns the page into a `select colname, count(*) ... group by colname` query - but keeps things like the filter interface, facet selection, search box and so on.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",
https://github.com/simonw/datasette/issues/1080#issuecomment-1029695083,https://api.github.com/repos/simonw/datasette/issues/1080,1029695083,IC_kwDOBm6k_c49X-Zr,9599,simonw,2022-02-04T06:24:40Z,2022-02-04T06:25:18Z,OWNER,"An initial prototype of that in my local `group-count` branch quickly started running into problems:
```diff
diff --git a/datasette/views/table.py b/datasette/views/table.py
index be9e9c3..d30efe1 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -105,8 +105,12 @@ class RowTableShared(DataView):
                 type_ = ""integer""
                 notnull = 0
             else:
-                type_ = column_details[r[0]].type
-                notnull = column_details[r[0]].notnull
+                try:
+                    type_ = column_details[r[0]].type
+                    notnull = column_details[r[0]].notnull
+                except KeyError: # Probably count(*)
+                    type_ = ""integer""
+                    notnull = False
             columns.append(
                 {
                     ""name"": r[0],
@@ -613,6 +617,15 @@ class TableView(RowTableShared):
             offset=offset,
         )
 
+        # If ?_group_count we convert the SQL query here
+        group_count = request.args.getlist(""_group_count"")
+        if group_count:
+            wrapped_sql = ""select {cols}, count(*) from ({sql}) group by {cols}"".format(
+                cols="", "".join(group_count),
+                sql=sql,
+            )
+            sql = wrapped_sql
+
         if request.args.get(""_timelimit""):
             extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit""))
```
Resulted in errors like this one:
```
    pk_path = path_from_row_pks(row, pks, not pks, False)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 82, in path_from_row_pks
    bits = [
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 83, in <listcomp>
    row[pk][""value""] if isinstance(row[pk], dict) else row[pk] for pk in pks
IndexError: No item with that key
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",
https://github.com/simonw/datasette/issues/1080#issuecomment-720695174,https://api.github.com/repos/simonw/datasette/issues/1080,720695174,MDEyOklzc3VlQ29tbWVudDcyMDY5NTE3NA==,9599,simonw,2020-11-02T20:05:26Z,2020-11-02T20:05:26Z,OWNER,"URL design:

`/database/table/-/facet/colname`

And for other types of facet (to be supported later):

`/database/table/-/facet/colname?_type=m2m`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",
https://github.com/simonw/datasette/issues/1080#issuecomment-720696827,https://api.github.com/repos/simonw/datasette/issues/1080,720696827,MDEyOklzc3VlQ29tbWVudDcyMDY5NjgyNw==,9599,simonw,2020-11-02T20:08:49Z,2020-11-02T20:13:56Z,OWNER,"Implementing pagination for facets will be interesting. Would be easier if I had a nicer reusable internal pagination mechanism, which is also needed for #856 (pagination of canned queries).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",
https://github.com/simonw/datasette/issues/1080#issuecomment-720697226,https://api.github.com/repos/simonw/datasette/issues/1080,720697226,MDEyOklzc3VlQ29tbWVudDcyMDY5NzIyNg==,9599,simonw,2020-11-02T20:09:38Z,2020-11-02T20:09:38Z,OWNER,"Maybe this ends up being code that defers to a simulated canned query, rendered using the existing `query.html` template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",
https://github.com/simonw/datasette/issues/1080#issuecomment-720698577,https://api.github.com/repos/simonw/datasette/issues/1080,720698577,MDEyOklzc3VlQ29tbWVudDcyMDY5ODU3Nw==,9599,simonw,2020-11-02T20:12:26Z,2020-11-02T20:12:26Z,OWNER,"For regular column faceting, here's the query that is used:

https://github.com/simonw/datasette/blob/13d1228d80c91d382a05b1a9549ed02c300ef851/datasette/facets.py#L196-L204

Since it uses `order by count desc, value` maybe those values could be used to implement cursor-based pagination.

That wouldn't be robust in the face of changing data, but I'm not sure it's possible to implement paginated faceting in a way that survives ongoing changes to the underlying data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",
https://github.com/simonw/datasette/issues/1080#issuecomment-720699160,https://api.github.com/repos/simonw/datasette/issues/1080,720699160,MDEyOklzc3VlQ29tbWVudDcyMDY5OTE2MA==,9599,simonw,2020-11-02T20:13:42Z,2020-11-02T20:13:42Z,OWNER,Also relevant to this issue: #830 - redesigning the facet plugin hook in preparation for Datasette 1.0. And #972 supporting faceting against arbitrary queries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",
https://github.com/simonw/datasette/issues/1082#issuecomment-721545090,https://api.github.com/repos/simonw/datasette/issues/1082,721545090,MDEyOklzc3VlQ29tbWVudDcyMTU0NTA5MA==,9599,simonw,2020-11-04T06:47:15Z,2020-11-04T06:47:15Z,OWNER,"I've run into a similar problem with Google Cloud Run: beyond a certain size of database file I find myself needing to run instances there with more RAM assigned to them.

I haven't yet figured out a method to estimate the amount of RAM that will be needed to successfully serve a database file of a specific size- I've been using trial and error.

5GB is quite a big database file, so it doesn't surprise me that it may need a bigger instance. I recommend trying it on a 1GB or 2GB of RAM Digital Ocean instance (their default is 512MB) and see if that works.

Let me know what you find out!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735852274,DigitalOcean buildpack memory errors for large sqlite db?,
https://github.com/simonw/datasette/issues/1082#issuecomment-721547177,https://api.github.com/repos/simonw/datasette/issues/1082,721547177,MDEyOklzc3VlQ29tbWVudDcyMTU0NzE3Nw==,39538958,justmars,2020-11-04T06:52:30Z,2020-11-04T06:53:16Z,NONE,"I think I tried the same db size on the following scenarios in Digital Ocean:
1. Basic ($5/month) with 512MB RAM
2. Basic ($10/month) with 1GB RAM
3. Pro ($12/month) with 1GB RAM

All such attempts conked out with ""out of memory"" errors","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735852274,DigitalOcean buildpack memory errors for large sqlite db?,
https://github.com/simonw/datasette/issues/1082#issuecomment-721931504,https://api.github.com/repos/simonw/datasette/issues/1082,721931504,MDEyOklzc3VlQ29tbWVudDcyMTkzMTUwNA==,9599,simonw,2020-11-04T19:32:47Z,2020-11-04T19:35:44Z,OWNER,"I wonder if setting a soft memory limit within Datasette would help here: https://www.sqlite.org/malloc.html#_setting_memory_usage_limits 

>  If attempts are made to allocate more memory than specified by the soft heap limit, then SQLite will first attempt to free cache memory before continuing with the allocation request.

https://www.sqlite.org/pragma.html#pragma_soft_heap_limit

> **PRAGMA soft_heap_limit**
> **PRAGMA soft_heap_limit=N**
> 
> This pragma invokes the [sqlite3_soft_heap_limit64()](https://www.sqlite.org/c3ref/hard_heap_limit64.html) interface with the argument N, if N is specified and is a non-negative integer. The soft_heap_limit pragma always returns the same integer that would be returned by the [sqlite3_soft_heap_limit64](https://www.sqlite.org/c3ref/hard_heap_limit64.html)(-1) C-language function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735852274,DigitalOcean buildpack memory errors for large sqlite db?,
https://github.com/simonw/datasette/issues/1083#issuecomment-721926827,https://api.github.com/repos/simonw/datasette/issues/1083,721926827,MDEyOklzc3VlQ29tbWVudDcyMTkyNjgyNw==,9599,simonw,2020-11-04T19:23:42Z,2020-11-04T19:23:42Z,OWNER,"https://latest.datasette.io/fixtures/sortable#export has advanced export options, but https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+order+by+pk1%2C+pk2+limit+101 does not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736365306,Advanced CSV export for arbitrary queries,
https://github.com/simonw/datasette/issues/1083#issuecomment-721927254,https://api.github.com/repos/simonw/datasette/issues/1083,721927254,MDEyOklzc3VlQ29tbWVudDcyMTkyNzI1NA==,9599,simonw,2020-11-04T19:24:34Z,2020-11-04T19:24:34Z,OWNER,"Related: #856 - if it's possible to paginate correctly configured canned query then the CSV option to ""stream all rows"" could work for queries as well as tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736365306,Advanced CSV export for arbitrary queries,
https://github.com/simonw/datasette/issues/1084#issuecomment-731644064,https://api.github.com/repos/simonw/datasette/issues/1084,731644064,MDEyOklzc3VlQ29tbWVudDczMTY0NDA2NA==,9599,simonw,2020-11-21T22:10:15Z,2020-11-21T22:10:15Z,OWNER,"Another example of this bug: https://datasette-graphql-demo.datasette.io/github/users

<img width=""350"" alt=""github__users__4_823_rows_and_Comparing_1_1___main_·_simonw_datasette-graphql"" src=""https://user-images.githubusercontent.com/9599/99888639-3889a180-2c03-11eb-83b9-41f2eb12287e.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737394470,Table/database action menu cut off if too short ,
https://github.com/simonw/datasette/issues/1084#issuecomment-731652991,https://api.github.com/repos/simonw/datasette/issues/1084,731652991,MDEyOklzc3VlQ29tbWVudDczMTY1Mjk5MQ==,9599,simonw,2020-11-21T23:34:22Z,2020-11-21T23:34:22Z,OWNER,"Fixed by positioning the menu relative to the `<div class=""page-header"">` element rather than the cog icon:

<img width=""966"" alt=""data"" src=""https://user-images.githubusercontent.com/9599/99889950-02eab580-2c0f-11eb-85dc-dd4bc840a92e.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737394470,Table/database action menu cut off if too short ,
https://github.com/simonw/datasette/issues/1084#issuecomment-731653083,https://api.github.com/repos/simonw/datasette/issues/1084,731653083,MDEyOklzc3VlQ29tbWVudDczMTY1MzA4Mw==,9599,simonw,2020-11-21T23:35:07Z,2020-11-21T23:35:07Z,OWNER,I got to use CSS `calc()` for this: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/static/app.css#L367-L371,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737394470,Table/database action menu cut off if too short ,
https://github.com/simonw/datasette/issues/1084#issuecomment-731654132,https://api.github.com/repos/simonw/datasette/issues/1084,731654132,MDEyOklzc3VlQ29tbWVudDczMTY1NDEzMg==,9599,simonw,2020-11-21T23:45:59Z,2020-11-21T23:45:59Z,OWNER,https://datasette-graphql-demo.datasette.io/github/users now demonstrates the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737394470,Table/database action menu cut off if too short ,
https://github.com/simonw/datasette/issues/1086#issuecomment-725622784,https://api.github.com/repos/simonw/datasette/issues/1086,725622784,MDEyOklzc3VlQ29tbWVudDcyNTYyMjc4NA==,9599,simonw,2020-11-11T19:41:45Z,2020-11-11T19:41:45Z,OWNER,Maybe use a grey hyphen here?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741021342,Foreign keys with blank titles result in non-clickable links,
https://github.com/simonw/datasette/issues/1086#issuecomment-725714908,https://api.github.com/repos/simonw/datasette/issues/1086,725714908,MDEyOklzc3VlQ29tbWVudDcyNTcxNDkwOA==,9599,simonw,2020-11-11T23:17:26Z,2020-11-11T23:17:26Z,OWNER,I'm just going to use a regular coloured hyphen.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741021342,Foreign keys with blank titles result in non-clickable links,
https://github.com/simonw/datasette/issues/1086#issuecomment-725729857,https://api.github.com/repos/simonw/datasette/issues/1086,725729857,MDEyOklzc3VlQ29tbWVudDcyNTcyOTg1Nw==,9599,simonw,2020-11-11T23:55:39Z,2020-11-11T23:55:39Z,OWNER,"Demo: https://latest.datasette.io/fixtures/foreign_key_references?_facet=foreign_key_with_blank_label

<img width=""772"" alt=""fixtures__foreign_key_references__2_rows"" src=""https://user-images.githubusercontent.com/9599/98877868-53575b80-2436-11eb-8669-f24eeb8d0f3c.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741021342,Foreign keys with blank titles result in non-clickable links,
https://github.com/simonw/datasette/issues/1088#issuecomment-725829903,https://api.github.com/repos/simonw/datasette/issues/1088,725829903,MDEyOklzc3VlQ29tbWVudDcyNTgyOTkwMw==,9599,simonw,2020-11-12T04:33:14Z,2020-11-12T04:33:14Z,OWNER,"I'm suspicious of this code: https://github.com/simonw/datasette/blob/e8e0a6f284ca953b2980186c4356594c07bd1929/datasette/views/table.py#L1032-L1045

This code uses a different exception: https://github.com/simonw/datasette/blob/e8e0a6f284ca953b2980186c4356594c07bd1929/datasette/views/table.py#L658-L663

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741268956,OperationalError('interrupted') can 500 on row page,
https://github.com/simonw/datasette/issues/1088#issuecomment-725830533,https://api.github.com/repos/simonw/datasette/issues/1088,725830533,MDEyOklzc3VlQ29tbWVudDcyNTgzMDUzMw==,9599,simonw,2020-11-12T04:35:08Z,2020-11-12T04:35:08Z,OWNER,"Yup, swapping `QueryInterrupted` fixed this against my giant database.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741268956,OperationalError('interrupted') can 500 on row page,
https://github.com/simonw/datasette/issues/1088#issuecomment-725830716,https://api.github.com/repos/simonw/datasette/issues/1088,725830716,MDEyOklzc3VlQ29tbWVudDcyNTgzMDcxNg==,9599,simonw,2020-11-12T04:35:38Z,2020-11-12T04:35:38Z,OWNER,"I'm going to fix this without a test, because writing a test for this is a bit fiddly and it's a very minor bug. If it comes back again I'll do the work to test for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741268956,OperationalError('interrupted') can 500 on row page,
https://github.com/simonw/datasette/issues/1089#issuecomment-726127465,https://api.github.com/repos/simonw/datasette/issues/1089,726127465,MDEyOklzc3VlQ29tbWVudDcyNjEyNzQ2NQ==,9599,simonw,2020-11-12T14:54:11Z,2020-11-12T14:54:11Z,OWNER,"Suggested list to look out for from that PR:

- simply/simple
- easy/easier/easiest
- obvious/obviously
- just
- merely
- straightforward
- ridiculous","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741665726,Sweep documentation for words that minimize involved difficulty,
https://github.com/simonw/datasette/issues/109#issuecomment-344986423,https://api.github.com/repos/simonw/datasette/issues/109,344986423,MDEyOklzc3VlQ29tbWVudDM0NDk4NjQyMw==,9599,simonw,2017-11-16T16:53:26Z,2017-11-16T16:53:26Z,OWNER,http://datasette.readthedocs.io/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274378301,Set up readthedocs,
https://github.com/simonw/datasette/issues/1090#issuecomment-766463496,https://api.github.com/repos/simonw/datasette/issues/1090,766463496,MDEyOklzc3VlQ29tbWVudDc2NjQ2MzQ5Ng==,9599,simonw,2021-01-24T23:57:00Z,2021-01-24T23:57:00Z,OWNER,Related: I built [datasette-leaflet-freedraw](https://datasette.io/plugins/datasette-leaflet-freedraw) which turns any canned query field called `freedraw` or `something_freedraw` into an interactive map that you can draw on to create a GeoJSON MultiPolygon.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741862364,Custom widgets for canned query forms,
https://github.com/simonw/datasette/issues/1090#issuecomment-808757155,https://api.github.com/repos/simonw/datasette/issues/1090,808757155,MDEyOklzc3VlQ29tbWVudDgwODc1NzE1NQ==,9599,simonw,2021-03-27T16:21:43Z,2021-03-27T16:21:43Z,OWNER,"Idea for these: imitate https://django-sql-dashboard.readthedocs.io/en/latest/widgets.html#custom-widgets and drive them with templates.

So a custom widget type of `textarea` would look for a template called `widgets/textarea.html` - which means users could define brand new custom widgets just by creating their own template files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741862364,Custom widgets for canned query forms,
https://github.com/simonw/datasette/issues/1090#issuecomment-808757659,https://api.github.com/repos/simonw/datasette/issues/1090,808757659,MDEyOklzc3VlQ29tbWVudDgwODc1NzY1OQ==,9599,simonw,2021-03-27T16:25:25Z,2021-03-27T16:25:25Z,OWNER,Related feature request: ability to set default values for canned queries: #1258,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741862364,Custom widgets for canned query forms,
https://github.com/simonw/datasette/issues/1091#issuecomment-726413829,https://api.github.com/repos/simonw/datasette/issues/1091,726413829,MDEyOklzc3VlQ29tbWVudDcyNjQxMzgyOQ==,9599,simonw,2020-11-12T23:52:50Z,2020-11-12T23:54:16Z,OWNER,"Hmm... it's not just the `.csv` and `.json` export links - it's the column headings (which can be clicked to change the sort order) as well. Here's an extract of the HTML from that page:
```html
<p class=""export-links"">This data as 
  <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json"">json</a>, 
  <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.csv?_size=max"">CSV</a> (
  <a href=""#export"">advanced</a>)
</p>
<div class=""table-wrapper"">
  <table class=""rows-and-columns"">
    <thead>
      <tr>
        <th class=""col-Link"" scope=""col"" data-column=""Link"" data-column-type="""" data-column-not-null=""0"" data-is-pk=""0"">
          Link
        </th>
        <th class=""col-rowid"" scope=""col"" data-column=""rowid"" data-column-type=""integer"" data-column-not-null=""0"" data-is-pk=""1"">
          <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort_desc=rowid"" rel=""nofollow"">rowid&nbsp;▼</a>
        </th>
        <th class=""col-PARSE_EXCEPTION_ID"" scope=""col"" data-column=""PARSE_EXCEPTION_ID"" data-column-type=""INTEGER"" data-column-not-null=""0"" data-is-pk=""0"">
          <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort=PARSE_EXCEPTION_ID"" rel=""nofollow"">PARSE_EXCEPTION_ID</a>
        </th>
        <th class=""col-PARSE_EXCEPTION_DESCRIPTION"" scope=""col"" data-column=""PARSE_EXCEPTION_DESCRIPTION"" data-column-type=""VARCHAR(128)"" data-column-not-null=""0"" data-is-pk=""0"">
          <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort=PARSE_EXCEPTION_DESCRIPTION"" rel=""nofollow"">PARSE_EXCEPTION_DESCRIPTION</a>
        </th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td class=""col-Link type-pk"">
          <a href=""/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/1"">1</a>
        </td>
        <td class=""col-rowid type-int"">1</td>
        <td class=""col-PARSE_EXCEPTION_ID type-int"">0</td>
        <td class=""col-PARSE_EXCEPTION_DESCRIPTION type-str"">RUNTIME</td>
      </tr>
      <tr>
        <td class=""col-Link type-pk"">
          <a href=""/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/2"">2</a>
        </td>
        <td class=""col-rowid type-int"">2</td>
        <td class=""col-PARSE_EXCEPTION_ID type-int"">1</td>
        <td class=""col-PARSE_EXCEPTION_DESCRIPTION type-str"">ENCRYPTION</td>
      </tr>
      <tr>
        <td class=""col-Link type-pk"">
          <a href=""/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/3"">3</a>
        </td>
        <td class=""col-rowid type-int"">3</td>
        <td class=""col-PARSE_EXCEPTION_ID type-int"">2</td>
        <td class=""col-PARSE_EXCEPTION_DESCRIPTION type-str"">ACCESS_PERMISSION</td>
      </tr>
      <tr>
        <td class=""col-Link type-pk"">
          <a href=""/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/4"">4</a>
        </td>
        <td class=""col-rowid type-int"">4</td>
        <td class=""col-PARSE_EXCEPTION_ID type-int"">3</td>
        <td class=""col-PARSE_EXCEPTION_DESCRIPTION type-str"">UNSUPPORTED_VERSION</td>
      </tr>
    </tbody>
  </table>
</div>
<div id=""export"" class=""advanced-export"">
  <h3>Advanced export</h3>
  <p>JSON shape:
    <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json"">default</a>,
    <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json?_shape=array"">array</a>,
    <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json?_shape=array&amp;_nl=on"">newline-delimited</a>
  </p>
  <form action=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.csv"" method=""get"">
    <p>
      CSV options:            
      <label>
        <input type=""checkbox"" name=""_dl""> download file
        </label>
        <input type=""submit"" value=""Export CSV"">
          <input type=""hidden"" name=""_size"" value=""max"">
          </p>
        </form>
      </div>
```
But here's something _really_ weird - the links to the individual rows DO include the `/datasette/` prefix:
```html
<td class=""col-Link type-pk"">
    <a href=""/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/2"">2</a>
</td>
```
The navigation bar on that page is correct too:
```html
<p class=""crumbs"">
        <a href=""/datasette/"">home</a> /
        <a href=""/datasette/corpora-metadata"">corpora-metadata</a>
</p>
```
I've also been unable to replicate this in my own local environment, running `datasette fixtures.db --config base_url:/datasette/`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-726415019,https://api.github.com/repos/simonw/datasette/issues/1091,726415019,MDEyOklzc3VlQ29tbWVudDcyNjQxNTAxOQ==,9599,simonw,2020-11-12T23:56:23Z,2020-11-12T23:56:23Z,OWNER,@tballison is there any chance you're running any custom templates in that installation? I'm really confused as to why I can't replicate the bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-726415991,https://api.github.com/repos/simonw/datasette/issues/1091,726415991,MDEyOklzc3VlQ29tbWVudDcyNjQxNTk5MQ==,9599,simonw,2020-11-12T23:59:34Z,2020-11-12T23:59:34Z,OWNER,"The sort headers are generated by this template code: https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/templates/_table.html#L11-L15

The export links use this code:

https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/templates/table.html#L134

https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/templates/table.html#L180-L201","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-726416330,https://api.github.com/repos/simonw/datasette/issues/1091,726416330,MDEyOklzc3VlQ29tbWVudDcyNjQxNjMzMA==,9599,simonw,2020-11-13T00:00:43Z,2020-11-13T00:00:43Z,OWNER,Here's where `url_csv` comes from: https://github.com/simonw/datasette/blob/11eb1e026f3d84cb771f8d6e204939cbaee130cd/datasette/views/base.py#L542-L545,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-726798745,https://api.github.com/repos/simonw/datasette/issues/1091,726798745,MDEyOklzc3VlQ29tbWVudDcyNjc5ODc0NQ==,6739646,tballison,2020-11-13T14:35:22Z,2020-11-13T14:35:22Z,NONE,"I'm starting this with docker like so:

`docker run --name datasette -d -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/file_profiles.db --config sql_time_limit_ms:120000 --config max_returned_rows:100000 --config base_url:/datasette/ --config cache_size_kb:50000`

I'm not doing any templating or anything else custom.

Apropos of nothing, I swapped out a simpler db, so this query should now work:

https://corpora.tika.apache.org/datasette/file_profiles?sql=select%0D%0A++*%0D%0Afrom%0D%0A++file_profiles+fp%0D%0Alimit%0D%0A++10","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-726801731,https://api.github.com/repos/simonw/datasette/issues/1091,726801731,MDEyOklzc3VlQ29tbWVudDcyNjgwMTczMQ==,6739646,tballison,2020-11-13T14:40:56Z,2020-11-13T14:40:56Z,NONE,"My headers aren't clickable/sortable with custom sql, but I think that's by design.

In the default view, https://corpora.tika.apache.org/datasette/file_profiles/file_profiles, ah, y, now I see that the headers should be sortable, but you're right the base_url is not applied.

base_url works with ""View and Edit SQL"" and with ""(advanced)""

As you point out, does not work with the export csv, json, other or with the ""Next page"" navigational button at the bottom.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-727233553,https://api.github.com/repos/simonw/datasette/issues/1091,727233553,MDEyOklzc3VlQ29tbWVudDcyNzIzMzU1Mw==,9599,simonw,2020-11-14T16:46:52Z,2020-11-14T16:46:52Z,OWNER,@tballison could I see the section of your Apache config that configures the proxying to `/datasette/`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-728262974,https://api.github.com/repos/simonw/datasette/issues/1091,728262974,MDEyOklzc3VlQ29tbWVudDcyODI2Mjk3NA==,9599,simonw,2020-11-16T19:05:08Z,2020-11-16T19:05:08Z,OWNER,I have a hunch that there may be some extra configuration in play here - could Apache itself be rewriting some of the links using [mod_proxy_html](https://httpd.apache.org/docs/2.4/mod/mod_proxy_html.html)?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-729018386,https://api.github.com/repos/simonw/datasette/issues/1091,729018386,MDEyOklzc3VlQ29tbWVudDcyOTAxODM4Ng==,6739646,tballison,2020-11-17T15:48:58Z,2020-11-17T15:48:58Z,NONE,"I don't think we are, but I'll check with Maruan.

I think this is the relevant part of our config?

```
  Alias ""/base/"" ""/usr/share/corpora/""
  <Directory ""/usr/share/corpora/"">
    Options +Indexes -Multiviews
    AllowOverride None
  </Directory>

  ProxyPreserveHost On

  ProxyPass /datasette http://0.0.0.0:8001
  ProxyPassReverse /datasette http://0.0.0.0:8001

</VirtualHost>
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-729045320,https://api.github.com/repos/simonw/datasette/issues/1091,729045320,MDEyOklzc3VlQ29tbWVudDcyOTA0NTMyMA==,6739646,tballison,2020-11-17T16:31:00Z,2020-11-17T16:31:00Z,NONE,We're using mod_proxy.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-741804334,https://api.github.com/repos/simonw/datasette/issues/1091,741804334,MDEyOklzc3VlQ29tbWVudDc0MTgwNDMzNA==,6739646,tballison,2020-12-09T14:26:05Z,2020-12-09T14:26:05Z,NONE,"Anything we can do to help debug this?  Thank you, again!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-741992106,https://api.github.com/repos/simonw/datasette/issues/1091,741992106,MDEyOklzc3VlQ29tbWVudDc0MTk5MjEwNg==,9599,simonw,2020-12-09T19:19:54Z,2020-12-09T20:27:45Z,OWNER,"Could you try removing the `ProxyPassReverse /datasette http://0.0.0.0:8001` line?

My hunch is that `ProxyPassReverse` is rewriting some of the links in the HTML (or maybe in the HTTP headers) in a way that breaks things.

Normally you would need `ProxyPassReverse` to compensate for the underlying application being unable to rewrite its links - but Datasette's `base_url` setting causes Datasette to rewrite all of the links for you, so `ProxyPassReverse` should be unneccessary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-742001510,https://api.github.com/repos/simonw/datasette/issues/1091,742001510,MDEyOklzc3VlQ29tbWVudDc0MjAwMTUxMA==,6739646,tballison,2020-12-09T19:36:42Z,2020-12-09T19:38:04Z,NONE,"I don't think this fixes it:

```
grep -R datasette .
./sites-available/000-default.conf:        ProxyPass /datasette http://127.0.0.1:8001/
./sites-available/000-default.conf:        #ProxyPassReverse /datasette http://127.0.0.1:8001/
./sites-available/corpora-le-ssl.conf:  ProxyPass /datasette http://0.0.0.0:8001
./sites-available/corpora-le-ssl.conf:  #ProxyPassReverse /datasette http://0.0.0.0:8001
./sites-enabled/corpora-le-ssl.conf:  ProxyPass /datasette http://0.0.0.0:8001
./sites-enabled/corpora-le-ssl.conf:  #ProxyPassReverse /datasette http://0.0.0.0:8001
```

And I confirmed that I actually restarted the server. :rofl: 

https://corpora.tika.apache.org/datasette/file_profiles","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-742010306,https://api.github.com/repos/simonw/datasette/issues/1091,742010306,MDEyOklzc3VlQ29tbWVudDc0MjAxMDMwNg==,6739646,tballison,2020-12-09T19:53:18Z,2020-12-09T19:59:52Z,NONE,"I can't imagine this helps (esp. given your point about potential rewrites), but you can see that /datasette/ was correctly added to the sql form, but not to the ""export-links""

 <img width=""484"" alt=""Screen Shot 2020-12-09 at 2 51 09 PM"" src=""https://user-images.githubusercontent.com/6739646/101680055-234baa00-3a2e-11eb-8650-2b369bc6f031.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-742023541,https://api.github.com/repos/simonw/datasette/issues/1091,742023541,MDEyOklzc3VlQ29tbWVudDc0MjAyMzU0MQ==,9599,simonw,2020-12-09T20:17:54Z,2020-12-09T20:17:54Z,OWNER,OK that is really weird. I'll have another go at replicating this locally.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-756425587,https://api.github.com/repos/simonw/datasette/issues/1091,756425587,MDEyOklzc3VlQ29tbWVudDc1NjQyNTU4Nw==,19328961,henry501,2021-01-07T22:27:19Z,2021-01-07T22:27:19Z,NONE,"I found this issue while troubleshooting the same behavior with an nginx reverse proxy. The solution was to make sure I set:

`proxy_pass http://server:8001/baseurl/ 
`
instead of just:

`proxy_pass http://server:8001
`
The custom SQL query and header links are now correct.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-756453010,https://api.github.com/repos/simonw/datasette/issues/1091,756453010,MDEyOklzc3VlQ29tbWVudDc1NjQ1MzAxMA==,9599,simonw,2021-01-07T23:39:58Z,2021-01-07T23:40:25Z,OWNER,"@tballison I think that's the solution! It looks like you need to use this in your config:

`ProxyPass /datasette http://127.0.0.1:8001/datasette`

Instead of this:

`ProxyPass /datasette http://127.0.0.1:8001/`

Give that a go and let me know if it fixes it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-756453945,https://api.github.com/repos/simonw/datasette/issues/1091,756453945,MDEyOklzc3VlQ29tbWVudDc1NjQ1Mzk0NQ==,9599,simonw,2021-01-07T23:42:50Z,2021-01-07T23:42:50Z,OWNER,"@henry501 it looks like you spotted a bug in the documentation - I just addressed that, the fix is now live here: https://docs.datasette.io/en/latest/deploying.html#running-datasette-behind-a-proxy","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-758280611,https://api.github.com/repos/simonw/datasette/issues/1091,758280611,MDEyOklzc3VlQ29tbWVudDc1ODI4MDYxMQ==,6739646,tballison,2021-01-11T23:06:10Z,2021-01-11T23:06:10Z,NONE,"+1

Yep!  Fixes it.  If I navigate to https://corpora.tika.apache.org/datasette, I get a 404 (database not found: datasette), but if I navigate to https://corpora.tika.apache.org/datasette/file_profiles/, everything WORKS!

Thank you!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-758283074,https://api.github.com/repos/simonw/datasette/issues/1091,758283074,MDEyOklzc3VlQ29tbWVudDc1ODI4MzA3NA==,9599,simonw,2021-01-11T23:12:46Z,2021-01-11T23:12:46Z,OWNER,Fantastic!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-758448525,https://api.github.com/repos/simonw/datasette/issues/1091,758448525,MDEyOklzc3VlQ29tbWVudDc1ODQ0ODUyNQ==,19328961,henry501,2021-01-12T06:55:08Z,2021-01-12T06:55:08Z,NONE,"Great, really happy I could help! Reverse proxies get tricky.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-758668359,https://api.github.com/repos/simonw/datasette/issues/1091,758668359,MDEyOklzc3VlQ29tbWVudDc1ODY2ODM1OQ==,6739646,tballison,2021-01-12T13:52:29Z,2021-01-12T13:52:29Z,NONE,"Y, thank you to both of you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1094#issuecomment-731260091,https://api.github.com/repos/simonw/datasette/issues/1094,731260091,MDEyOklzc3VlQ29tbWVudDczMTI2MDA5MQ==,4808085,bapowell,2020-11-20T16:11:29Z,2020-11-20T16:11:29Z,NONE,"I can confirm this issue, running version 0.51.1 under Windows.

Fixed by commenting out the following line near the top of datasette\utils\asgi.py :

`#from os import EX_CANTCREAT`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743011397,import EX_CANTCREAT means datasette fails to work on Windows,
https://github.com/simonw/datasette/issues/1096#issuecomment-732542285,https://api.github.com/repos/simonw/datasette/issues/1096,732542285,MDEyOklzc3VlQ29tbWVudDczMjU0MjI4NQ==,9599,simonw,2020-11-24T02:16:22Z,2020-11-24T02:16:22Z,OWNER,"I'd like to implement this by first extending the `register_output_renderer()` hook to support streaming huge responses, then switching CSV to use the plugin hook in addition to TSV using it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743359646,TSV should be a default export option,
https://github.com/simonw/datasette/issues/1098#issuecomment-727655636,https://api.github.com/repos/simonw/datasette/issues/1098,727655636,MDEyOklzc3VlQ29tbWVudDcyNzY1NTYzNg==,9599,simonw,2020-11-15T23:22:27Z,2020-11-15T23:22:27Z,OWNER,"Need to replicate this in the fixtures, then fix it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1098#issuecomment-727656208,https://api.github.com/repos/simonw/datasette/issues/1098,727656208,MDEyOklzc3VlQ29tbWVudDcyNzY1NjIwOA==,9599,simonw,2020-11-15T23:26:14Z,2020-11-15T23:26:14Z,OWNER,"Schema for that broken example:
```sql
CREATE TABLE generators_eia860 (
	id INTEGER NOT NULL, 
	plant_id_eia INTEGER, 
	generator_id TEXT, 
	report_date DATE, 
	operational_status_code TEXT, 
	operational_status TEXT, 
	ownership_code TEXT, 
	utility_id_eia INTEGER, 
	capacity_mw FLOAT, 
	summer_capacity_mw FLOAT, 
	winter_capacity_mw FLOAT, 
	energy_source_code_1 TEXT, 
	energy_source_code_2 TEXT, 
	energy_source_code_3 TEXT, 
	energy_source_code_4 TEXT, 
	energy_source_code_5 TEXT, 
	energy_source_code_6 TEXT, 
	fuel_type_code_pudl TEXT, 
	multiple_fuels BOOLEAN, 
	deliver_power_transgrid BOOLEAN, 
	syncronized_transmission_grid BOOLEAN, 
	turbines_num INTEGER, 
	planned_modifications BOOLEAN, 
	planned_net_summer_capacity_uprate_mw FLOAT, 
	planned_net_winter_capacity_uprate_mw FLOAT, 
	planned_uprate_date DATE, 
	planned_net_summer_capacity_derate_mw FLOAT, 
	planned_net_winter_capacity_derate_mw FLOAT, 
	planned_derate_date DATE, 
	planned_new_prime_mover_code TEXT, 
	planned_energy_source_code_1 TEXT, 
	planned_repower_date DATE, 
	other_planned_modifications BOOLEAN, 
	other_modifications_date DATE, 
	planned_retirement_date DATE, 
	carbon_capture BOOLEAN, 
	startup_source_code_1 TEXT, 
	startup_source_code_2 TEXT, 
	startup_source_code_3 TEXT, 
	startup_source_code_4 TEXT, 
	technology_description TEXT, 
	turbines_inverters_hydrokinetics TEXT, 
	time_cold_shutdown_full_load_code TEXT, 
	planned_new_capacity_mw FLOAT, 
	cofire_fuels BOOLEAN, 
	switch_oil_gas BOOLEAN, 
	nameplate_power_factor FLOAT, 
	minimum_load_mw FLOAT, 
	uprate_derate_during_year BOOLEAN, 
	uprate_derate_completed_date DATE, 
	current_planned_operating_date DATE, 
	summer_estimated_capability_mw FLOAT, 
	winter_estimated_capability_mw FLOAT, 
	retirement_date DATE, 
	PRIMARY KEY (id), 
	FOREIGN KEY(plant_id_eia, generator_id) REFERENCES generators_entity_eia (plant_id_eia, generator_id), 
	FOREIGN KEY(utility_id_eia) REFERENCES utilities_entity_eia (utility_id_eia), 
	CHECK (multiple_fuels IN (0, 1)), 
	CHECK (deliver_power_transgrid IN (0, 1)), 
	CHECK (syncronized_transmission_grid IN (0, 1)), 
	CHECK (planned_modifications IN (0, 1)), 
	CHECK (other_planned_modifications IN (0, 1)), 
	CHECK (carbon_capture IN (0, 1)), 
	CHECK (cofire_fuels IN (0, 1)), 
	CHECK (switch_oil_gas IN (0, 1)), 
	CHECK (uprate_derate_during_year IN (0, 1))
);
```
https://pudl-datasette-xl7xwcpe2a-uc.a.run.app/pudl/generators_entity_eia is:
```sql
CREATE TABLE generators_entity_eia (
	plant_id_eia INTEGER NOT NULL, 
	generator_id TEXT NOT NULL, 
	prime_mover_code TEXT, 
	duct_burners BOOLEAN, 
	operating_date DATE, 
	topping_bottoming_code TEXT, 
	solid_fuel_gasification BOOLEAN, 
	pulverized_coal_tech BOOLEAN, 
	fluidized_bed_tech BOOLEAN, 
	subcritical_tech BOOLEAN, 
	supercritical_tech BOOLEAN, 
	ultrasupercritical_tech BOOLEAN, 
	stoker_tech BOOLEAN, 
	other_combustion_tech BOOLEAN, 
	bypass_heat_recovery BOOLEAN, 
	rto_iso_lmp_node_id TEXT, 
	rto_iso_location_wholesale_reporting_id TEXT, 
	associated_combined_heat_power BOOLEAN, 
	original_planned_operating_date DATE, 
	operating_switch TEXT, 
	previously_canceled BOOLEAN, 
	PRIMARY KEY (plant_id_eia, generator_id), 
	FOREIGN KEY(plant_id_eia) REFERENCES plants_entity_eia (plant_id_eia), 
	CHECK (duct_burners IN (0, 1)), 
	CHECK (solid_fuel_gasification IN (0, 1)), 
	CHECK (pulverized_coal_tech IN (0, 1)), 
	CHECK (fluidized_bed_tech IN (0, 1)), 
	CHECK (subcritical_tech IN (0, 1)), 
	CHECK (supercritical_tech IN (0, 1)), 
	CHECK (ultrasupercritical_tech IN (0, 1)), 
	CHECK (stoker_tech IN (0, 1)), 
	CHECK (other_combustion_tech IN (0, 1)), 
	CHECK (bypass_heat_recovery IN (0, 1)), 
	CHECK (associated_combined_heat_power IN (0, 1)), 
	CHECK (previously_canceled IN (0, 1))
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1098#issuecomment-735442226,https://api.github.com/repos/simonw/datasette/issues/1098,735442226,MDEyOklzc3VlQ29tbWVudDczNTQ0MjIyNg==,9599,simonw,2020-11-29T19:28:04Z,2020-11-29T19:28:04Z,OWNER,"For the moment I'm going to solve this by teaching Datasette's internal introspection methods - in particular these ones - to ignore compound foreign keys entirely:

https://github.com/simonw/datasette/blob/e800ffcf7cc6a915eb554b369c654f87162575e5/datasette/utils/__init__.py#L470-L505","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1098#issuecomment-735443654,https://api.github.com/repos/simonw/datasette/issues/1098,735443654,MDEyOklzc3VlQ29tbWVudDczNTQ0MzY1NA==,9599,simonw,2020-11-29T19:41:01Z,2020-11-29T19:41:01Z,OWNER,Fix is out in 0.52.1: https://docs.datasette.io/en/latest/changelog.html#v0-52-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1098#issuecomment-735444698,https://api.github.com/repos/simonw/datasette/issues/1098,735444698,MDEyOklzc3VlQ29tbWVudDczNTQ0NDY5OA==,9599,simonw,2020-11-29T19:50:14Z,2020-11-29T19:50:14Z,OWNER,Demo of the fix: https://latest.datasette.io/fixtures/foreign_key_references (the compound foreign key columns do not link to anything),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1099#issuecomment-1402563930,https://api.github.com/repos/simonw/datasette/issues/1099,1402563930,IC_kwDOBm6k_c5TmW1a,536941,fgregg,2023-01-24T20:11:11Z,2023-01-24T20:11:11Z,CONTRIBUTOR,"hi @simonw, this bug bit me today.

the UX for linking from a table to the foreign key seems tough! 

the design in the other direction seems a lot easier, for a given primary key detail page, add links back to the tables that refer to the row.

would you be open to a PR that solved the second problem but not the first?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/1099#issuecomment-1402898291,https://api.github.com/repos/simonw/datasette/issues/1099,1402898291,IC_kwDOBm6k_c5Tnodz,536941,fgregg,2023-01-25T00:55:06Z,2023-01-25T00:55:06Z,CONTRIBUTOR,"I went ahead and spiked something together, in #2003 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/1099#issuecomment-1402900354,https://api.github.com/repos/simonw/datasette/issues/1099,1402900354,IC_kwDOBm6k_c5Tno-C,536941,fgregg,2023-01-25T00:58:26Z,2023-01-25T00:58:26Z,CONTRIBUTOR,"> My original idea for compound foreign keys was to turn both of those columns into links, but that doesn't fit here because `database_name` is already part of a different foreign key.

it's pretty hard to know what the right thing to do is if a field is part of multiple foreign keys. 

but, if that's not the case, what about making each of the columns a link. seems like an improvement over the status quo.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/1099#issuecomment-735444858,https://api.github.com/repos/simonw/datasette/issues/1099,735444858,MDEyOklzc3VlQ29tbWVudDczNTQ0NDg1OA==,9599,simonw,2020-11-29T19:51:58Z,2020-11-29T19:51:58Z,OWNER,"My fix in deb0be4ae56f191f121239b29e83dd53b62d6305 for #1098 was to have Datasette deliberately pretend that compound foreign keys don't exist: https://github.com/simonw/datasette/blob/deb0be4ae56f191f121239b29e83dd53b62d6305/datasette/utils/__init__.py#L470-L495

This workaround will need to be rethought to implement real support for them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/1099#issuecomment-749771231,https://api.github.com/repos/simonw/datasette/issues/1099,749771231,MDEyOklzc3VlQ29tbWVudDc0OTc3MTIzMQ==,9599,simonw,2020-12-22T20:54:25Z,2020-12-22T20:54:25Z,OWNER,"https://latest.datasette.io/_internal/foreign_keys (use https://latest.datasette.io/login-as-root first) is now a compound foreign key table:
```sql
CREATE TABLE foreign_keys (
        ""database_name"" TEXT,
        ""table_name"" TEXT,
        ""id"" INTEGER,
        ""seq"" INTEGER,
        ""table"" TEXT,
        ""from"" TEXT,
        ""to"" TEXT,
        ""on_update"" TEXT,
        ""on_delete"" TEXT,
        ""match"" TEXT,
        PRIMARY KEY (database_name, table_name, id, seq),
        FOREIGN KEY (database_name) REFERENCES databases(database_name),
        FOREIGN KEY (database_name, table_name) REFERENCES tables(database_name, table_name)
    );
```
Currently the `database_name` column becomes a link (because it's a single foreign key) but the `table_name` one remains a non-link:

<img width=""1079"" alt=""_internal__foreign_keys__24_rows"" src=""https://user-images.githubusercontent.com/9599/102932110-87ba3080-4454-11eb-9ad5-b70a65129588.png"">

My original idea for compound foreign keys was to turn both of those columns into links, but that doesn't fit here because `database_name` is already part of a different foreign key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/1099#issuecomment-749845797,https://api.github.com/repos/simonw/datasette/issues/1099,749845797,MDEyOklzc3VlQ29tbWVudDc0OTg0NTc5Nw==,9599,simonw,2020-12-23T00:13:29Z,2020-12-23T00:14:25Z,OWNER,"Also need to solve displaying these links in the opposite direction:

https://latest.datasette.io/_internal/tables/fixtures,facet_cities

<img width=""617"" alt=""_internal__tables"" src=""https://user-images.githubusercontent.com/9599/102944742-84cd3900-4470-11eb-95a0-bfaf00b17e82.png"">

That page should link to lists of records in columns, foreign_keys and indexes - like this example: https://latest.datasette.io/fixtures/roadside_attractions/1

<img width=""712"" alt=""fixtures__roadside_attractions"" src=""https://user-images.githubusercontent.com/9599/102944816-c1009980-4470-11eb-9dbd-50d0ba4fd137.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/11#issuecomment-338530704,https://api.github.com/repos/simonw/datasette/issues/11,338530704,MDEyOklzc3VlQ29tbWVudDMzODUzMDcwNA==,9599,simonw,2017-10-23T02:18:36Z,2017-10-23T02:18:36Z,OWNER,Needed by https://github.com/simonw/stateless-datasets/issues/4#issuecomment-338530389,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267522549,Code that generates compile-time properties about the database ,
https://github.com/simonw/datasette/issues/110#issuecomment-344988263,https://api.github.com/repos/simonw/datasette/issues/110,344988263,MDEyOklzc3VlQ29tbWVudDM0NDk4ODI2Mw==,9599,simonw,2017-11-16T16:58:48Z,2017-11-16T16:58:48Z,OWNER,"Here's how I tested this.

First I downloaded and started a docker container using https://hub.docker.com/r/prolocutor/python3-sqlite-ext - which includes the compiled spatialite extension. This downloads it, then starts a shell in that container.

    docker run -it -p 8018:8018 prolocutor/python3-sqlite-ext:3.5.1-spatialite /bin/sh

Installed a pre-release build of datasette which includes the new `--load-extension` option.

    pip install https://static.simonwillison.net/static/2017/datasette-0.13-py3-none-any.whl

Now grab a sample database from https://www.gaia-gis.it/spatialite-2.3.1/resources.html - and unzip and rename it (datasette doesn't yet like databases with dots in their filename):

    wget http://www.gaia-gis.it/spatialite-2.3.1/test-2.3.sqlite.gz
    gunzip test-2.3.sqlite.gz
    mv test-2.3.sqlite test23.sqlite

Now start datasette on port 8018 (the port I exposed earlier) with the extension loaded:

    datasette test23.sqlite  -p 8018 -h 0.0.0.0 --load-extension /usr/local/lib/mod_spatialite.so

Now I can confirm that it worked:

http://localhost:8018/test23-c88bc35?sql=select+ST_AsText%28Geometry%29+from+HighWays+limit+1

<img width=""743"" alt=""test23"" src=""https://user-images.githubusercontent.com/9599/32904449-39789a3a-caac-11e7-9531-b49f06051e34.png"">

If I run datasette without `--load-extension` I get this:

    datasette test23.sqlite  -p 8018 -h 0.0.0.0

<img width=""747"" alt=""test23_and_turn_on_auto-escaping_in_jinja_ _simonw_datasette_82261a6"" src=""https://user-images.githubusercontent.com/9599/32904508-54e53d78-caac-11e7-9f80-4d96e9f9fb5f.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274578142,Add --load-extension option to datasette for loading extra SQLite extensions,