id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,pull_request,body,repo,type,active_lock_reason,performed_via_github_app,reactions,draft,state_reason
504720731,MDU6SXNzdWU1MDQ3MjA3MzE=,1,Add more details on how to request data from google takeout correctly.,1055831,open,0,,,0,2019-10-09T15:17:34Z,2019-10-09T15:17:34Z,,NONE,,"The default is to download everything.  This can result in an enormous amount of data when you only really need 2 types of data for now:

- My Activity
- Location History

In addition unless you specify that ""My Activity"" is downloaded in JSON format the default is HTML.  This then causes the 

`google-takeout-to-sqlite my-activity takeout.db takeout.zip`

command to fail as it only contains html files not json files.

Thanks",206649770,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
769376447,MDU6SXNzdWU3NjkzNzY0NDc=,2,killed by oomkiller on large location-history,231498,open,0,,,2,2020-12-17T00:32:24Z,2020-12-17T00:48:32Z,,NONE,,"memory seems to grow unbounded and is oom-killed after about 20GB memory usage.

this is happening while loading a ~1GB uncompressed location history.",206649770,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
769397742,MDU6SXNzdWU3NjkzOTc3NDI=,3,sqlite-utils error on takeout import,231498,open,0,,,0,2020-12-17T01:18:48Z,2020-12-17T01:19:04Z,,NONE,,"```
$ google-takeout-to-sqlite my-activity takeout.db /path/to/zip
...
sqlite3.OperationalError: no such table: main.my_activity
```

there is no table create in `utils.py`, unlike other importers such as github-to-sqlite

additionally, this package and hackernews-to-sqlite have conflicting `sqlite-utils` dep with datasette and dogsheep-beta",206649770,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/3/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
778380836,MDU6SXNzdWU3NzgzODA4MzY=,4,Feature Request: Gmail,203343,open,0,,,5,2021-01-04T21:31:09Z,2021-03-04T20:54:44Z,,NONE,,"From takeout, I only exported my Gmail account.  Ideally I could parse this into sqlite via this tool.",206649770,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
813880401,MDExOlB1bGxSZXF1ZXN0NTc3OTUzNzI3,5,WIP: Add Gmail takeout mbox import,306240,open,0,,,25,2021-02-22T21:30:40Z,2021-07-28T07:18:56Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/google-takeout-to-sqlite/pulls/5,"WIP

This PR adds the ability to import emails from a Gmail mbox export from Google Takeout.

This is my first PR to a datasette/dogsheep repo. I've tested this on my personal Google Takeout mbox with ~520,000 emails going back to 2004. This took around ~20 minutes to process.

To provide some feedback on the progress of the import I added the ""rich"" python module. I'm happy to remove that if adding a dependency is discouraged. However, I think it makes a nice addition to give feedback on the progress of a long import.

Do we want to log emails that have errors when trying to import them?

Dealing with encodings with emails is a bit tricky. I'm very open to feedback on how to deal with those better. As well as any other feedback for improvements.",206649770,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
821841046,MDU6SXNzdWU4MjE4NDEwNDY=,6,Upgrade to latest sqlite-utils,9599,open,0,,,1,2021-03-04T07:21:54Z,2021-03-04T07:22:51Z,,MEMBER,,This is pinned to v1 at the moment.,206649770,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
930946817,MDU6SXNzdWU5MzA5NDY4MTc=,7,KeyError: 'accuracy' when processing Location History,403152,open,0,,,0,2021-06-27T14:39:43Z,2021-06-27T14:39:43Z,,NONE,,"I'm new to both the dogsheep tools and datasette but have been experimenting a bit the last few days and these are really cool tools!

I encountered a problem running my Google location history through this tool running the latest release in a docker container:

```
Traceback (most recent call last):
  File ""/usr/local/bin/google-takeout-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/cli.py"", line 49, in my_activity
    utils.save_location_history(db, zf)
  File ""/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py"", line 27, in save_location_history
    db[""location_history""].upsert_all(
  File ""/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1105, in upsert_all
    return self.insert_all(
  File ""/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py"", line 990, in insert_all
    chunk = list(chunk)
  File ""/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py"", line 33, in <genexpr>
    ""accuracy"": row[""accuracy""],
KeyError: 'accuracy'
```

It looks like the tool assumes the `accuracy` key will be in every location history entry. 

My first attempt at a local patch to get myself going was to convert accessing the `accuracy` key to a `.get` instead to hopefully make the row nullable but I wasn't quite sure what `sqlite_utils` would do there. That did work in that the import happened and so I was going to propose a patch that made that change but in updating the existing test to include an entry with a missing accuracy entry, I noticed the expected type of the field appeared to be changing to a string in the test (and from a quick scan through the sqlite_utils code, probably TEXT in the database). Given this change in column type, it seemed that opening an issue first before proposing a fix seemed warranted. It seems the schema would need to be explicitly specified if you wanted a nullable integer column.

Now that I've done a successful import run using my initial fix of calling `.get` on the row dict, I can see with datasette that I only have 7 data points (out of ~250k) that have a null accuracy column. They are all from 2011-2012 in an import that includes points spanning ~2010-2016 so perhaps another approach might be to filter those entries out during import if it really is that infrequent?

I'm happy to provide a PR for a fix but figured I'd ask about which direction is preferred first.",206649770,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/7/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
954546309,MDExOlB1bGxSZXF1ZXN0Njk4NDIzNjY3,8,Add Gmail takeout mbox import (v2),28565,open,0,,,4,2021-07-28T07:05:32Z,2021-12-31T19:06:20Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/google-takeout-to-sqlite/pulls/8,"WIP

This PR builds on #5 to continue implementing gmail import support.

Building on @UtahDave's work, these commits add a few performance and bug fixes:

* Decreased memory overhead for import by manually parsing mbox headers.
* Fixed error where some messages in the mbox would yield a row with NULL in all columns.

I will send more commits to fix any errors I encounter as I run the importer on my personal takeout data.",206649770,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1046887492,PR_kwDODFE5qs4uMsMJ,9,Removed space from filename My Activity.json,91880982,open,0,,,0,2021-11-08T00:04:31Z,2021-11-08T00:04:31Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/google-takeout-to-sqlite/pulls/9,"File name from google takeout has no space. The code only runs without error if filename is ""MyActivity.json"" and not ""My Activity.json"". Is it a new change by Google?",206649770,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/9/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1123393829,I_kwDODFE5qs5C9aEl,10,sqlite3.OperationalError: no such table: main.my_activity,69208826,open,0,,,1,2022-02-03T17:59:29Z,2022-03-20T02:38:07Z,,NONE,,"Hello,
When i run the command `google-takeout-to-sqlite my-activity db.db takeout-20220203T174446Z-001.zip`, i get this error :
```
Traceback (most recent call last):
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\runpy.py"", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\runpy.py"", line 87, in _run_code
    exec(code, run_globals)
  File ""C:\Users\julie\AppData\Local\Programs\Python\Python39-32\Scripts\google-takeout-to-sqlite.exe\__main__.py"", line 7, in <module>
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py"", line 1128, in __call__
    return self.main(*args, **kwargs)
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py"", line 1053, in main
    rv = self.invoke(ctx)
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py"", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py"", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py"", line 754, in invoke
    return __callback(*args, **kwargs)
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\google_takeout_to_sqlite\cli.py"", line 31, in my_activity
    utils.save_my_activity(db, zf)
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\google_takeout_to_sqlite\utils.py"", line 19, in save_my_activity
    db[""my_activity""].create_index([""time""])
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\sqlite_utils\db.py"", line 629, in create_index
    self.db.conn.execute(sql)
sqlite3.OperationalError: no such table: main.my_activity
```
Thank you for your help
Sorry for my bad English
EDIT: i used the json format",206649770,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/10/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1250287607,PR_kwDODFE5qs44jvRV,11,Update README.md,11887,open,0,,,0,2022-05-27T03:13:59Z,2022-05-27T03:13:59Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/google-takeout-to-sqlite/pulls/11,Fix typo,206649770,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/11/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1557599877,I_kwDODFE5qs5c1xaF,12,location history changes,14809320,open,0,,,0,2023-01-26T03:57:25Z,2023-01-26T03:57:25Z,,NONE,,"not sure if each download is unique, but I had to change some things to work with the takeout zip I made 2023-01-25

filename changed from ""Location History.json"" to ""Records.json""

`""timestampMs""` is not present, `""timestamp""` is roughly iso timestamp

```py
def get_timestamp_ms(raw_timestamp):
    try:
        return datetime.datetime.strptime(raw_timestamp, ""%Y-%m-%dT%H:%M:%SZ"").timestamp()
    except ValueError:
        return datetime.datetime.strptime(raw_timestamp, ""%Y-%m-%dT%H:%M:%S.%fZ"").timestamp()

def save_location_history(db, zf):
    location_history = json.load(
        zf.open(""Takeout/Location History/Records.json"")
    )
    db[""location_history""].upsert_all(
        (
            {
                ""id"": id_for_location_history(row),
                ""latitude"": row[""latitudeE7""] / 1e7,
                ""longitude"": row[""longitudeE7""] / 1e7,
                ""accuracy"": row[""accuracy""],
                ""timestampMs"": get_timestamp_ms(row[""timestamp""]),
                ""when"": row[""timestamp""],
            }
            for row in location_history[""locations""]
        ),
        pk=""id"",
    )


def id_for_location_history(row):
    # We want an ID that is unique but can be sorted by in
    # date order - so we use the isoformat date + the first
    # 6 characters of a hash of the JSON
    first_six = hashlib.sha1(
        json.dumps(row, separators=("","", "":""), sort_keys=True).encode(""utf8"")
    ).hexdigest()[:6]
    return ""{}-{}"".format(
        row['timestamp'],
        first_six,
    )
```

example locations from mine

```json
{
    ""latitudeE7"": 427220206,
    ""longitudeE7"": -923423972,
    ""accuracy"": 10,
    ""deviceTag"": -1312429967,
    ""deviceDesignation"": ""PRIMARY"",
    ""timestamp"": ""2019-01-08T23:31:50.867Z""
  }
```

```json
{
    ""latitudeE7"": 427011317,
    ""longitudeE7"": -923448300,
    ""accuracy"": 5,
    ""deviceTag"": -1312429967,
    ""deviceDesignation"": ""PRIMARY"",
    ""timestamp"": ""2019-01-08T23:33:53Z""
  }, 
```",206649770,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/12/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",,