html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408370,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,624408370,MDEyOklzc3VlQ29tbWVudDYyNDQwODM3MA==,9599,simonw,2020-05-06T02:19:27Z,2020-05-06T02:19:27Z,MEMBER,"The plugin can be generalized: it can be configured to know how to take the URL path, look it up in ANY table (via a custom SQL query) to get a path on disk and then serve that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk, https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408738,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,624408738,MDEyOklzc3VlQ29tbWVudDYyNDQwODczOA==,9599,simonw,2020-05-06T02:21:05Z,2020-05-06T02:21:32Z,MEMBER,"Here's rendering code from my hacked-together not-yet-released S3 image proxy: ```python from starlette.responses import Response from PIL import Image, ExifTags import pyheif for ORIENTATION_TAG in ExifTags.TAGS.keys(): if ExifTags.TAGS[ORIENTATION_TAG] == ""Orientation"": break ... # Load it into Pillow if ext == ""heic"": heic = pyheif.read_heif(image_response.content) image = Image.frombytes(mode=heic.mode, size=heic.size, data=heic.data) else: image = Image.open(io.BytesIO(image_response.content)) # Does EXIF tell us to rotate it? try: exif = dict(image._getexif().items()) if exif[ORIENTATION_TAG] == 3: image = image.rotate(180, expand=True) elif exif[ORIENTATION_TAG] == 6: image = image.rotate(270, expand=True) elif exif[ORIENTATION_TAG] == 8: image = image.rotate(90, expand=True) except (AttributeError, KeyError, IndexError): pass # Resize based on ?w= and ?h=, if set width, height = image.size w = request.query_params.get(""w"") h = request.query_params.get(""h"") if w is not None or h is not None: if h is None: # Set h based on w w = int(w) h = int((float(height) / width) * w) elif w is None: h = int(h) # Set w based on h w = int((float(width) / height) * h) w = int(w) h = int(h) image.thumbnail((w, h)) # ?bw= converts to black and white if request.query_params.get(""bw""): image = image.convert(""L"") # ?q= sets the quality - defaults to 75 quality = 75 q = request.query_params.get(""q"") if q and q.isdigit() and 1 <= int(q) <= 100: quality = int(q) # Output as JPEG or PNG output_image = io.BytesIO() image_type = ""JPEG"" kwargs = {""quality"": quality} if image.format == ""PNG"": image_type = ""PNG"" kwargs = {} image.save(output_image, image_type, **kwargs) return Response( output_image.getvalue(), media_type=""image/jpeg"", headers={""cache-control"": ""s-maxage={}, public"".format(365 * 24 * 60 * 60)}, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk, https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-625947133,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,625947133,MDEyOklzc3VlQ29tbWVudDYyNTk0NzEzMw==,9599,simonw,2020-05-08T18:13:06Z,2020-05-08T18:13:06Z,MEMBER,`datasette-media` will be able to handle this once I implement https://github.com/simonw/datasette-media/issues/3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk, https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633626741,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633626741,MDEyOklzc3VlQ29tbWVudDYzMzYyNjc0MQ==,9599,simonw,2020-05-25T15:38:55Z,2020-05-25T15:38:55Z,MEMBER,"Sure, I should absolutely document this!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk, https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633629944,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633629944,MDEyOklzc3VlQ29tbWVudDYzMzYyOTk0NA==,9599,simonw,2020-05-25T15:47:42Z,2020-05-25T15:47:42Z,MEMBER,"I'll add a proper section to the README, but for the moment here's how I do this. First, install `datasette` and the `datasette-media` plugin. Create a `metadata.yaml` file with the following content: ```yaml plugins: datasette-media: photo: sql: |- select path as filepath, 200 as resize_height from apple_photos where uuid = :key photo-big: sql: |- select path as filepath, 1024 as resize_height from apple_photos where uuid = :key ``` Now run `datasette -m metadata.yaml photos.db` - thumbnails will be served at http://127.0.0.1:8001/-/media/photo/F4469918-13F3-43D8-9EC1-734C0E6B60AD and larger sizes of the image at http://127.0.0.1:8001/-/media/photo-big/A8B02C7D-365E-448B-9510-69F80C26304D I also made myself two custom pages, one showing recent images and one showing random images. To do this, install the `datasette-template-sql` plugin and then create a `templates/pages` directory and add these files: `recent-photos.html` ```html
TemporaryRedirect
I wish ..."",
""points"": null,
""parent_id"": 27941108,
""story_id"": 27941108
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952189173,Use HN algolia endpoint to retrieve trees,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886241674,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3,886241674,IC_kwDODtX3eM400vmK,9599,simonw,2021-07-25T18:41:17Z,2021-07-25T18:41:17Z,MEMBER,Got a TIL out of this: https://til.simonwillison.net/jq/extracting-objects-recursively,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952189173,Use HN algolia endpoint to retrieve trees,
https://github.com/dogsheep/healthkit-to-sqlite/issues/1#issuecomment-513437463,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/1,513437463,MDEyOklzc3VlQ29tbWVudDUxMzQzNzQ2Mw==,9599,simonw,2019-07-20T05:19:59Z,2019-07-20T05:19:59Z,MEMBER,"I ran xml_analyser against the XML HealthKit `export.xml` file and got the following results:
```python
{
'ActivitySummary': {'attr_counts': {'activeEnergyBurned': 980,
'activeEnergyBurnedGoal': 980,
'activeEnergyBurnedUnit': 980,
'appleExerciseTime': 980,
'appleExerciseTimeGoal': 980,
'appleStandHours': 980,
'appleStandHoursGoal': 980,
'dateComponents': 980},
'child_counts': {},
'count': 980,
'parent_counts': {'HealthData': 980}},
'Correlation': {'attr_counts': {'creationDate': 1,
'endDate': 1,
'sourceName': 1,
'sourceVersion': 1,
'startDate': 1,
'type': 1},
'child_counts': {'MetadataEntry': 1, 'Record': 2},
'count': 1,
'parent_counts': {'HealthData': 1}},
'ExportDate': {'attr_counts': {'value': 1},
'child_counts': {},
'count': 1,
'parent_counts': {'HealthData': 1}},
'HealthData': {'attr_counts': {'locale': 1},
'child_counts': {'ActivitySummary': 980,
'Correlation': 1,
'ExportDate': 1,
'Me': 1,
'Record': 2672231,
'Workout': 663},
'count': 1,
'parent_counts': {}},
'HeartRateVariabilityMetadataList': {'attr_counts': {},
'child_counts': {'InstantaneousBeatsPerMinute': 93653},
'count': 2318,
'parent_counts': {'Record': 2318}},
'InstantaneousBeatsPerMinute': {'attr_counts': {'bpm': 93653, 'time': 93653},
'child_counts': {},
'count': 93653,
'parent_counts': {'HeartRateVariabilityMetadataList': 93653}},
'Location': {'attr_counts': {'altitude': 398683,
'course': 398683,
'date': 398683,
'horizontalAccuracy': 398683,
'latitude': 398683,
'longitude': 398683,
'speed': 398683,
'verticalAccuracy': 398683},
'child_counts': {},
'count': 398683,
'parent_counts': {'WorkoutRoute': 398683}},
'Me': {'attr_counts': {'HKCharacteristicTypeIdentifierBiologicalSex': 1,
'HKCharacteristicTypeIdentifierBloodType': 1,
'HKCharacteristicTypeIdentifierDateOfBirth': 1,
'HKCharacteristicTypeIdentifierFitzpatrickSkinType': 1},
'child_counts': {},
'count': 1,
'parent_counts': {'HealthData': 1}},
'MetadataEntry': {'attr_counts': {'key': 290449, 'value': 290449},
'child_counts': {},
'count': 290449,
'parent_counts': {'Correlation': 1,
'Record': 287974,
'Workout': 1928,
'WorkoutRoute': 546}},
'Record': {'attr_counts': {'creationDate': 2672233,
'device': 2665111,
'endDate': 2672233,
'sourceName': 2672233,
'sourceVersion': 2671779,
'startDate': 2672233,
'type': 2672233,
'unit': 2650012,
'value': 2672232},
'child_counts': {'HeartRateVariabilityMetadataList': 2318,
'MetadataEntry': 287974},
'count': 2672233,
'parent_counts': {'Correlation': 2, 'HealthData': 2672231}},
'Workout': {'attr_counts': {'creationDate': 663,
'device': 230,
'duration': 663,
'durationUnit': 663,
'endDate': 663,
'sourceName': 663,
'sourceVersion': 663,
'startDate': 663,
'totalDistance': 663,
'totalDistanceUnit': 663,
'totalEnergyBurned': 663,
'totalEnergyBurnedUnit': 663,
'workoutActivityType': 663},
'child_counts': {'MetadataEntry': 1928,
'WorkoutEvent': 2094,
'WorkoutRoute': 340},
'count': 663,
'parent_counts': {'HealthData': 663}},
'WorkoutEvent': {'attr_counts': {'date': 2094,
'duration': 837,
'durationUnit': 837,
'type': 2094},
'child_counts': {},
'count': 2094,
'parent_counts': {'Workout': 2094}},
'WorkoutRoute': {'attr_counts': {'creationDate': 340,
'endDate': 340,
'sourceName': 340,
'sourceVersion': 340,
'startDate': 340},
'child_counts': {'Location': 398683, 'MetadataEntry': 546},
'count': 340,
'parent_counts': {'Workout': 340}}}
```
The most interesting bit is this:
```python
'HealthData': {'attr_counts': {'locale': 1},
'child_counts': {'ActivitySummary': 980,
'Correlation': 1,
'ExportDate': 1,
'Me': 1,
'Record': 2672231,
'Workout': 663},
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470637068,Use XML Analyser to figure out the structure of the export XML,
https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550783316,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550783316,MDEyOklzc3VlQ29tbWVudDU1MDc4MzMxNg==,9599,simonw,2019-11-07T05:16:56Z,2019-11-07T05:34:29Z,MEMBER,"It looks like Apple changed the location of these in iOS 13 - they are now in separate `.gpx` files:
![2FF70E95-CDEE-4241-A5C5-EE95A862E519](https://user-images.githubusercontent.com/9599/68362042-be12e000-00da-11ea-8925-7397410332d8.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points,
https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550806302,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550806302,MDEyOklzc3VlQ29tbWVudDU1MDgwNjMwMg==,9599,simonw,2019-11-07T05:33:31Z,2019-11-07T05:33:31Z,MEMBER,"The XML now includes references to these new files:
![CBBA54FC-51FB-4BB3-927C-C2CA99237B04](https://user-images.githubusercontent.com/9599/68362716-121ec400-00dd-11ea-9846-387c7cd64c8b.jpeg)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points,
https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550824838,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550824838,MDEyOklzc3VlQ29tbWVudDU1MDgyNDgzOA==,9599,simonw,2019-11-07T05:47:07Z,2019-11-07T05:47:07Z,MEMBER,"Relevant code:
https://github.com/dogsheep/healthkit-to-sqlite/blob/d16f45f06fbae6ec8a78cc9ca7b5b7db0413f139/healthkit_to_sqlite/utils.py#L58-L64","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points,
https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550828084,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550828084,MDEyOklzc3VlQ29tbWVudDU1MDgyODA4NA==,9599,simonw,2019-11-07T05:49:24Z,2019-11-07T05:49:24Z,MEMBER,"So the fix there is going to be to detect the new `FileReference` element and load the corresponding points data from it.
This will be a little tricky because that function will need access to the zip file.
It probably won't work at all for the mode where the `export.xml` file is passed directly using the `--xml` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711074031,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711074031,MDEyOklzc3VlQ29tbWVudDcxMTA3NDAzMQ==,9599,simonw,2020-10-17T20:14:01Z,2020-10-17T20:14:01Z,MEMBER,I'd be happy to teach the tool to look for `export.xml` or `eksport.xml` - and then expand that list to other languages.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711074306,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711074306,MDEyOklzc3VlQ29tbWVudDcxMTA3NDMwNg==,9599,simonw,2020-10-17T20:16:22Z,2020-10-17T20:16:22Z,MEMBER,"The ""first XML file in the root"" solution is probably easier though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711078917,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711078917,MDEyOklzc3VlQ29tbWVudDcxMTA3ODkxNw==,9599,simonw,2020-10-17T20:51:55Z,2020-10-17T20:52:03Z,MEMBER,"I switched my phone to Spanish and ran an export - I got a file called `exportar.zip`. Unzipped I still got a `apple_
health_export` folder but the root contained:
```
electrocardiograms/
export_cda.xml
exportar.xml
workout-routes/
```
It looks like `export_cda.xml` does not have a translated name, so maybe I can ignore it and look for the _other_ `.xml` file in that directory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711079056,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711079056,MDEyOklzc3VlQ29tbWVudDcxMTA3OTA1Ng==,9599,simonw,2020-10-17T20:53:00Z,2020-10-17T20:53:00Z,MEMBER,"I think the safest thing is to sniff the first few lines of the file. Those should be the same no matter the language that was used:
```xml
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470856782,Break up records into different tables for each type,
https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514496725,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7,514496725,MDEyOklzc3VlQ29tbWVudDUxNDQ5NjcyNQ==,9599,simonw,2019-07-24T06:20:59Z,2019-07-24T06:20:59Z,MEMBER,"I'm using https://pypi.org/project/memory-profiler/ to explore this in more detail:
```
$ pip install memory-profiler matplotlib
```
Then:
```
$ mprof run healthkit-to-sqlite ~/Downloads/healthkit-export.zip healthkit.db
$ mprof plot
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472097220,Script uses a lot of RAM,
https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514498221,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7,514498221,MDEyOklzc3VlQ29tbWVudDUxNDQ5ODIyMQ==,9599,simonw,2019-07-24T06:26:49Z,2019-07-24T06:26:49Z,MEMBER,"Adding `el.clear()` got me a huge improvement:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472097220,Script uses a lot of RAM,
https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514500253,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7,514500253,MDEyOklzc3VlQ29tbWVudDUxNDUwMDI1Mw==,9599,simonw,2019-07-24T06:34:28Z,2019-07-24T06:34:28Z,MEMBER,"Clearing the root element each time saved even more:
","{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 2, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472097220,Script uses a lot of RAM,
https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515226724,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9,515226724,MDEyOklzc3VlQ29tbWVudDUxNTIyNjcyNA==,9599,simonw,2019-07-25T21:46:01Z,2019-07-25T21:46:01Z,MEMBER,I can work around this here (prior to the fix in sqlite-utils) by setting the batch size to something a bit lower here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472429048,Too many SQL variables,
https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515322294,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9,515322294,MDEyOklzc3VlQ29tbWVudDUxNTMyMjI5NA==,9599,simonw,2019-07-26T06:07:12Z,2019-07-26T06:07:12Z,MEMBER,@tholo this should be fixed in just-released version 0.3.2 - could you run a `pip install -U healthkit-to-sqlite` and let me know if it works for you now?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472429048,Too many SQL variables,
https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-538847446,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1,538847446,MDEyOklzc3VlQ29tbWVudDUzODg0NzQ0Ng==,9599,simonw,2019-10-07T05:41:17Z,2019-10-07T05:41:17Z,MEMBER,"Prototype code:
```python
offset = 0
fetched = []
size = 500
while True:
page = requests.get(""https://getpocket.com/v3/get"", {
""consumer_key"": consumer_key,
""access_token"": access_token,
""sort"": ""oldest"",
""detailType"": ""complete"",
""count"": size,
""offset"": offset,
}).json()
print(offset)
fetched.append(page)
offset += size
if not len(page[""list""]):
break
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503233021,Use better pagination (and implement progress bar),
https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605316146,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1,605316146,MDEyOklzc3VlQ29tbWVudDYwNTMxNjE0Ng==,9599,simonw,2020-03-27T21:09:15Z,2020-03-27T21:09:22Z,MEMBER,"For a progress bar I need to know how many total items there are.
I found an undocumented API for this! `/v3/stats` which returns:
```json
{
""count_list"": 7394,
""count_read"": 1016,
""count_unread"": 6378,
""status"": 1
}
```
I guessed this based on the documented v2 API: https://getpocket.com/api/v2_docs/#stats","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503233021,Use better pagination (and implement progress bar),
https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605325897,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1,605325897,MDEyOklzc3VlQ29tbWVudDYwNTMyNTg5Nw==,9599,simonw,2020-03-27T21:37:26Z,2020-03-27T21:38:37Z,MEMBER,"I keep getting 503 errors even though I appear to be staying within the rate limit:
```
{'Date': 'Fri, 27 Mar 2020 21:35:57 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding':
'chunked', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location':
'get.php', 'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...; httponly',
'X-Frame-Options': 'SAMEORIGIN', 'Status': '200 OK', 'X-Limit-Key-Limit': '10000',
'X-Limit-Key-Remaining': '9960', 'X-Limit-Key-Reset': '282', 'X-Source': 'Pocket',
'P3P': 'policyref=""/w3c/p3p.xml"", CP=""ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE""'}
[##----------------------------------] 6% 06:49:27
{'Date': 'Fri, 27 Mar 2020 21:36:06 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Content-Length':
'23', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location': 'get.php',
'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...', 'X-Frame-Options': 'SAMEORIGIN',
'X-Error': 'Pocket is currently under heavy load. Please wait a moment and try again.',
'X-Error-Code': '199', 'Status': '503 Service Unavailable', 'X-Source': 'Pocket',
'P3P': 'policyref=""/w3c/p3p.xml"", CP=""ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE""'}
```
I'm going to try doing a few automatic retries any time I see a 503 error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503233021,Use better pagination (and implement progress bar),
https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-605327655,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1,605327655,MDEyOklzc3VlQ29tbWVudDYwNTMyNzY1NQ==,9599,simonw,2020-03-27T21:42:49Z,2020-03-27T21:42:49Z,MEMBER,Or maybe it was because of the current Google Cloud outage? https://news.ycombinator.com/item?id=22706677,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503233021,Use better pagination (and implement progress bar),
https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1221622873,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10,1221622873,IC_kwDODLZ_YM5I0HxZ,9599,simonw,2022-08-21T21:19:25Z,2022-08-21T21:19:25Z,MEMBER,"Agreed, that would be a much better implementation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1246826792,"When running `auth` command, don't overwrite an existing auth.json file",
https://github.com/dogsheep/pocket-to-sqlite/issues/10#issuecomment-1221623052,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10,1221623052,IC_kwDODLZ_YM5I0H0M,9599,simonw,2022-08-21T21:20:33Z,2022-08-21T21:20:33Z,MEMBER,"That was clearly the intention from the description of this issue:
- #4","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1246826792,"When running `auth` command, don't overwrite an existing auth.json file",
https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221621466,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11,1221621466,IC_kwDODLZ_YM5I0Hba,9599,simonw,2022-08-21T21:09:47Z,2022-08-21T21:09:47Z,MEMBER,"Great catch, thanks.
I'm going to use it to mean `--auth` - since other tools in the Dogsheep family have the same convention.
`--all` will be the only way to specify all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1345452427,"-a option is used for ""--auth"" and for ""--all""",
https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221621529,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11,1221621529,IC_kwDODLZ_YM5I0HcZ,9599,simonw,2022-08-21T21:10:15Z,2022-08-21T21:11:26Z,MEMBER,"Just saw that's what's implemented here already!
- #7","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1345452427,"-a option is used for ""--auth"" and for ""--all""",
https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627563202,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12,1627563202,IC_kwDODLZ_YM5hAqTC,9599,simonw,2023-07-09T01:14:27Z,2023-07-09T01:14:27Z,MEMBER,I tested this locally with `python -m build` and then `pip install ...whl` in a fresh virtual environment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795187493,Switch to pyproject.toml,
https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627564127,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12,1627564127,IC_kwDODLZ_YM5hAqhf,9599,simonw,2023-07-09T01:19:42Z,2023-07-09T01:19:42Z,MEMBER,https://github.com/dogsheep/pocket-to-sqlite/tree/0.2.3 and https://pypi.org/project/pocket-to-sqlite/0.2.3/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795187493,Switch to pyproject.toml,
https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-538847796,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2,538847796,MDEyOklzc3VlQ29tbWVudDUzODg0Nzc5Ng==,9599,simonw,2019-10-07T05:43:30Z,2019-10-07T05:43:30Z,MEMBER,We can persist the `since` value in its own single-row table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503234169,Track and use the 'since' value,
https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-605337941,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2,605337941,MDEyOklzc3VlQ29tbWVudDYwNTMzNzk0MQ==,9599,simonw,2020-03-27T22:16:32Z,2020-03-27T22:16:32Z,MEMBER,"Need to test this. I have 7,394 items in my database right now. I'm going to save a new thing.
Then I ran this:
```
pocket-to-sqlite fetch pocket-simon.db
```
And it worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503234169,Track and use the 'since' value,
https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-605338322,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2,605338322,MDEyOklzc3VlQ29tbWVudDYwNTMzODMyMg==,9599,simonw,2020-03-27T22:18:02Z,2020-03-27T22:18:02Z,MEMBER,Just needs documentation now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503234169,Track and use the 'since' value,
https://github.com/dogsheep/pocket-to-sqlite/issues/3#issuecomment-684424396,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3,684424396,MDEyOklzc3VlQ29tbWVudDY4NDQyNDM5Ng==,9599,simonw,2020-09-01T06:17:45Z,2020-09-01T06:17:45Z,MEMBER,It looks like I could ignore the `image` column and synthesize a unique key from the data in the `images` column using `$item_id/$image_id`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503243784,Extract images into separate tables,
https://github.com/dogsheep/pocket-to-sqlite/issues/5#issuecomment-684425714,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5,684425714,MDEyOklzc3VlQ29tbWVudDY4NDQyNTcxNA==,9599,simonw,2020-09-01T06:18:32Z,2020-09-01T06:18:32Z,MEMBER,"Good suggestion, I'll setup a demo somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629473827,Set up a demo,
https://github.com/dogsheep/pocket-to-sqlite/pull/7#issuecomment-1221621700,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/7,1221621700,IC_kwDODLZ_YM5I0HfE,9599,simonw,2022-08-21T21:11:12Z,2022-08-21T21:11:12Z,MEMBER,I thought this might need a documentation update but `--all` is already covered: https://github.com/dogsheep/pocket-to-sqlite/blob/0.2.1/README.md,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750141615,Fixed conflicting CLI flags,
https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-727692413,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11,727692413,MDEyOklzc3VlQ29tbWVudDcyNzY5MjQxMw==,9599,simonw,2020-11-16T02:15:22Z,2020-11-16T02:15:22Z,MEMBER,"Thanks, I'll look into this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743400216,Error thrown: sqlite3.OperationalError: table users has no column named lastName,
https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-761967094,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11,761967094,MDEyOklzc3VlQ29tbWVudDc2MTk2NzA5NA==,9599,simonw,2021-01-18T04:11:13Z,2021-01-18T04:11:13Z,MEMBER,"I just got a similar error:
```
File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/swarm_to_sqlite/utils.py"", line 79, in save_checkin
checkins_table.m2m(""users"", user, m2m_table=""with"", pk=""id"")
File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 2048, in m2m
id = other_table.insert(record, pk=pk, replace=True).last_pk
File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1781, in insert
return self.insert_all(
File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1899, in insert_all
self.insert_chunk(
File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1709, in insert_chunk
result = self.db.execute(query, params)
File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 226, in execute
return self.conn.execute(sql, parameters)
pysqlite3.dbapi2.OperationalError: table users has no column named countryCode
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743400216,Error thrown: sqlite3.OperationalError: table users has no column named lastName,
https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502543165,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13,1502543165,IC_kwDODD6af85Zjv09,9599,simonw,2023-04-11T01:10:36Z,2023-04-11T01:11:47Z,MEMBER,"I just had that error myself on macOS while running the tests:
```
ERROR tests/test_save_checkin.py::test_tables - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_venue - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_event - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_sticker - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_likes - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_with_ - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_users - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_photos - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_posts - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_view - sqlite3.OperationalError: table sqlite_master may not be modified
```
`pytest --pdb` shows it happening in the bit that adds foreign keys:
```
> /Users/simon/.local/share/virtualenvs/swarm-to-sqlite-daPW7yIJ/lib/python3.9/site-packages/sqlite_utils/db.py(1096)add_foreign_keys()
-> cursor.execute(
(Pdb) list
1096 >> cursor.execute(
1097 ""UPDATE sqlite_master SET sql = ? WHERE name = ?"",
1098 (new_sql, table_name),
1099 )
1100 cursor.execute(""PRAGMA schema_version = %d"" % (schema_version + 1))
1101 -> cursor.execute(""PRAGMA writable_schema = 0"")
1102 # Have to VACUUM outside the transaction to ensure .foreign_keys property
1103 # can see the newly created foreign key.
1104 self.vacuum()
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373210675,fails before generating views. ERR: table sqlite_master may not be modified,
https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502546045,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13,1502546045,IC_kwDODD6af85Zjwh9,9599,simonw,2023-04-11T01:14:50Z,2023-04-11T01:14:50Z,MEMBER,"Related:
- https://github.com/simonw/sqlite-utils/issues/235","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373210675,fails before generating views. ERR: table sqlite_master may not be modified,
https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502629219,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13,1502629219,IC_kwDODD6af85ZkE1j,9599,simonw,2023-04-11T03:15:26Z,2023-04-11T03:15:26Z,MEMBER,"OK, I figured this out. Unfortunately it's an error that occurs on Python versions that have defensive mode turned on, and it doesn't look like there's a way to turn that mode off. See notes above.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373210675,fails before generating views. ERR: table sqlite_master may not be modified,
https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502629404,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13,1502629404,IC_kwDODD6af85ZkE4c,9599,simonw,2023-04-11T03:15:47Z,2023-04-11T03:46:17Z,MEMBER,"I think `swarm-to-sqlite` needs to avoid this error, maybe by setting up foreign keys in another way - or even by skipping foreign keys entirely on databases that don't support this kind of operation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373210675,fails before generating views. ERR: table sqlite_master may not be modified,
https://github.com/dogsheep/swarm-to-sqlite/issues/2#issuecomment-526701674,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/2,526701674,MDEyOklzc3VlQ29tbWVudDUyNjcwMTY3NA==,9599,simonw,2019-08-30T18:24:26Z,2019-08-30T18:24:26Z,MEMBER,I renamed `--file` to `--load` in 0e5b6025c6f9823ff81aa8aae1cbff5c45e57baf,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487598468,--save option to dump checkins to a JSON file on disk,
https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-527200332,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3,527200332,MDEyOklzc3VlQ29tbWVudDUyNzIwMDMzMg==,9599,simonw,2019-09-02T16:32:20Z,2019-09-02T16:32:39Z,MEMBER,"Also needed: an option for ""fetch all checkins created within the last X days"".
This should help provide support for that Swarm feature where you can retroactively checkin to places in the past.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487600595,Option to fetch only checkins more recent than the current max checkin,
https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542875885,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3,542875885,MDEyOklzc3VlQ29tbWVudDU0Mjg3NTg4NQ==,9599,simonw,2019-10-16T20:23:08Z,2019-10-16T20:23:08Z,MEMBER,"https://developer.foursquare.com/docs/api/users/checkins documents `afterTimestamp`:
> Retrieve the first results to follow these seconds since epoch. This should be useful for paging forward in time, or when polling for changes. To avoid missing results when polling, we recommend subtracting several seconds from the last poll time and then de-duplicating.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487600595,Option to fetch only checkins more recent than the current max checkin,
https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542876047,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3,542876047,MDEyOklzc3VlQ29tbWVudDU0Mjg3NjA0Nw==,9599,simonw,2019-10-16T20:23:36Z,2019-10-16T20:23:36Z,MEMBER,I'm going to go with `--since=1d/2w/3h` for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487600595,Option to fetch only checkins more recent than the current max checkin,
https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542882604,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3,542882604,MDEyOklzc3VlQ29tbWVudDU0Mjg4MjYwNA==,9599,simonw,2019-10-16T20:41:23Z,2019-10-16T20:41:23Z,MEMBER,Documented here: https://github.com/dogsheep/swarm-to-sqlite/blob/0.2/README.md#usage,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487600595,Option to fetch only checkins more recent than the current max checkin,
https://github.com/dogsheep/swarm-to-sqlite/issues/4#issuecomment-526853542,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/4,526853542,MDEyOklzc3VlQ29tbWVudDUyNjg1MzU0Mg==,9599,simonw,2019-08-31T18:06:32Z,2019-08-31T18:06:32Z,MEMBER,"https://your-foursquare-oauth-token.glitch.me/
Source code: https://glitch.com/~your-foursquare-oauth-token","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487601121,Online tool for getting a Foursquare OAuth token,
https://github.com/dogsheep/swarm-to-sqlite/pull/6#issuecomment-605382373,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/6,605382373,MDEyOklzc3VlQ29tbWVudDYwNTM4MjM3Mw==,9599,simonw,2020-03-28T02:27:32Z,2020-03-28T02:27:32Z,MEMBER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",543355051,don't break if source is missing,
https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-669241341,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8,669241341,MDEyOklzc3VlQ29tbWVudDY2OTI0MTM0MQ==,9599,simonw,2020-08-05T14:55:14Z,2020-08-05T14:55:14Z,MEMBER,"Looks like there's a column that's not consistently there, so the table got created without it.
Easiest fix is to add `alter=True` to this line:
https://github.com/dogsheep/swarm-to-sqlite/blob/f4a82633da927cde672c9d9af92930bfca2e3ddf/swarm_to_sqlite/utils.py#L94
That will cause `sqlite-utils` to notice if there's a missing column and add it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648245071,Error thrown: table photos has no column named hasSticker,
https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-707332912,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8,707332912,MDEyOklzc3VlQ29tbWVudDcwNzMzMjkxMg==,9599,simonw,2020-10-12T20:35:06Z,2020-10-12T20:35:06Z,MEMBER,Shipped a fix for this in [swarm-to-sqlite 0.3.2](https://github.com/dogsheep/swarm-to-sqlite/releases/tag/0.3.2).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648245071,Error thrown: table photos has no column named hasSticker,
https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-541748580,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,541748580,MDEyOklzc3VlQ29tbWVudDU0MTc0ODU4MA==,9599,simonw,2019-10-14T15:30:44Z,2019-10-14T15:30:44Z,MEMBER,Had several recommendations for https://github.com/tqdm/tqdm which is what goodreads-to-sqlite uses.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands,
https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543269396,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,543269396,MDEyOklzc3VlQ29tbWVudDU0MzI2OTM5Ng==,9599,simonw,2019-10-17T17:02:07Z,2019-10-17T17:02:07Z,MEMBER,A neat trick that Click does is detecting if an interactive terminal is attached and NOT showing a progress bar if there isn't one. Need to figure out how to do that with tqdm.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands,
https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543270714,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,543270714,MDEyOklzc3VlQ29tbWVudDU0MzI3MDcxNA==,9599,simonw,2019-10-17T17:05:16Z,2019-10-17T17:05:16Z,MEMBER,"https://github.com/pallets/click/blob/716a5be90f56ce6cd506bb53d5739d09374b1636/click/_termui_impl.py#L93 is how Click does this:
```
self.is_hidden = not isatty(self.file)
```
Where `isatty` is a Click utility function: `from ._compat import isatty`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands,
https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543271000,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,543271000,MDEyOklzc3VlQ29tbWVudDU0MzI3MTAwMA==,9599,simonw,2019-10-17T17:05:59Z,2019-10-17T17:05:59Z,MEMBER,Looks like tqdm already does a TTY check here: https://github.com/tqdm/tqdm/blob/89b73bdc30c099c5b53725806e7edf3a121c9b3a/tqdm/std.py#L889-L890,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands,
https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-607010791,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,607010791,MDEyOklzc3VlQ29tbWVudDYwNzAxMDc5MQ==,9599,simonw,2020-04-01T03:45:48Z,2020-04-01T03:45:48Z,MEMBER,I'm happy with the recent work I did on this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands,
https://github.com/dogsheep/twitter-to-sqlite/issues/11#issuecomment-538711918,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/11,538711918,MDEyOklzc3VlQ29tbWVudDUzODcxMTkxOA==,9599,simonw,2019-10-06T04:54:17Z,2019-10-06T04:54:17Z,MEMBER,Shipped in 0.6. Here's the documentation: https://github.com/dogsheep/twitter-to-sqlite#capturing-tweets-in-real-time-with-track-and-follow,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503045221,Commands for recording real-time tweets from the streaming API,
https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855081,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12,542855081,MDEyOklzc3VlQ29tbWVudDU0Mjg1NTA4MQ==,9599,simonw,2019-10-16T19:26:56Z,2019-10-16T19:26:56Z,MEMBER,This may be the first case where I want to be able to repair existing databases rather than discarding their contents.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503053800,"Extract ""source"" into a separate lookup table",
https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855427,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12,542855427,MDEyOklzc3VlQ29tbWVudDU0Mjg1NTQyNw==,9599,simonw,2019-10-16T19:27:55Z,2019-10-16T19:27:55Z,MEMBER,I can do that by keeping `source` as a `TEXT` column but turning it into a non-enforced foreign key against a new `sources` table. Then I can run code that scans that column for any values beginning with a `<` and converts them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503053800,"Extract ""source"" into a separate lookup table",
https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542858025,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12,542858025,MDEyOklzc3VlQ29tbWVudDU0Mjg1ODAyNQ==,9599,simonw,2019-10-16T19:35:31Z,2019-10-16T19:36:09Z,MEMBER,Maybe this means I need an `upgrade` command to apply these kinds of migrations? Total feature creep!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503053800,"Extract ""source"" into a separate lookup table",
https://github.com/dogsheep/twitter-to-sqlite/issues/13#issuecomment-538804815,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/13,538804815,MDEyOklzc3VlQ29tbWVudDUzODgwNDgxNQ==,9599,simonw,2019-10-07T00:33:49Z,2019-10-07T00:33:49Z,MEMBER,Documentation: https://github.com/dogsheep/twitter-to-sqlite#retrieve-tweets-in-bulk,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503085013,statuses-lookup command,
https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112108,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17,541112108,MDEyOklzc3VlQ29tbWVudDU0MTExMjEwOA==,9599,simonw,2019-10-11T15:30:15Z,2019-10-11T15:30:15Z,MEMBER,It should delete the tables entirely. That way it will work even if the table schema has changed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505674949,import command should empty all archive-* tables first,
https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112588,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17,541112588,MDEyOklzc3VlQ29tbWVudDU0MTExMjU4OA==,9599,simonw,2019-10-11T15:31:30Z,2019-10-11T15:31:30Z,MEMBER,"No need for an option:
> This command will delete and recreate all of your `archive-*` tables every time you run it. If this is not what you want, run the command against a fresh SQLite database rather than running it again one that already exists.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505674949,import command should empty all archive-* tables first,
https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541118773,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18,541118773,MDEyOklzc3VlQ29tbWVudDU0MTExODc3Mw==,9599,simonw,2019-10-11T15:48:31Z,2019-10-11T15:48:31Z,MEMBER,https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-home_timeline,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505928530,Command to import home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541118934,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18,541118934,MDEyOklzc3VlQ29tbWVudDU0MTExODkzNA==,9599,simonw,2019-10-11T15:48:54Z,2019-10-11T15:48:54Z,MEMBER,Rate limit is tight: 15 requests every 15 mins!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505928530,Command to import home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541119834,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18,541119834,MDEyOklzc3VlQ29tbWVudDU0MTExOTgzNA==,9599,simonw,2019-10-11T15:51:22Z,2019-10-11T16:51:33Z,MEMBER,"In order to support multiple user timelines being saved in the same database, I'm going to import the tweets into the `tweets` table AND add a new `timeline_tweets` table recording that a specific tweet showed up in a specific user's timeline.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505928530,Command to import home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541141169,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18,541141169,MDEyOklzc3VlQ29tbWVudDU0MTE0MTE2OQ==,9599,simonw,2019-10-11T16:51:29Z,2019-10-11T16:51:29Z,MEMBER,Documented here: https://github.com/dogsheep/twitter-to-sqlite/blob/master/README.md#retrieving-tweets-from-your-home-timeline,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505928530,Command to import home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-541248629,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19,541248629,MDEyOklzc3VlQ29tbWVudDU0MTI0ODYyOQ==,9599,simonw,2019-10-11T22:48:56Z,2019-10-11T22:48:56Z,MEMBER,"`since_id` documented here: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-home_timeline
> Returns results with an ID greater than (that is, more recent than) the specified ID. There are limits to the number of Tweets which can be accessed through the API. If the limit of Tweets has occured since the since_id, the since_id will be forced to the oldest ID available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506087267,since_id support for home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542832952,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19,542832952,MDEyOklzc3VlQ29tbWVudDU0MjgzMjk1Mg==,9599,simonw,2019-10-16T18:30:11Z,2019-10-16T18:30:11Z,MEMBER,"The `--since` option will derive the `since_id` from the max ID in the `timeline_tweets` table:
$ twitter-to-sqlite home-timeline --since
The `--since_id=xxx` option lets you specify that ID directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506087267,since_id support for home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542849963,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19,542849963,MDEyOklzc3VlQ29tbWVudDU0Mjg0OTk2Mw==,9599,simonw,2019-10-16T19:13:06Z,2019-10-16T19:13:06Z,MEMBER,Updated documentation: https://github.com/dogsheep/twitter-to-sqlite/blob/fced2a9b67d2cbdf9817f1eb75f7c28e413c963b/README.md#retrieving-tweets-from-your-home-timeline,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506087267,since_id support for home-timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527954898,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2,527954898,MDEyOklzc3VlQ29tbWVudDUyNzk1NDg5OA==,9599,simonw,2019-09-04T15:31:46Z,2019-09-04T15:31:46Z,MEMBER,I'm going to call this `twitter-to-sqlite user-timeline` to reflect the language used to describe the API endpoint: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833698,"""twitter-to-sqlite user-timeline"" command for pulling tweets by a specific user",
https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527955302,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2,527955302,MDEyOklzc3VlQ29tbWVudDUyNzk1NTMwMg==,9599,simonw,2019-09-04T15:32:39Z,2019-09-04T15:32:39Z,MEMBER,Rate limit is 900 / 15 minutes which is 1 call per second.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833698,"""twitter-to-sqlite user-timeline"" command for pulling tweets by a specific user",
https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527990908,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2,527990908,MDEyOklzc3VlQ29tbWVudDUyNzk5MDkwOA==,9599,simonw,2019-09-04T16:57:24Z,2019-09-04T16:57:24Z,MEMBER,"I just tried this using `max_id=` pagination as described in [Working with timelines](https://developer.twitter.com/en/docs/tweets/timelines/guides/working-with-timelines) and I got back all 17,759 of my tweets.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833698,"""twitter-to-sqlite user-timeline"" command for pulling tweets by a specific user",
https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-541388038,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20,541388038,MDEyOklzc3VlQ29tbWVudDU0MTM4ODAzOA==,9599,simonw,2019-10-13T05:31:58Z,2019-10-13T05:31:58Z,MEMBER,For favourites a `--stop_after=200` option is probably good enough.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506268945,--since support for various commands for refresh-by-cron,
https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-542854749,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20,542854749,MDEyOklzc3VlQ29tbWVudDU0Mjg1NDc0OQ==,9599,simonw,2019-10-16T19:26:01Z,2019-10-16T19:26:01Z,MEMBER,"I'm not going to do this for ""accounts that have followed me"" and ""new accounts that I have followed"" - instead I will recommend running the `friend_ids` and `followers_ids` commands on a daily basis since that data doesn't really change much by the hour.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506268945,--since support for various commands for refresh-by-cron,
https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-544335363,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20,544335363,MDEyOklzc3VlQ29tbWVudDU0NDMzNTM2Mw==,9599,simonw,2019-10-21T03:32:04Z,2019-10-21T03:32:04Z,MEMBER,"In case anyone is interested, here's an extract from the crontab I'm running these under at the moment:
```
1,11,21,31,41,51 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite user-timeline /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --since
2,7,12,17,22,27,32,37,42,47,52,57 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite home-timeline /home/ubuntu/timeline.db -a /home/ubuntu/auth.json --since
6,16,26,36,46,56 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite favorites /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --stop_after=50
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506268945,--since support for various commands for refresh-by-cron,
https://github.com/dogsheep/twitter-to-sqlite/issues/21#issuecomment-542333836,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/21,542333836,MDEyOklzc3VlQ29tbWVudDU0MjMzMzgzNg==,9599,simonw,2019-10-15T18:00:48Z,2019-10-15T18:00:48Z,MEMBER,I'll use `html.unescape()` for this: https://docs.python.org/3/library/html.html#html.unescape,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506432572,Fix & escapes in tweet text,
https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543217890,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23,543217890,MDEyOklzc3VlQ29tbWVudDU0MzIxNzg5MA==,9599,simonw,2019-10-17T15:03:10Z,2019-10-17T15:03:10Z,MEMBER,"Thinking about this further: the concept of migrations may end up being in direct conflict with the `sqlite-utils` concept of creating tables on demand the first time they are used - and of creating table schemas automatically to fit the shape of the JSON that is being inserted into them.
I'm going to forge ahead anyway and build this because I think it will be an interesting exploration, but it's very likely this turns out to be a bad idea in the long run!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508190730,Extremely simple migration system,
https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543222239,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23,543222239,MDEyOklzc3VlQ29tbWVudDU0MzIyMjIzOQ==,9599,simonw,2019-10-17T15:12:33Z,2019-10-17T15:12:33Z,MEMBER,"Migrations will run only if you open a database that previously existed (as opposed to opening a brand new empty database).
This means that the first time you run a command against a fresh database, migrations will not run and the `migrations` table will not be created. The _second_ time you run any command against that database the migrations will execute and populate the `migrations` table.
This also means that each migration needs to be able to sanity check the database to see if it should run or not. If it should NOT run, it will do nothing but still be marked as having executed by adding to the `migrations` table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508190730,Extremely simple migration system,
https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543265058,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25,543265058,MDEyOklzc3VlQ29tbWVudDU0MzI2NTA1OA==,9599,simonw,2019-10-17T16:51:12Z,2019-10-17T16:51:12Z,MEMBER,"This migration function only runs if there is a table called `tweets` and the migration has not run before.
I think this can happen if the database has just been freshly created (by a command that fetches the user's user timeline for example) and is then run a SECOND time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508578780,Ensure migrations don't accidentally create foreign key twice,
https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543266947,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25,543266947,MDEyOklzc3VlQ29tbWVudDU0MzI2Njk0Nw==,9599,simonw,2019-10-17T16:56:06Z,2019-10-17T16:56:06Z,MEMBER,I wrote a test that proves that this is a problem. Should be an easy fix though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508578780,Ensure migrations don't accidentally create foreign key twice,
https://github.com/dogsheep/twitter-to-sqlite/issues/26#issuecomment-547713287,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/26,547713287,MDEyOklzc3VlQ29tbWVudDU0NzcxMzI4Nw==,9599,simonw,2019-10-30T02:36:13Z,2019-10-30T02:36:13Z,MEMBER,"Shipped this in 0.13: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.13
See also this Twitter thread: https://twitter.com/simonw/status/1189369677509623809","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",513074501,Command for importing mentions timeline,
https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095217,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27,549095217,MDEyOklzc3VlQ29tbWVudDU0OTA5NTIxNw==,9599,simonw,2019-11-03T01:06:25Z,2019-11-03T01:06:25Z,MEMBER,"Wow, that `retweets_of_me` endpoint is almost completely useless:
```
$ twitter-to-sqlite fetch https://api.twitter.com/1.1/statuses/retweets_of_me.json
```
It returns my own tweets that have been retweeted, but with no indication at all of who retweeted them.
It looks like this needs to be combined with this API - https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-retweets-id - to fetch the details of up to 100 recent users who actually DID retweet an individual status. But that has a one-every-12-seconds rate limit on it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",514459062,retweets-of-me command,
https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095317,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27,549095317,MDEyOklzc3VlQ29tbWVudDU0OTA5NTMxNw==,9599,simonw,2019-11-03T01:08:10Z,2019-11-03T01:08:10Z,MEMBER,"Hmm... one thing that could be useful is that `retweets_of_me` can support a `--since` parameter - so if run frequently it should hopefully let us know which tweets we would need to run `statuses/retweets/:id.json` against.
I'm not sure if the `--since` parameter would show me a tweet that was previously retweeted but has now been retweeted again. I'll have a bit of a test and see.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",514459062,retweets-of-me command,
https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095463,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27,549095463,MDEyOklzc3VlQ29tbWVudDU0OTA5NTQ2Mw==,9599,simonw,2019-11-03T01:10:52Z,2019-11-03T01:10:52Z,MEMBER,"I imagine it won't, since the data I would be recording and then passing to `since_id` would be the highest ID of my own tweets that have been retweeted at least once. So it won't be able to spot if I should check for fresh retweets of a given tweet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",514459062,retweets-of-me command,
https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095641,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27,549095641,MDEyOklzc3VlQ29tbWVudDU0OTA5NTY0MQ==,9599,simonw,2019-11-03T01:12:58Z,2019-11-03T01:12:58Z,MEMBER,"It looks like Twitter really want you to subscribe to a premium API for this kind of thing and consume retweets via webhooks: https://developer.twitter.com/en/docs/accounts-and-users/subscribe-account-activity/api-reference
I'm going to give up on this for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",514459062,retweets-of-me command,
https://github.com/dogsheep/twitter-to-sqlite/issues/28#issuecomment-552133488,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/28,552133488,MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ4OA==,9599,simonw,2019-11-09T20:15:42Z,2019-11-09T20:15:42Z,MEMBER,Released: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.15,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",515658861,Add indexes to followers table,
https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129686,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29,552129686,MDEyOklzc3VlQ29tbWVudDU1MjEyOTY4Ng==,9599,simonw,2019-11-09T19:27:39Z,2019-11-09T19:27:39Z,MEMBER,I think this is fixed by the latest version of `sqlite-utils` - https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-12-1 - I'll bump the dependency.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",518725064,`import` command fails on empty files,
https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129921,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29,552129921,MDEyOklzc3VlQ29tbWVudDU1MjEyOTkyMQ==,9599,simonw,2019-11-09T19:30:42Z,2019-11-09T19:30:42Z,MEMBER,"Confirmed, that seems to fix it:
```
(twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js
Traceback (most recent call last):
File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite"", line 11, in