home / github

Menu
  • Search all tables
  • GraphQL API

issues

Table actions
  • GraphQL API for issues

12 rows where repo = 197882382 and state = "open" sorted by updated_at descending

✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: user, comments, author_association, created_at (date), updated_at (date)

type 2

  • issue 6
  • pull 6

state 1

  • open · 12 ✖

repo 1

  • healthkit-to-sqlite · 12 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
771608692 MDU6SXNzdWU3NzE2MDg2OTI= 14 UNIQUE constraint failed: workouts.id n8henrie 1234956 open 0     5 2020-12-20T15:11:20Z 2023-07-10T14:46:52Z   NONE  

I'm getting an error on my initial attempt to import data:

console $ healthkit-to-sqlite 20201119\ healthkit\ export.zip healthkit.db Importing from HealthKit [###################################-] 98% 00:00:01 Traceback (most recent call last): File "venv/bin/healthkit-to-sqlite", line 8, in <module> sys.exit(cli()) File "venv/lib/python3.9/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "venv/lib/python3.9/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "venv/lib/python3.9/site-packages/healthkit_to_sqlite/cli.py", line 57, in cli convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf) File "venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py", line 34, in convert_xml_to_sqlite workout_to_db(el, db, zipfile) File "venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py", line 57, in workout_to_db pk = db["workouts"].insert(record, alter=True, hash_id="id").last_pk File "venv/lib/python3.9/site-packages/sqlite_utils/db.py", line 1660, in insert return self.insert_all( File "venv/lib/python3.9/site-packages/sqlite_utils/db.py", line 1778, in insert_all self.insert_chunk( File "venv/lib/python3.9/site-packages/sqlite_utils/db.py", line 1588, in insert_chunk result = self.db.execute(query, params) File "venv/lib/python3.9/site-packages/sqlite_utils/db.py", line 213, in execute return self.conn.execute(sql, parameters) sqlite3.IntegrityError: UNIQUE constraint failed: workouts.id

healthkit-to-sqlite 197882382 issue    
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   
1515883470 I_kwDOC8tyDs5aWovO 24 DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 mmngreco 6231413 open 0     2 2023-01-01T23:00:38Z 2023-03-30T10:17:31Z   NONE  

Hi @simonw

I hope you find this issue ok, the idea is provide some documentation to other users like me about how to solve this problem and save some time.

Following the instructions from the README.md I've faced this error:

bash (venv) mgreco@pop-os apple-health master* (23:44|0s) $ healthkit-to-sqlite apple_health_export/export.xml healthkit.db --xml Importing from HealthKit [------------------------------------] 0% Traceback (most recent call last): File "/home/mgreco/github/mmngreco/apple-health/venv/bin/healthkit-to-sqlite", line 33, in <module> sys.exit(load_entry_point('healthkit-to-sqlite', 'console_scripts', 'healthkit-to-sqlite')()) File "/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/cli.py", line 57, in cli convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf) File "/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/utils.py", line 25, in convert_xml_to_sqlite for tag, el in find_all_tags( File "/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/utils.py", line 12, in find_all_tags for event, el in parser.read_events(): File "/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/xml/etree/ElementTree.py", line 1324, in read_events raise event File "/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/xml/etree/ElementTree.py", line 1296, in feed self._parser.feed(data) xml.etree.ElementTree.ParseError: syntax error: line 156, column 0

So, after debugging and searching on internet I found this useful link: https://discussions.apple.com/thread/254202523 (etresoft, the real hero). Which basically says that the xml given by the health app (healthkit version 12) has some bugs but fortunately, they can be solved with a couple of commads:

  1. Uncompress the zip and move the new folder where export.xml is.
  2. Create a patch.txt with the following content

    ```diff --- export.xml 2022-09-18 15:17:09.000000000 -0400 +++ export-fixed.xml 2022-09-18 16:37:08.000000000 -0400 @@ -15,6 +15,7 @@ HKCharacteristicTypeIdentifierBiologicalSex CDATA #REQUIRED HKCharacteristicTypeIdentifierBloodType CDATA #REQUIRED HKCharacteristicTypeIdentifierFitzpatrickSkinType CDATA #REQUIRED + HKCharacteristicTypeIdentifierCardioFitnessMedicationsUse CDATA #IMPLIED

    <!ELEMENT Record ((MetadataEntry|HeartRateVariabilityMetadataList)*)> <!ATTLIST Record @@ -39,7 +40,7 @@ startDate CDATA #REQUIRED endDate CDATA #REQUIRED

    -<!ELEMENT Workout ((MetadataEntry|WorkoutEvent|WorkoutRoute))> +<!ELEMENT Workout ((MetadataEntry|WorkoutEvent|WorkoutRoute|WorkoutStatistics))> <!ATTLIST Workout workoutActivityType CDATA #REQUIRED duration CDATA #IMPLIED @@ -63,7 +64,7 @@ duration CDATA #IMPLIED durationUnit CDATA #IMPLIED

    -<!ELEMENT WorkoutEvent EMPTY> +<!ELEMENT WorkoutEvent (MetadataEntry?)> <!ATTLIST WorkoutEvent type CDATA #REQUIRED date CDATA #REQUIRED @@ -79,6 +80,7 @@ minimum CDATA #IMPLIED maximum CDATA #IMPLIED sum CDATA #IMPLIED + unit CDATA #IMPLIED

    <!ELEMENT WorkoutRoute ((MetadataEntry|FileReference)*)> <!ATTLIST WorkoutRoute @@ -153,6 +155,7 @@ dateIssued CDATA #REQUIRED expirationDate CDATA #REQUIRED brand CDATA #IMPLIED +> <!ELEMENT RightEye EMPTY> <!ATTLIST RightEye sphere CDATA #IMPLIED @@ -203,13 +206,6 @@ diameter CDATA #IMPLIED diameterUnit CDATA #IMPLIED

    • device CDATA #IMPLIED -<!ELEMENT MetadataEntry EMPTY> -<!ATTLIST MetadataEntry
    • key CDATA #IMPLIED
    • value CDATA #IMPLIED -> -> ]> <HealthData> <ExportDate/> ```
    • Apply the path with the command: patch < patch.txt
    • Fix endDates with the command sed 's/startDate/endDate/2' export.xml > export-fixed.xml
    • Try again healthkit-to-sqlite export-fixed.xml healthkit.db --xml
healthkit-to-sqlite 197882382 issue    
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   
1515717718 PR_kwDOC8tyDs5Gc-VH 23 Include workout statistics badboy 2129 open 0     0 2023-01-01T17:29:57Z 2023-01-01T17:29:57Z   FIRST_TIME_CONTRIBUTOR dogsheep/healthkit-to-sqlite/pulls/23

Not sure when this changed (iOS 16 maybe?), but the WorkoutStatistics now has a whole bunch of information about workouts, e.g. for runs it contains the distance (as a <WorkoutStatistics type="HKQuantityTypeIdentifierDistanceWalkingRunning ...> element).

Adding it as another column at leat allows me to pull these out (using SQLite's JSON support). I'm running with this patch on my own data now.

healthkit-to-sqlite 197882382 pull    
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/23/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
727848625 MDU6SXNzdWU3Mjc4NDg2MjU= 12 Some workout columns should be float, not text simonw 9599 open 0     4 2020-10-23T02:47:02Z 2022-06-23T04:35:02Z   MEMBER  

Columns duration, totalDistance and totalEnergyBurned should be converted to float.

https://github.com/dogsheep/healthkit-to-sqlite/blob/71e36e1cf034b96de2a8e6652265d782d3fdf63b/healthkit_to_sqlite/utils.py#L50-L57

healthkit-to-sqlite 197882382 issue    
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   
743071410 MDExOlB1bGxSZXF1ZXN0NTIxMDU0NjEy 13 SQLite does not have case sensitive columns tomaskrehlik 1689944 open 0     1 2020-11-14T20:12:32Z 2021-08-24T13:28:26Z   FIRST_TIME_CONTRIBUTOR dogsheep/healthkit-to-sqlite/pulls/13

This solves a weird issue when there is record with metadata key that is only different in letter cases.

See the test for details.

healthkit-to-sqlite 197882382 pull    
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/13/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
978086284 MDExOlB1bGxSZXF1ZXN0NzE4NzM0MTkx 22 Make sure that case-insensitive column names are unique FabianHertwig 32016596 open 0     1 2021-08-24T13:13:38Z 2021-08-24T13:26:20Z   FIRST_TIME_CONTRIBUTOR dogsheep/healthkit-to-sqlite/pulls/22

This closes #21.

When there are metadata entries with the same case insensitive string, then there is an error when trying to create a new column for that metadata entry in the database table, because a column with that case insensitive name already exists.

xml <Record type="HKQuantityTypeIdentifierDietaryCholesterol" sourceName="MyFitnessPal" sourceVersion="35120" unit="mg" creationDate="2021-07-04 20:55:27 +0200" startDate="2021-07-04 20:55:00 +0200" endDate="2021-07-04 20:55:00 +0200" value="124"> <MetadataEntry key="meal" value="Dinner"/> <MetadataEntry key="Meal" value="Dinner"/> </Record>

The code added in this PR checks if a key already exists in a record and if so adds a number at its end. The resulting column names look like the example below then. Interestingly, the column names viewed with Datasette are not case insensitive.

text startDate, endDate, value, unit, sourceName, sourceVersion, creationDate, metadata_meal, metadata_Meal_2, metadata_Mahlzeit

healthkit-to-sqlite 197882382 pull    
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/22/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
977128935 MDU6SXNzdWU5NzcxMjg5MzU= 21 Duplicate Column FabianHertwig 32016596 open 0     1 2021-08-23T15:00:44Z 2021-08-23T17:00:59Z   NONE  

Hey, thank you for this repo!

When I try to convert my export, I get a multiple column error. Here is the stack trace:

sh (.venv) (base) computer:bodyweight_app user$ healthkit-to-sqlite ./data/Health_export.zip ./data/healthkit.db Importing from HealthKit [###############################-----] 87% 00:00:22 Traceback (most recent call last): File "/MyProject/.venv/bin/healthkit-to-sqlite", line 10, in <module> sys.exit(cli()) File "/MyProject/.venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/MyProject/.venv/lib/python3.7/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/MyProject/.venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/MyProject/.venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/cli.py", line 57, in cli convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf) File "/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py", line 41, in convert_xml_to_sqlite write_records(records, db) File "/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py", line 146, in write_records batch_size=50, File "/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py", line 2579, in insert_all extracts=extracts, File "/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py", line 1246, in create extracts=extracts, File "/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py", line 767, in create_table self.execute(sql) File "/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py", line 421, in execute return self.conn.execute(sql) sqlite3.OperationalError: duplicate column name: metadata_Meal

healthkit-to-sqlite 197882382 issue    
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/21/reactions",
    "total_count": 5,
    "+1": 5,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   
975166271 MDU6SXNzdWU5NzUxNjYyNzE= 20 Add index on workout_points.date simonw 9599 open 0     2 2021-08-20T01:08:04Z 2021-08-20T01:12:48Z   MEMBER  

Sorting that by date makes sense for seeing most recent points, and my DB has 2.5m points in so it's an expensive sort!

healthkit-to-sqlite 197882382 issue    
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   
836064851 MDExOlB1bGxSZXF1ZXN0NTk2NjI3Nzgw 18 Add datetime parsing n8henrie 1234956 open 0     0 2021-03-19T14:34:22Z 2021-03-19T14:34:22Z   FIRST_TIME_CONTRIBUTOR dogsheep/healthkit-to-sqlite/pulls/18

Parses the datetime columns so they are subsequently properly recognized as datetime.

Fixes https://github.com/dogsheep/healthkit-to-sqlite/issues/17

healthkit-to-sqlite 197882382 pull    
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/18/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
836063389 MDU6SXNzdWU4MzYwNjMzODk= 17 Datetime columns are not properly formatted to be recognizes as datetime n8henrie 1234956 open 0     0 2021-03-19T14:33:04Z 2021-03-19T14:33:04Z   NONE  

Currently, the datetimes are formatted in a way that is not recognized by datasette-vega for plotting with a Date/time type for the axis.

For example, if you have datasette running locally with datasette-vega installed and have a database that includes resting heart rate:

http://localhost:8001/healthkit/rRestingHeartRate#g.mark=line&g.x_column=startDate&g.x_type=temporal&g.y_column=value&g.y_type=quantitative

The plot is blank unless you choose Label as the type for the date data.

The startDate (and creationDate and endDate) columns appear like: 2019-11-14 18:22:18 -0700

If instead the format for this column is changed slightly: 2019-11-14T18:22:18-07:00 they are recognized as proper dates and the charting works as expected.

I have a PR that addresses this issue, will submit shortly.

healthkit-to-sqlite 197882382 issue    
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/17/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   
830901133 MDExOlB1bGxSZXF1ZXN0NTkyMzY0MjU1 16 Add a fallback ID, print if no ID found n8henrie 1234956 open 0     0 2021-03-13T13:38:29Z 2021-03-13T14:44:04Z   FIRST_TIME_CONTRIBUTOR dogsheep/healthkit-to-sqlite/pulls/16

Fixes https://github.com/dogsheep/healthkit-to-sqlite/issues/14

healthkit-to-sqlite 197882382 pull    
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/16/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
793907673 MDExOlB1bGxSZXF1ZXN0NTYxNTEyNTAz 15 added try / except to write_records ryancheley 9857779 open 0     0 2021-01-26T03:56:21Z 2021-01-26T03:56:21Z   FIRST_TIME_CONTRIBUTOR dogsheep/healthkit-to-sqlite/pulls/15

to keep the data write from failing if it came across an error during processing. In particular when trying to convert my HealthKit zip file (and that of my wife's) it would consistently error out with the following:

``` db.py 1709 insert_chunk result = self.db.execute(query, params)

db.py 226 execute return self.conn.execute(sql, parameters)

sqlite3.OperationalError: too many SQL variables


db.py 1709 insert_chunk result = self.db.execute(query, params)

db.py 226 execute return self.conn.execute(sql, parameters)

sqlite3.OperationalError: too many SQL variables


db.py 1709 insert_chunk result = self.db.execute(query, params)

db.py 226 execute return self.conn.execute(sql, parameters)

sqlite3.OperationalError: table rBodyMass has no column named metadata_HKWasUserEntered


healthkit-to-sqlite 8 <module> sys.exit(cli())

core.py 829 call return self.main(args, *kwargs)

core.py 782 main rv = self.invoke(ctx)

core.py 1066 invoke return ctx.invoke(self.callback, **ctx.params)

core.py 610 invoke return callback(args, *kwargs)

cli.py 57 cli convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)

utils.py 42 convert_xml_to_sqlite write_records(records, db)

utils.py 143 write_records db[table].insert_all(

db.py 1899 insert_all self.insert_chunk(

db.py 1720 insert_chunk self.insert_chunk(

db.py 1720 insert_chunk self.insert_chunk(

db.py 1714 insert_chunk result = self.db.execute(query, params)

db.py 226 execute return self.conn.execute(sql, parameters)

sqlite3.OperationalError: table rBodyMass has no column named metadata_HKWasUserEntered ```

Adding the try / except in the write_records seems to fix that issue.

healthkit-to-sqlite 197882382 pull    
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/15/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);
Powered by Datasette · Queries took 30.907ms · About: github-to-sqlite
  • Sort ascending
  • Sort descending
  • Facet by this
  • Hide this column
  • Show all columns
  • Show not-blank rows