home / github

Menu
  • Search all tables
  • GraphQL API

issues

Table actions
  • GraphQL API for issues

11 rows where repo = 207052882 and type = "pull" sorted by updated_at descending

✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: comments, author_association, created_at (date), updated_at (date), closed_at (date)

state 2

  • open 6
  • closed 5

type 1

  • pull · 11 ✖

repo 1

  • github-to-sqlite · 11 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
923270900 MDExOlB1bGxSZXF1ZXN0NjcyMDUzODEx 65 basic support for events khimaros 231498 open 0     2 2021-06-17T00:51:30Z 2022-10-03T22:35:03Z   FIRST_TIME_CONTRIBUTOR dogsheep/github-to-sqlite/pulls/65

a quick first pass at implementing the feature requested in https://github.com/dogsheep/github-to-sqlite/issues/64

testing instructions:

$ github-to-sqlite events events.db user/khimaros

if the specified user is the authenticated user, it will also include private events.

caveat: pagination appears to be broken (i don't see next in the response JSON from GitHub)

github-to-sqlite 207052882 pull    
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1363280254 PR_kwDODFdgUs4-cIa_ 76 Add organization support to repos command OverkillGuy 2757699 open 0     1 2022-09-06T13:21:42Z 2022-09-06T13:59:08Z   FIRST_TIME_CONTRIBUTOR dogsheep/github-to-sqlite/pulls/76

New --organization flag to signify all given "usernames" are private orgs. Adapts API URL to the organization path instead.

Not the best implementation, but a first draft to talk around

Fixes #75 (badly, no tests, overly vague, untested)

github-to-sqlite 207052882 pull    
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/76/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1261884917 PR_kwDODFdgUs45K1L3 73 Fixing 'NoneType' object has no attribute 'items' empjustine 1224205 closed 0     1 2022-06-06T13:58:11Z 2022-07-18T19:40:12Z 2022-07-18T19:40:12Z CONTRIBUTOR dogsheep/github-to-sqlite/pulls/73

Under some conditions, GitHub caches removed starred repositories and ends up leaving dangling None user references.

Traceback (most recent call last):
  File "/home/dogsheep/dogsheep/github-to-sqlite/bin/github-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/cli.py", line 181, in starred
    utils.save_stars(db, user, stars)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py", line 494, in save_stars
    repo_id = save_repo(db, repo)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py", line 308, in save_repo
    to_save["owner"] = save_user(db, to_save["owner"])
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py", line 229, in save_user
    for key, value in user.items()
AttributeError: 'NoneType' object has no attribute 'items'
github-to-sqlite 207052882 pull    
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/73/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1149402080 PR_kwDODFdgUs4zaUta 70 scrape-dependents: enable paging through package menu option if present stanbiryukov 36061055 open 0     0 2022-02-24T15:07:25Z 2022-02-24T15:07:25Z   FIRST_TIME_CONTRIBUTOR dogsheep/github-to-sqlite/pulls/70

Some repos organize network dependents by a Package toggle. This PR adds the ability to page through those options and scrape underlying dependents.

github-to-sqlite 207052882 pull    
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/70/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1013506559 PR_kwDODFdgUs4skaNS 68 Add support for retrieving teams / members philwills 68329 open 0     0 2021-10-01T15:55:02Z 2021-10-01T15:59:53Z   FIRST_TIME_CONTRIBUTOR dogsheep/github-to-sqlite/pulls/68

Adds a method for retrieving all the teams within an organisation and all the members in those teams. The latter is stored as a join table team_members beteween teams and users.

github-to-sqlite 207052882 pull    
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/68/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
975161924 MDExOlB1bGxSZXF1ZXN0NzE2MzU3OTgy 66 Add --merged-by flag to pull-requests sub command sarcasticadmin 30531572 open 0     1 2021-08-20T00:57:55Z 2021-09-28T21:50:31Z   FIRST_TIME_CONTRIBUTOR dogsheep/github-to-sqlite/pulls/66

Description

Proposing a solution to the API limitation for merged_by in pull_requests. Specifically the following called out in the readme:

Note that the merged_by column on the pull_requests table will only be populated for pull requests that are loaded using the --pull-request option - the GitHub API does not return this field for pull requests that are loaded in bulk.

This approach might cause larger repos to hit rate limits called out in https://github.com/dogsheep/github-to-sqlite/issues/51 but seems to work well in the repos I tested and included below.

Old Behavior

  • Had to list out the pull-requests individually via multiple --pull-request flags

New Behavior

  • --merged-by flag for getting 'merge_by' information out of pull-requests without having to specify individual PR numbers.

Testing

Picking some repo that has more than one merger (datasette only has 1 😉 )

$ github-to-sqlite pull-requests ./github.db opnsense/tools --merged-by $ echo "select id, url, merged_by from pull_requests;" | sqlite3 ./github.db 83533612|https://github.com/opnsense/tools/pull/39|1915288 102632885|https://github.com/opnsense/tools/pull/43|1915288 149114810|https://github.com/opnsense/tools/pull/57|1915288 160394495|https://github.com/opnsense/tools/pull/64|1915288 163308408|https://github.com/opnsense/tools/pull/67|1915288 169723264|https://github.com/opnsense/tools/pull/69|1915288 171381422|https://github.com/opnsense/tools/pull/72|1915288 179938195|https://github.com/opnsense/tools/pull/77|1915288 196233824|https://github.com/opnsense/tools/pull/82|1915288 215289964|https://github.com/opnsense/tools/pull/93| 219696100|https://github.com/opnsense/tools/pull/97|1915288 223664843|https://github.com/opnsense/tools/pull/99| 228446172|https://github.com/opnsense/tools/pull/103|1915288 238930434|https://github.com/opnsense/tools/pull/110|1915288 255507110|https://github.com/opnsense/tools/pull/119|1915288 255980675|https://github.com/opnsense/tools/pull/120|1915288 261906770|https://github.com/opnsense/tools/pull/125| 263800503|https://github.com/opnsense/tools/pull/127|1915288 264038685|https://github.com/opnsense/tools/pull/128|1915288 264696704|https://github.com/opnsense/tools/pull/129|1915288 266660547|https://github.com/opnsense/tools/pull/130|1915288 273120409|https://github.com/opnsense/tools/pull/133|1915288 274370803|https://github.com/opnsense/tools/pull/135| 276600629|https://github.com/opnsense/tools/pull/139| 277303655|https://github.com/opnsense/tools/pull/141|1915288 293033714|https://github.com/opnsense/tools/pull/145| 294827649|https://github.com/opnsense/tools/pull/146| 295140008|https://github.com/opnsense/tools/pull/147|1915288 305690829|https://github.com/opnsense/tools/pull/150|9783985 307077931|https://github.com/opnsense/tools/pull/152|1915288 321782100|https://github.com/opnsense/tools/pull/155| 337265672|https://github.com/opnsense/tools/pull/160| 337267484|https://github.com/opnsense/tools/pull/161|1915288 368251763|https://github.com/opnsense/tools/pull/169| 428262505|https://github.com/opnsense/tools/pull/181| 437557011|https://github.com/opnsense/tools/pull/182|1915288 447079893|https://github.com/opnsense/tools/pull/185| 461822092|https://github.com/opnsense/tools/pull/191| 463290142|https://github.com/opnsense/tools/pull/193|1915288 470112962|https://github.com/opnsense/tools/pull/194|1915288 472644649|https://github.com/opnsense/tools/pull/195|1915288 488696898|https://github.com/opnsense/tools/pull/198| 513289902|https://github.com/opnsense/tools/pull/201| 522530265|https://github.com/opnsense/tools/pull/203| 564443347|https://github.com/opnsense/tools/pull/213| 597579516|https://github.com/opnsense/tools/pull/220|1915288 602860357|https://github.com/opnsense/tools/pull/221|1915288 608744738|https://github.com/opnsense/tools/pull/222|1915288 623279673|https://github.com/opnsense/tools/pull/228|1915288 664656182|https://github.com/opnsense/tools/pull/233| 664781786|https://github.com/opnsense/tools/pull/234|1915288 670683636|https://github.com/opnsense/tools/pull/235|1915288 683150764|https://github.com/opnsense/tools/pull/237| 685016233|https://github.com/opnsense/tools/pull/238| 687099825|https://github.com/opnsense/tools/pull/239|1915288 715705652|https://github.com/opnsense/tools/pull/244|1915288 715721248|https://github.com/opnsense/tools/pull/245|1915288 userid are now present for those PRs that were merged.

Without the flag the merged_by behavior remains missing as expected when get PRs bulk:

$ github-to-sqlite pull-requests ./github.db opnsense/tools $ echo "select id, url, merged_by from pull_requests;" | sqlite3 ./github.db 83533612|https://github.com/opnsense/tools/pull/39| 102632885|https://github.com/opnsense/tools/pull/43| 149114810|https://github.com/opnsense/tools/pull/57| 160394495|https://github.com/opnsense/tools/pull/64| 163308408|https://github.com/opnsense/tools/pull/67| 169723264|https://github.com/opnsense/tools/pull/69| 171381422|https://github.com/opnsense/tools/pull/72| 179938195|https://github.com/opnsense/tools/pull/77| 196233824|https://github.com/opnsense/tools/pull/82| 215289964|https://github.com/opnsense/tools/pull/93| 219696100|https://github.com/opnsense/tools/pull/97| 223664843|https://github.com/opnsense/tools/pull/99| 228446172|https://github.com/opnsense/tools/pull/103| 238930434|https://github.com/opnsense/tools/pull/110| 255507110|https://github.com/opnsense/tools/pull/119| 255980675|https://github.com/opnsense/tools/pull/120| 261906770|https://github.com/opnsense/tools/pull/125| 263800503|https://github.com/opnsense/tools/pull/127| 264038685|https://github.com/opnsense/tools/pull/128| 264696704|https://github.com/opnsense/tools/pull/129| 266660547|https://github.com/opnsense/tools/pull/130| 273120409|https://github.com/opnsense/tools/pull/133| 274370803|https://github.com/opnsense/tools/pull/135| 276600629|https://github.com/opnsense/tools/pull/139| 277303655|https://github.com/opnsense/tools/pull/141| 293033714|https://github.com/opnsense/tools/pull/145| 294827649|https://github.com/opnsense/tools/pull/146| 295140008|https://github.com/opnsense/tools/pull/147| 305690829|https://github.com/opnsense/tools/pull/150| 307077931|https://github.com/opnsense/tools/pull/152| 321782100|https://github.com/opnsense/tools/pull/155| 337265672|https://github.com/opnsense/tools/pull/160| 337267484|https://github.com/opnsense/tools/pull/161| 368251763|https://github.com/opnsense/tools/pull/169| 428262505|https://github.com/opnsense/tools/pull/181| 437557011|https://github.com/opnsense/tools/pull/182| 447079893|https://github.com/opnsense/tools/pull/185| 461822092|https://github.com/opnsense/tools/pull/191| 463290142|https://github.com/opnsense/tools/pull/193| 470112962|https://github.com/opnsense/tools/pull/194| 472644649|https://github.com/opnsense/tools/pull/195| 488696898|https://github.com/opnsense/tools/pull/198| 513289902|https://github.com/opnsense/tools/pull/201| 522530265|https://github.com/opnsense/tools/pull/203| 564443347|https://github.com/opnsense/tools/pull/213| 597579516|https://github.com/opnsense/tools/pull/220| 602860357|https://github.com/opnsense/tools/pull/221| 608744738|https://github.com/opnsense/tools/pull/222| 623279673|https://github.com/opnsense/tools/pull/228| 664656182|https://github.com/opnsense/tools/pull/233| 664781786|https://github.com/opnsense/tools/pull/234| 670683636|https://github.com/opnsense/tools/pull/235| 683150764|https://github.com/opnsense/tools/pull/237| 685016233|https://github.com/opnsense/tools/pull/238| 687099825|https://github.com/opnsense/tools/pull/239| 715705652|https://github.com/opnsense/tools/pull/244| 715721248|https://github.com/opnsense/tools/pull/245|

Individual PRs passed via --pull-request flag behaves as expected (unchanged):

$ github-to-sqlite pull-requests ./github.db opnsense/tools --pull-request 39 --pull-request 237 $ echo "select id, url, merged_by from pull_requests;" | sqlite3 ./github.db 83533612|https://github.com/opnsense/tools/pull/39|1915288 683150764|https://github.com/opnsense/tools/pull/237|

Picking 1 PR that has a merged_by (39) and one that does not (237)

github-to-sqlite 207052882 pull    
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/66/reactions",
    "total_count": 3,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 1,
    "rocket": 0,
    "eyes": 0
}
0  
981690086 MDExOlB1bGxSZXF1ZXN0NzIxNjg2NzIx 67 Replacing step ID key with step_id jshcmpbll 16374374 open 0     0 2021-08-28T01:26:41Z 2021-08-28T01:27:00Z   FIRST_TIME_CONTRIBUTOR dogsheep/github-to-sqlite/pulls/67

Workflows that have an id in any step result in the following error when running workflows:

e.g.github-to-sqlite workflows github.db nixos/nixpkgs

Traceback (most recent call last): File "/usr/local/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1668, in invokeTraceback (most recent call last): File "/usr/local/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in call return self.main(args, kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 763, in invoke return __callback(args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/github_to_sqlite/cli.py", line 601, in workflows utils.save_workflow(db, repo_id, filename, content) File "/usr/local/lib/python3.8/dist-packages/github_to_sqlite/utils.py", line 865, in save_workflow db["steps"].insert_all( File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 2596, in insert_all self.insert_chunk( File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 2378, in insert_chunk result = self.db.execute(query, params) File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 419, in execute return self.conn.execute(sql, parameters) sqlite3.IntegrityError: datatype mismatch ```

  • Information about the ID key in a step for GHA
  • An example workflow from a public repo

Changes

I'm proposing that the key for id in step is replaced with step_id so that it no longer interferes with the table id for tracking the record.

Special thanks to @sarcasticadmin @egiffen and @ruebenramirez for helping a bit on this 😄

github-to-sqlite 207052882 pull    
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/67/reactions",
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
771872303 MDExOlB1bGxSZXF1ZXN0NTQzMjQ2NTM1 59 Remove unneeded exists=True for -a/--auth flag. frosencrantz 631242 closed 0     3 2020-12-21T06:03:55Z 2021-05-22T14:06:19Z 2021-05-19T16:08:12Z CONTRIBUTOR dogsheep/github-to-sqlite/pulls/59

The file does not need to exist when using an environment variable.

github-to-sqlite 207052882 pull    
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
797108702 MDExOlB1bGxSZXF1ZXN0NTY0MTcyMTQw 61 fixing typo in get cli help text daniel-butler 22578954 closed 0     1 2021-01-29T18:57:04Z 2021-05-19T16:07:09Z 2021-05-19T16:07:09Z CONTRIBUTOR dogsheep/github-to-sqlite/pulls/61
github-to-sqlite 207052882 pull    
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/61/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
681228542 MDExOlB1bGxSZXF1ZXN0NDY5NjUxNzMy 48 Add pull requests adamjonas 755825 closed 0     2 2020-08-18T17:58:44Z 2020-11-29T23:51:09Z 2020-11-29T23:51:09Z CONTRIBUTOR dogsheep/github-to-sqlite/pulls/48

ref #46

Issues don't have merge information on them, which means that PRs need to be pulled separately.

Did my best to mimic the API of issues.

github-to-sqlite 207052882 pull    
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
516763727 MDExOlB1bGxSZXF1ZXN0MzM1OTgwMjQ2 8 stargazers command, refs #4 simonw 9599 closed 0     5 2019-11-03T00:37:36Z 2020-05-02T20:00:27Z 2020-05-02T20:00:26Z MEMBER dogsheep/github-to-sqlite/pulls/8

Needs tests. Refs #4.

github-to-sqlite 207052882 pull    
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);
Powered by Datasette · Queries took 110.026ms · About: github-to-sqlite
  • Sort ascending
  • Sort descending
  • Facet by this
  • Hide this column
  • Show all columns
  • Show not-blank rows