issues

Table actions

2 rows where state = "open", type = "issue" and user = 601708 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

type 1

  • issue · 2 ✖

state 1

  • open · 2 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1740026046 I_kwDOCGYnMM5ntrC- 556 Support storing incrementally piped values mcint 601708 open 0     1 2023-06-04T00:45:23Z 2023-06-04T01:21:15Z   CONTRIBUTOR  

I'm trying to use sqlite-utils to data generated incrementally. There are a few aspects of this that I don't currently know how to handle. I would like an option to apply writes incrementally, line-by-line as they are received. I would like an option to echo incremental progress. And, it would be nice to have

In particular, I'm using CoreLocationCLI -w -j to generate, newline-delimited JSON.

One variant of the command

stdbuf -oL CoreLocationCLI -w -j | pee 'sqlite-utils insert loc.db loc -' nl

pee, from moreutils, is like tee but spawns and pipes to the processes created by invoking each of its arguments, so, for gratuitous demonstration, pee 'sponge out.log' cat would behave like tee.

It looks like I can get what I want with: stdbuf -oL CoreLocationCLI -w -j | while read line; do <<<"$line" sqlite-utils insert loc.db loc -; echo "$line"; done | nl

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/556/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1524431805 I_kwDODEm0Qs5a3Pu9 72 Import thread, including self- and others' replies mcint 601708 open 0     0 2023-01-08T09:51:06Z 2023-01-08T09:51:06Z   NONE  

statuses-lookup, home-timeline, mentions (only for auth'ed user) don't cover this.

twitter-to-sqlite fetch-thread tw-group1.db 1234123412341234

twitter-to-sqlite focuses on archiving users, but does not easily support archiving conversations or community activity.

For reference, this is implemented in twarc, using a search, optionally recursively.

Other research suggests that this formerly, or currently, requires a search query, use of undocumented related_results api, or with requested inclusion of newer conversation_id with subsequent query.

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/72/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);