issues

Table actions

5 rows where user = 601708 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: comments, updated_at, closed_at, author_association, body, created_at (date), updated_at (date), closed_at (date)

type 2

state 2

id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1740026046 I_kwDOCGYnMM5ntrC- 556 Support storing incrementally piped values mcint 601708 open 0     1 2023-06-04T00:45:23Z 2023-06-04T01:21:15Z   CONTRIBUTOR  

I'm trying to use sqlite-utils to data generated incrementally. There are a few aspects of this that I don't currently know how to handle. I would like an option to apply writes incrementally, line-by-line as they are received. I would like an option to echo incremental progress. And, it would be nice to have

In particular, I'm using CoreLocationCLI -w -j to generate, newline-delimited JSON.

One variant of the command

stdbuf -oL CoreLocationCLI -w -j | pee 'sqlite-utils insert loc.db loc -' nl

pee, from moreutils, is like tee but spawns and pipes to the processes created by invoking each of its arguments, so, for gratuitous demonstration, pee 'sponge out.log' cat would behave like tee.

It looks like I can get what I want with: stdbuf -oL CoreLocationCLI -w -j | while read line; do <<<"$line" sqlite-utils insert loc.db loc -; echo "$line"; done | nl

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/556/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1524431805 I_kwDODEm0Qs5a3Pu9 72 Import thread, including self- and others' replies mcint 601708 open 0     0 2023-01-08T09:51:06Z 2023-01-08T09:51:06Z   NONE  

statuses-lookup, home-timeline, mentions (only for auth'ed user) don't cover this.

twitter-to-sqlite fetch-thread tw-group1.db 1234123412341234

twitter-to-sqlite focuses on archiving users, but does not easily support archiving conversations or community activity.

For reference, this is implemented in twarc, using a search, optionally recursively.

Other research suggests that this formerly, or currently, requires a search query, use of undocumented related_results api, or with requested inclusion of newer conversation_id with subsequent query.

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/72/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
923697888 MDU6SXNzdWU5MjM2OTc4ODg= 278 Support db as first parameter before subcommand, or as environment variable mcint 601708 closed 0     3 2021-06-17T09:26:29Z 2021-06-20T22:39:57Z 2021-06-18T15:43:19Z CONTRIBUTOR  
sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/278/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
923602693 MDU6SXNzdWU5MjM2MDI2OTM= 276 support small help flag -h mcint 601708 closed 0     0 2021-06-17T07:59:31Z 2021-06-18T14:56:59Z 2021-06-18T14:56:59Z CONTRIBUTOR  
sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/276/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
923612361 MDExOlB1bGxSZXF1ZXN0NjcyMzU5NjA5 277 add -h support closes #276 mcint 601708 closed 0     2 2021-06-17T08:08:26Z 2021-06-18T14:56:59Z 2021-06-18T14:56:59Z CONTRIBUTOR simonw/sqlite-utils/pulls/277

This appears to be the canonical solution.

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/277/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);