issue_comments

Table actions

5 rows where author_association = "MEMBER", issue = 590669793 and user = 9599 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: updated_at (date)

user 1

  • simonw · 5 ✖

issue 1

  • Feature: record history of follower counts · 5 ✖

author_association 1

  • MEMBER · 5 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
607019151 https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607019151 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 MDEyOklzc3VlQ29tbWVudDYwNzAxOTE1MQ== simonw 9599 2020-04-01T04:11:10Z 2020-04-01T04:11:10Z MEMBER

In testing this collects a LOT of data. I'm going to skip tracking favourites_count and statuses_count and just track followers, friends and listed instead.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Feature: record history of follower counts 590669793  
607011421 https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011421 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 MDEyOklzc3VlQ29tbWVudDYwNzAxMTQyMQ== simonw 9599 2020-04-01T03:47:37Z 2020-04-01T03:55:08Z MEMBER

Actually a single table with a type integer ID referencing a count_history_types table would better match the way I implemented the since_ids table:

https://github.com/dogsheep/twitter-to-sqlite/blob/4b6c8d8c1cc6fefdb566ec8506157133f47c569a/twitter_to_sqlite/utils.py#L331-L341

In which case the compound primary key would be type, user, datetime

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Feature: record history of follower counts 590669793  
607011972 https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011972 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 MDEyOklzc3VlQ29tbWVudDYwNzAxMTk3Mg== simonw 9599 2020-04-01T03:49:02Z 2020-04-01T03:50:01Z MEMBER

I want the datetime value to look like 2020-04-01T03:34:58+00:00 (the format returned by the Twitter API which I am storing in other tables at the moment). ```

datetime.utcnow().isoformat().split('.')[0] + '+00:00' '2020-04-01T03:49:52+00:00' ```

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Feature: record history of follower counts 590669793  
606307376 https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307376 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 MDEyOklzc3VlQ29tbWVudDYwNjMwNzM3Ng== simonw 9599 2020-03-30T23:35:40Z 2020-03-30T23:39:15Z MEMBER

I think five separate tables:

  • followers_count_history
  • friends_count_history
  • listed_count_history
  • favourites_count_history
  • statuses_count_history

Each with the following structure:

  • datetime (ISO UTC)
  • user (ID, foreign key to users)
  • count (integer)

I'm tempted to have a compound primary key here - user, datetime

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Feature: record history of follower counts 590669793  
606307019 https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307019 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 MDEyOklzc3VlQ29tbWVudDYwNjMwNzAxOQ== simonw 9599 2020-03-30T23:34:27Z 2020-03-30T23:34:27Z MEMBER

The count properties available for a user are:

  • followers_count
  • friends_count
  • listed_count
  • favourites_count
  • statuses_count

May as well track history for all of them? Should be pretty cheap to store.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Feature: record history of follower counts 590669793  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);