issue_comments

Table actions

5 rows where issue = 732674148 sorted by updated_at descending

✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 2

author_association 2

issue 1

  • Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows · 5 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1260909128 https://github.com/simonw/datasette/issues/1062#issuecomment-1260909128 https://api.github.com/repos/simonw/datasette/issues/1062 IC_kwDOBm6k_c5LJ_JI fgregg 536941 2022-09-28T13:22:53Z 2022-09-28T14:09:54Z CONTRIBUTOR

if you went this route:

python with sqlite_timelimit(conn, time_limit_ms): c.execute(query) for chunk in c.fetchmany(chunk_size): yield from chunk

then time_limit_ms would probably have to be greatly extended, because the time spent in the loop will depend on the downstream processing.

i wonder if this was why you were thinking this feature would need a dedicated connection?

reading more, there's no real limit i can find on the number of active cursors (or more precisely active prepared statements objects, because sqlite doesn't really have cursors).

maybe something like this would be okay?

python with sqlite_timelimit(conn, time_limit_ms): c.execute(query) # step through at least one to evaluate the statement, not sure if this is necessary yield c.execute.fetchone() for chunk in c.fetchmany(chunk_size): yield from chunk

this seems quite weird that there's not more of limit of the number of active prepared statements, but i haven't been able to find one.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows 732674148  
1260829829 https://github.com/simonw/datasette/issues/1062#issuecomment-1260829829 https://api.github.com/repos/simonw/datasette/issues/1062 IC_kwDOBm6k_c5LJryF fgregg 536941 2022-09-28T12:27:19Z 2022-09-28T12:27:19Z CONTRIBUTOR

for teaching register_output_renderer to stream it seems like the two options are to

  1. a nested query technique to paginate through
  2. a fetching model that looks like something python with sqlite_timelimit(conn, time_limit_ms): c.execute(query) for chunk in c.fetchmany(chunk_size): yield from chunk currently db.execute is not a generator, so this would probably need a new method?
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows 732674148  
1068327874 https://github.com/simonw/datasette/issues/1062#issuecomment-1068327874 https://api.github.com/repos/simonw/datasette/issues/1062 IC_kwDOBm6k_c4_rWPC simonw 9599 2022-03-15T18:33:49Z 2022-03-15T18:33:49Z OWNER

I can get regular .json to stream too, using the pattern described in this TIL: https://til.simonwillison.net/python/output-json-array-streaming

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows 732674148  
853566011 https://github.com/simonw/datasette/issues/1062#issuecomment-853566011 https://api.github.com/repos/simonw/datasette/issues/1062 MDEyOklzc3VlQ29tbWVudDg1MzU2NjAxMQ== simonw 9599 2021-06-03T05:07:42Z 2021-06-03T05:07:42Z OWNER

Implementing this would make #1356 a whole lot more interesting.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows 732674148  
719031901 https://github.com/simonw/datasette/issues/1062#issuecomment-719031901 https://api.github.com/repos/simonw/datasette/issues/1062 MDEyOklzc3VlQ29tbWVudDcxOTAzMTkwMQ== simonw 9599 2020-10-29T21:25:54Z 2020-10-29T21:25:54Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows 732674148  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);