issue_comments

Table actions

3 rows where author_association = "CONTRIBUTOR" and issue = 749283032 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

issue 1

  • register_output_renderer() should support streaming data · 3 ✖

author_association 1

  • CONTRIBUTOR · 3 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1105642187 https://github.com/simonw/datasette/issues/1101#issuecomment-1105642187 https://api.github.com/repos/simonw/datasette/issues/1101 IC_kwDOBm6k_c5B5sLL eyeseast 25778 2022-04-21T18:59:08Z 2022-04-21T18:59:08Z CONTRIBUTOR

Ha! That was your idea (and a good one).

But it's probably worth measuring to see what overhead it adds. It did require both passing in the database and making the whole thing async.

Just timing the queries themselves:

  1. Using AsGeoJSON(geometry) as geometry takes 10.235 ms
  2. Leaving as binary takes 8.63 ms

Looking at the network panel:

  1. Takes about 200 ms for the fetch request
  2. Takes about 300 ms

I'm not sure how best to time the GeoJSON generation, but it would be interesting to check. Maybe I'll write a plugin to add query times to response headers.

The other thing to consider with async streaming is that it might be well-suited for a slower response. When I have to get the whole result and send a response in a fixed amount of time, I need the most efficient query possible. If I can hang onto a connection and get things one chunk at a time, maybe it's ok if there's some overhead.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 register_output_renderer() should support streaming data 749283032  
1105588651 https://github.com/simonw/datasette/issues/1101#issuecomment-1105588651 https://api.github.com/repos/simonw/datasette/issues/1101 IC_kwDOBm6k_c5B5fGr eyeseast 25778 2022-04-21T18:15:39Z 2022-04-21T18:15:39Z CONTRIBUTOR

What if you split rendering and streaming into two things:

  • render is a function that returns a response
  • stream is a function that sends chunks, or yields chunks passed to an ASGI send callback

That way current plugins still work, and streaming is purely additive. A stream function could get a cursor or iterator of rows, instead of a list, so it could more efficiently handle large queries.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 register_output_renderer() should support streaming data 749283032  
869191854 https://github.com/simonw/datasette/issues/1101#issuecomment-869191854 https://api.github.com/repos/simonw/datasette/issues/1101 MDEyOklzc3VlQ29tbWVudDg2OTE5MTg1NA== eyeseast 25778 2021-06-27T16:42:14Z 2021-06-27T16:42:14Z CONTRIBUTOR

This would really help with this issue: https://github.com/eyeseast/datasette-geojson/issues/7

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 register_output_renderer() should support streaming data 749283032  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);