html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/issues/1727#issuecomment-1258129113,https://api.github.com/repos/simonw/datasette/issues/1727,1258129113,IC_kwDOBm6k_c5K_YbZ,536941,2022-09-26T14:30:11Z,2022-09-26T14:48:31Z,CONTRIBUTOR,"from your analysis, it seems like the GIL is blocking on loading of the data from sqlite to python, (particularly in the `fetchmany` call)

this is probably a simplistic idea, but what if you had the python code in the `execute` method iterate over the cursor and yield out rows or small chunks of rows.

something like:             
```python
            with sqlite_timelimit(conn, time_limit_ms):
                try:
                    cursor = conn.cursor()
                    cursor.execute(sql, params if params is not None else {})
                except:
                    ...
            max_returned_rows = self.ds.max_returned_rows
            if max_returned_rows == page_size:
                max_returned_rows += 1
                if max_returned_rows and truncate:
                    for i, row in enumerate(cursor):
                        yield row
                        if i == max_returned_rows - 1:
                            break
                else:
                    for row in cursor:
                        yield row
                    truncated = False                  
```

this kind of thing works well with a postgres server side cursor, but i'm not sure if it will hold for sqlite. 

you would still spend about the same amount of time in python and would be contending for the gil, but it would be could be non blocking.

depending on the data flow, this could also some benefit for memory. (data stays in more compact sqlite-land until you need it)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,