issue_comments

@fgregg, I am happy to do that and just could not find a way to create issues at your fork repo.

@yejiyang why don’t you move this discussion to my fork to spare simon’s notifications

@fgregg Thanks for your reply. I tried to use your fork branch datasette = {url = "https://github.com/fgregg/datasette/archive/refs/heads/no_limit_csv_publish.zip"} and got error - TypeError: 'str' object is not callable. I used the same templates as in your branch here .

INFO: 127.0.0.1:47232 - "GET /-/static/sql-formatter-2.3.3.min.js HTTP/1.1" 200 OK Traceback (most recent call last): File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py", line 1632, in route_path response = await view(request, send) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py", line 1814, in async_view_fn response = await async_call_with_supported_arguments( File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/utils/__init__.py", line 1016, in async_call_with_supported_arguments return await fn(*call_with) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/views/table.py", line 673, in table_view response = await table_view_traced(datasette, request) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/views/table.py", line 822, in table_view_traced await datasette.render_template( File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py", line 1307, in render_template return await template.render_async(template_context) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1324, in render_async return self.environment.handle_exception() File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception raise rewrite_traceback_stack(source=source) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1321, in <listcomp> [n async for n in self.root_render_func(ctx)] # type: ignore File "templates/table.html", line 1, in top-level template code {% extends "base.html" %} File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/templates/base.html", line 62, in top-level template code {% block content %} File "templates/table.html", line 24, in block 'content' <div class="page-header" style="border-color: #{{ database_color(database) }}"> TypeError: 'str' object is not callable INFO: 127.0.0.1:47228 - "GET /zeropm-v0-0-3/api_ready_query HTTP/1.1" 500 Internal Server Error

hi @yejiyang, have your tried using my fork of datasette: https://github.com/fgregg/datasette/tree/no_limit_csv_publish

Hi @fgregg, I came across this issue and found your setup at labordata.bunkum.us can help me with a research project at https://database.zeropm.eu/. I really like the approach here when dealing with a custom SQL query returning more than 1000 rows: 1) At the table in HTML page, only first 1000 rows displayed; 2) When click the "Download this data as a CSV Spreadsheet(All Rows)" button, a csv with ALL ROWS (could be > 100 Mb) get downloaded.

I am trying to repeat the setup but have yet to be successful so far. What I tried: 1) copy the query.html & table.html templates from this github repo and use it my project 2) use the same datasette version 1.0a2.

Do you know what else I should try to set it correctly? I appreciate your help.

@simonw I would like to use this opportunity to thank you for developing & maintaining such an amazing project. I introduce your datasette to several projects in my institute. I am also interested in your cloud version.

I think that paginating, even in javascript, could be very helpful. Maybe render json or csv into the page and let javascript loading that into the dom?

i increased the max_returned_row, because I have some scripts that get CSVs from this site, and this makes doing pagination of CSVs less annoying for many cases. i know that's streaming csvs is something you are hoping to address in 1.0. let me know if there's anything i can do to help with that.

as for what if anything can be done about the size of the dom, I don't have any ideas right now, but i'll poke around.

https://labordata.bunkum.us/-/settings shows max_returned_rows had been increased to 5,000 for that instance - the default of 1,000 would help a bit here.

Any thoughts on how Datasette could handle this kind of thing better?