Adding &per_page=100 would reduce the number of API requests by 3x.

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (9286c1b) 95.77% compared to head (1698a9d) 95.72%. Report is 1 commits behind head on main.

:exclamation: Current head 1698a9d differs from pull request most recent head 61c6e26. Consider uploading reports for the commit 61c6e26 to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

If I uncheck expand labels in the Advanced CSV export dialog, the error does not occur. Re-checking that box and re-running the export does cause the error to occur.

Any progress on this? It would be very helpful on my end as well. Thanks!

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (452a587) 92.69% compared to head (c88414b) 92.69%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (452a587) 92.69% compared to head (eec10df) 92.69%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

thanks for documenting this @bendnorman! got stuck at exactly the same point gcloud builds submit ... returned non-zero exit status 1, without a clue why this was happening. i now managed to get the github action to deploy datasette by assigning the following roles to the service account: roles/run.admin, roles/storage.admin, roles/cloudbuild.builds.builder, roles/viewer, roles/iam.serviceAccountUser.

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (347fdc8) 95.74% compared to head (1f14df1) 95.74%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (622c3a5) 95.77% compared to head (b1a6076) 95.77%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

FWIW, code mirror 6 now has this standard although if you want table-specific suggestions, you'd have to handle parsing out which table the user is querying yourself.

Using the Counties example, I was able to pull out the MakePoint method as

MakePoint(cast(rm_rnb_history_pres.rx_lng as float), cast(rm_rnb_history_pres.rx_lat as float)) as geometry

which worked, giving me a geometry column.

gave

I believe it's the cast to float that does the trick. Prior to using the cast, I also received a 'wrong number of arguments' eror.

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (452a587) 92.69% compared to head (be4d0f0) 92.69%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@fgregg, I am happy to do that and just could not find a way to create issues at your fork repo.

@fgregg Thanks for your reply. I tried to use your fork branch datasette = {url = "https://github.com/fgregg/datasette/archive/refs/heads/no_limit_csv_publish.zip"} and got error - TypeError: 'str' object is not callable. I used the same templates as in your branch here .

INFO: 127.0.0.1:47232 - "GET /-/static/sql-formatter-2.3.3.min.js HTTP/1.1" 200 OK Traceback (most recent call last): File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py", line 1632, in route_path response = await view(request, send) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py", line 1814, in async_view_fn response = await async_call_with_supported_arguments( File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/utils/__init__.py", line 1016, in async_call_with_supported_arguments return await fn(*call_with) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/views/table.py", line 673, in table_view response = await table_view_traced(datasette, request) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/views/table.py", line 822, in table_view_traced await datasette.render_template( File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py", line 1307, in render_template return await template.render_async(template_context) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1324, in render_async return self.environment.handle_exception() File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception raise rewrite_traceback_stack(source=source) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1321, in <listcomp> [n async for n in self.root_render_func(ctx)] # type: ignore File "templates/table.html", line 1, in top-level template code {% extends "base.html" %} File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/templates/base.html", line 62, in top-level template code {% block content %} File "templates/table.html", line 24, in block 'content' <div class="page-header" style="border-color: #{{ database_color(database) }}"> TypeError: 'str' object is not callable INFO: 127.0.0.1:47228 - "GET /zeropm-v0-0-3/api_ready_query HTTP/1.1" 500 Internal Server Error

Hi @fgregg, I came across this issue and found your setup at labordata.bunkum.us can help me with a research project at https://database.zeropm.eu/. I really like the approach here when dealing with a custom SQL query returning more than 1000 rows: 1) At the table in HTML page, only first 1000 rows displayed; 2) When click the "Download this data as a CSV Spreadsheet(All Rows)" button, a csv with ALL ROWS (could be > 100 Mb) get downloaded.

I am trying to repeat the setup but have yet to be successful so far. What I tried: 1) copy the query.html & table.html templates from this github repo and use it my project 2) use the same datasette version 1.0a2.

Do you know what else I should try to set it correctly? I appreciate your help.

@simonw I would like to use this opportunity to thank you for developing & maintaining such an amazing project. I introduce your datasette to several projects in my institute. I am also interested in your cloud version.

That worked!

Codecov Report

All modified lines are covered by tests :white_check_mark:

Comparison is base (3feed1f) 92.46% compared to head (8ae479c) 92.69%. Report is 112 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

All modified lines are covered by tests :white_check_mark:

Comparison is base (6ed7908) 92.69% compared to head (0135e7c) 92.68%. Report is 14 commits behind head on main.

:exclamation: Current head 0135e7c differs from pull request most recent head 18b48f8. Consider uploading reports for the commit 18b48f8 to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Hello! Resurrecting this issue since we're running into something similar with data.catalyst.coop as our database files have ballooned up to several GB. Our Cloud Run revisions now require huge amounts of RAM to start up without receiving a SIGBUS.

I'd love to see this fix merged in. It sounds like we want to make the immutable/read-only mode decision more flexible before doing so, so that we can use ro in Docker and immutable outside. If that sounds right, I'm happy to take a crack at adding that as a command-line flag or something that gets set automatically based on the expected execution environment.

@simonw Any chance we can get this tiny patch merged for an upcoming release?

Never mind

When I created the connection using sqlite_utils.Database(path)

I just needed to add the following statement right after and it did the trick

self.db.conn.execute("PRAGMA foreign_keys = ON")

Hope this helps people in the future 👍

This would be awesome to have for multi-gig tsv and csv files! I'm currently looking at a 10 hour countdown for one such important. Not a problem because I'm lazy and happy to let it run and check on it tomorrow..

/me munches popcorn at a furious rate, utterly entralled

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.02% :tada:

Comparison is base (6ed7908) 92.69% compared to head (4e6a341) 92.72%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage has no change and project coverage change: -0.04% :warning:

Comparison is base (6ed7908) 92.69% compared to head (fe5f881) 92.66%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.03% :warning:

Comparison is base (6ed7908) 92.69% compared to head (fc7dbe0) 92.67%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Can confirm that this bug can be reproduced as follows:

docker run datasetteproject/datasette datasette serve --reload

which produces the following output:

Starting monitor for PID 10. Error: Invalid value for '[FILES]...': Path 'serve' does not exist.

This is such a well thought out contribution. I don't think I've seen such a thoroughly considered PR on any project in recent memory.

Codecov Report

Patch coverage: 95.00% and project coverage change: -0.04% :warning:

Comparison is base (a4c96d0) 92.69% compared to head (659dcbd) 92.66%.

:exclamation: Current head 659dcbd differs from pull request most recent head acca338. Consider uploading reports for the commit acca338 to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (5d123f0) 95.77% compared to head (b86374f) 95.77%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (c263704) 92.68% compared to head (e1c5a9d) 92.69%.

:umbrella: View full report in Codecov by Sentry.

:loudspeaker: Have feedback on the report? Share it here.

Makes sense, thanks for explaining!

Codecov Report

Patch and project coverage have no change.

Comparison is base (05707aa) 92.68% compared to head (d75b519) 92.68%. Report is 7 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.

:loudspeaker: Have feedback on the report? Share it here.

@maxhawkins curious why you didn't use the stdlib mailbox to parse the mbox files?

Mailbox parses the entire mbox into memory. Using the lower level library lets us stream the emails in one at a time to support larger archives. Both libraries are in the stdlib.

I ran into the same issue on Windows using

datasette publish cloudrun mydatabase.db --service=my-database do do a google cloud publish.

@Rik-de-Kort your fix worked perfectly! Thanks!

I can always go back and delete the temp directories myself :)

@maxhawkins curious why you didn't use the stdlib mailbox to parse the mbox files?

Codecov Report

Patch and project coverage have no change.

Comparison is base (05707aa) 92.68% compared to head (7c9df6e) 92.68%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (fd083e3) 92.68% compared to head (a481ebd) 92.68%. Report is 1 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Hey @simonw I hit the same issue as mentionned by @chmaynard on a fresh install, "/mnt/fixtures.db" doesn't seem to exist in the docker image

Codecov Report

Patch coverage: 26.66% and project coverage change: -0.17% :warning:

Comparison is base (30b28c8) 92.84% compared to head (5eddf34) 92.68%.

:exclamation: Current head 5eddf34 differs from pull request most recent head 6321c9c. Consider uploading reports for the commit 6321c9c to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.03% :tada:

Comparison is base (2e28258) 92.82% compared to head (3e49fd3) 92.85%. Report is 3 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.03% :warning:

Comparison is base (2e28258) 92.82% compared to head (73489ca) 92.79%. Report is 1 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (2e28258) 92.82% compared to head (c81b024) 92.82%.

:exclamation: Current head c81b024 differs from pull request most recent head 3529c9c. Consider uploading reports for the commit 3529c9c to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 75.00% and project coverage change: +0.05% :tada:

Comparison is base (527cec6) 92.76% compared to head (a5cbf80) 92.81%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (527cec6) 92.76% compared to head (2e45686) 92.76%.

:exclamation: Current head 2e45686 differs from pull request most recent head 5dfa305. Consider uploading reports for the commit 5dfa305 to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Something to think about, but I hate how long the url is when sharing a custom SQL query. Would it be possible to hash the query and state of a page instead so the url is more manageable? The mapping from hash to query would have to be stored in order to recover/lookup the page after sharing.

It's not uncommon to have things like this currently:

https://global-power-plants.datasettes.com/global-power-plants?sql=select+rowid%2C+country%2C+country_long%2C+name%2C+gppd_idnr%2C+capacity_mw%2C+latitude%2C+longitude%2C+primary_fuel%2C+other_fuel1%2C+other_fuel2%2C+other_fuel3%2C+commissioning_year%2C+owner%2C+source%2C+url%2C+geolocation_source%2C+wepp_id%2C+year_of_capacity_data%2C+generation_gwh_2013%2C+generation_gwh_2014%2C+generation_gwh_2015%2C+generation_gwh_2016%2C+generation_gwh_2017%2C+generation_gwh_2018%2C+generation_gwh_2019%2C+generation_data_source%2C+estimated_generation_gwh_2013%2C+estimated_generation_gwh_2014%2C+estimated_generation_gwh_2015%2C+estimated_generation_gwh_2016%2C+estimated_generation_gwh_2017%2C+estimated_generation_note_2013%2C+estimated_generation_note_2014%2C+estimated_generation_note_2015%2C+estimated_generation_note_2016%2C+estimated_generation_note_2017+from+%5Bglobal-power-plants%5D+order+by+rowid+limit+101

I'm thinking a plugin like https://datasette.io/plugins/datasette-query-files, but could be created and managed from the UI (with the right permissions).

If I may, the "path-like" configuration is great but one thing that would be even greater: allowing the same configuration to be provided using environment variables.

For instance:

datasette -s plugins.datasette-complex-plugin.configs '{"foo": [1,2,3], "bar": "baz"}'

could also be provided using:

export DS_PLUGINS_DATASETTE-COMPLEX-PLUGIN_CONFIGS='{"foo": [1,2,3], "bar": "baz"}' datasette

(I do not like mixing - and _ in env vars but I do not have a best easily reversible example at the moment)

FYI, you could take some inspiration from another great open source data project, Metabase: https://www.metabase.com/docs/latest/configuring-metabase/config-file https://www.metabase.com/docs/latest/configuring-metabase/environment-variables

That's fair. The best idea I can think of is that if a plugin wanted to limit intensive queries, it could add LIMITs or something. A hook that gives you visibility of queries and maybe the option to reject felt a little more limited than the existing plugin hooks, so I was trying to think of what else one might want to do while looking at to-be-run queries.

But without a real motivating example, I see why you don't want to add that.

@simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.

Codecov Report

Patch and project coverage have no change.

Comparison is base (bdf59eb) 92.73% compared to head (b895cd2) 92.73%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage has no change and project coverage change: +0.01% :tada:

Comparison is base (17ec309) 92.71% compared to head (a651d3c) 92.73%. Report is 2 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 76.47% and project coverage change: +0.10% :tada:

Comparison is base (943df09) 92.06% compared to head (db720cd) 92.17%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks for all this! You're totally right that the ASGI option is doable, if a bit low level and coupled to the current URI design. I'm totally fine with that being the final answer.

process_view is interesting and in the general direction of what I had in mind.

A somewhat less powerful idea: Is there value in giving a hook for just the query that's about to be run? Maybe I'm thinking a little narrowly about this problem I decided I wanted to solve, but I could see other uses for a hook of the sketch below:

def prepare_query(database, table, query): """Modify query that is about to be run in some way. Return the (possibly rewritten) query to run, or None to disallow running the query""" (Maybe you actually want to return a tuple so there can be an error message when you disallow, or something.)

Maybe it's too narrowly useful and some of the other pieces of datasette obviate some of these ideas, but off the cuff I could imagine using it to: * Require a LIMIT. Either fail the query or add the limit if it's not there. * Do logging, like my usecase. * Do other analysis on whether you want to allow the query to run; a linter? query complexity?

Definitely feel free to say no, or not now. This is all me just playing around with what datasette and its plugin architecture can do with toy ideas, so don't let me push you to commit to a hook you don't feel confident fits well in the design.

@simonw, since you're referencing "rowid" column by name, I just want to note that there may be an existing rowid column with completely different semantics (https://www.sqlite.org/lang_createtable.html#rowid), which is likely to break this logic. I don't see a good way to detect a proper "rowid" name short of checking if there is a field with that name and using the alternative (_rowid_ or oid), which is not ideal, but may work.

In terms of the original issue, maybe a way to deal with it is to use rowid by default and then use primary key for WITHOUT ROWID tables (as they are guaranteed to be not null), but I suspect it may require significant changes to the API (and doesn't fully address the issue of what value to pass to indicate NULL when editing records). Would it make sense to generate a random string to indicate NULL values when editing?

This also makes it simple to separate out secrets.

datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml

settings.yaml settings: default_page_size: 10 max_returned_rows: 3000 sql_time_limit_ms": 8000 plugins: datasette-ripgrep: path: /usr/local/lib/python3.11/site-packages

secrets.yaml plugins: datasette-auth-github: client_secret: SUCH_SECRET

db-docs.yaml databases: docs: permissions: create-table: id: editor

db-fixtures.yaml databases: fixtures: tables: no_primary_key: hidden: true queries: neighborhood_search: sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood; title: Search neighborhoods description_html: |- <p>This demonstrates <em>basic</em> LIKE search

https://pypi.org/project/deep-chainmap/

https://docs.python.org/3/library/collections.html#collections.ChainMap

To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate "settings" from "configuration" is, I believe, an uphill fight. Even separating db/table-scope settings from pure descriptive metadata is not always easy. Like, do canned queries belong to database metadata or to settings? Do I need two separate files for this?

One pragmatic solution I used in a project is stacking yaml configuration files. Basically, have an arbitrary number of yaml or json settings files that you load in a specified order. Every file adds to the corresponding settings in the earlier-loaded file (if it already existed). I implemented this myself but found later that there is an existing Python "cascading dict" type of thing, I forget what it's called. There is a bit of a challenge deciding whether there is "replacement" or "addition" (I think I pragmatically ran update on the second level of the dict but better solutions are certainly possible).

This way, one allows separation of settings into different blocks, while not imposing a specific idea of what belongs where that might not apply equally to all cases.

Codecov Report

Patch and project coverage have no change.

Comparison is base (943df09) 92.06% compared to head (3a97755) 92.06%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 92.85% and project coverage change: -0.07% :warning:

Comparison is base (1dc6b5a) 95.82% compared to head (2915050) 95.76%. Report is 1 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (943df09) 92.06% compared to head (6b617f7) 92.06%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (943df09) 92.06% compared to head (e2be94a) 92.06%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

I see :) yeah, I’m on the stable version installed from homebrew on macOS

It looks very nice now. Finally, no more manual installation of plugins one by one. Thank you, Simon! ❤️

Yes, using this approach(datasette install -r requirements.txt) will result in more consistency.

I'm curious about the results of the datasette plugins --all command. Where will we use the output of this command? Will it include configuration information for these plugins in the future? If so, will we need to consider the configuration of these plugins in addition to installing them on different computers?

You can also install this with pip using this oneliner:

pip install git+https://github.com/RhetTbull/dogsheep-photos.git@update_for_bigsur

Codecov Report

Patch and project coverage have no change.

Comparison is base (adf54f5) 92.38% compared to head (935d336) 92.38%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

And in similar fashion, how can I assign the edit-tiddlywiki permission to my user myuser in metadata.yml / metadata.json?

Codecov Report

Patch and project coverage have no change.

Comparison is base (0818182) 92.38% compared to head (730f554) 92.38%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

+1 to getting this merged down.

For future googlers, I installed by... git clone git@github.com:RhetTbull/dogsheep-photos.git cd dogsheep-photos git checkout update_for_bigsur python setup.py install

Duplicate of https://github.com/dogsheep/dogsheep-photos/pull/36 - closing.

Ran across https://github.com/dogsheep/dogsheep-photos/issues/33 which is the same subject. My PR just fixes docs

Codecov Report

Patch and project coverage have no change.

Comparison is base (0818182) 92.38% compared to head (9388102) 92.38%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (0818182) 92.38% compared to head (3aca01f) 92.38%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (278ac91) 92.38% compared to head (7fc79ba) 92.38%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

My bad, although I could not find how to use urls, I was using a function call syntax for the filters instead of jinja's pipe syntax; closing this issue.

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.03 :tada:

Comparison is base (86a352f) 96.09% compared to head (faf398f) 96.12%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Patch and project coverage have no change.

Comparison is base (3a51ca9) 92.39% compared to head (3d45262) 92.39%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

I must be doing something wrong. On page https://192.168.1.3:5432/datasette-llm/logs/_llm_migrations there's the option to filter the results, there's an Apply button. It also tries to load an URL with an extra URL element in it:

https://192.168.1.3:5432/datasette-llm/datasette-llm/logs/_llm_migrations?_sort=name&name__contains=initial

I forgot to add that I followed these instructions to set up the python llm:

https://simonwillison.net/2023/Jul/18/accessing-llama-2/

The same problem can be seen in the links:

Advanced export

JSON shape: default, array, newline-delimited, object

Came here to report this, but instead I'll confirm the issue across two terminal emulators (Gnome Terminal and Alacritty) on Pop_OS! 22.04 (currently based on Ubuntu/Gnome). Also messes up the formatting of the terminal. Can also confirm that reset fixes it until the next sqlite-utils command.

Is there any working version of datasette/postgresql ?

@asg017 - the docs say that the autodetection only occurs in configuration directory mode. I for one would also be interested in the --settings settings.json feature.

For context, I am developing a large database for use with Datasette, but the database lives in a different network volume than my source code, since the volume in which my source code lives is aggressively backed up, while the location where the database lives is meant for temporary files and is not as aggressively backed up (since the backups would get unreasonably large).

@simonw any chance to get this fixed soon?

I'm not keen on requiring metadata to be within the database. I commonly have multiple DBs, from various sources, and having one config file to provide the metadata works out very well. I use Datasette with databases where I'm not the original source, needing to mutate them to add a metadata table or sqlite-docs makes me uncomfortable.

Codecov Report

Patch and project coverage have no change.

Comparison is base (ede6203) 92.40% compared to head (9785c4f) 92.40%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Patch coverage: 80.55% and project coverage change: -0.15 :warning:

Comparison is base (2747257) 96.36% compared to head (5e7d27e) 96.22%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

piping to jq is good enough usually

... or python -m json.tool[^1], if you don't have jq installed.

[^1]: no fancy colouring, like jq; only pretty-printing

I also ran into this recently. See below for a patch for one possible solution (tested via "it works on my machine", but I don't expect that this behavior would vary a whole lot across terminal emulators and shells). Another possible solution might be to subclass click's ProgressBar to keep the logic within the original context manager. Happy to send a PR or for this patch to serve as the basis for a fix that someone else authors.

```patch diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py index 06c1a4c..530a3a3 100644 --- a/sqlite_utils/utils.py +++ b/sqlite_utils/utils.py @@ -147,14 +147,23 @@ def decode_base64_values(doc):

class UpdateWrapper: - def init(self, wrapped, update): + def init(self, wrapped, update, render_finish): self._wrapped = wrapped self._update = update + self._render_finish = render_finish

 def __iter__(self):

• for line in self._wrapped:
• self._update(len(line))
• yield line
• return self +
• def next(self):
• try:
• line = next(self._wrapped)
• except StopIteration as e:
• self._render_finish()
• raise +
• self._update(len(line))

• return line

  def read(self, size=-1): data = self._wrapped.read(size) @@ -178,7 +187,7 @@ def file_progress(file, silent=False, kwargs): else: file_length = os.path.getsize(file.name) with click.progressbar(length=file_length, kwargs) as bar: - yield UpdateWrapper(file, bar.update) + yield UpdateWrapper(file, bar.update, bar.render_finish)

class Format(enum.Enum):

```

Codecov Report

Patch coverage: 95.45% and project coverage change: +0.01 :tada:

Comparison is base (b49fa44) 92.40% compared to head (df5fd73) 92.41%.

:exclamation: Current head df5fd73 differs from pull request most recent head e990fbc. Consider uploading reports for the commit e990fbc to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Oh and for context - this goes away if I use .upsert instead of insert(..., ignore=True), but I don't want to update the value if it's written, just do an insert if it's new. The code is basically:

```py def save_items(table, items): db["users"].insert(build_user(items[0]), pk="id",ignore=True) db[table].insert_all(items)

if comments := fetch_comments(): save_items('comments', comments)

if posts := fetch_posts(): save_items('posts', posts) ```

So either comments or post could create the relevant user if those items exist. In cases where they both exist, I get this error. I need the pk because either call could create the table.

Codecov Report

Patch and project coverage have no change.

Comparison is base (e240133) 96.36% compared to head (0b81794) 96.36%.

:exclamation: Current head 0b81794 differs from pull request most recent head 21036a5. Consider uploading reports for the commit 21036a5 to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Patch coverage: 83.33% and project coverage change: +0.06 :tada:

Comparison is base (b3b100d) 96.30% compared to head (948692a) 96.36%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Patch coverage: 93.75% and no project coverage change.

Comparison is base (b3b100d) 96.30% compared to head (9f23e68) 96.31%.

:exclamation: Current head 9f23e68 differs from pull request most recent head 2eca17d. Consider uploading reports for the commit 2eca17d to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Using this thread and some other resources I managed to cobble together a couple of sqlite-utils lines to add a geometry column for a table that already has a lat/lng column.

```

add a geometry column

sqlite-utils add-geometry-column [db name] [table name] geometry --type POINT --srid 4326

add a point for each row to geometry column

sqlite-utils --load-extension=spatialite [db name] 'update [table name] SET Geometry=MakePoint(longitude, latitude, 4326);' ```

Codecov Report

Patch and project coverage have no change.

Comparison is base (49184c5) 92.40% compared to head (b99e1d3) 92.40%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

perfect, thank you!

Hey @simonw, urllib3 maintainer here :wave:

Sorry for breaking your CI. I understand you may prefer to pin the Python version, but note that specifying just python: "3" will get you the latest. We use that in urllib3: https://github.com/urllib3/urllib3/blob/main/.readthedocs.yml

I can open PRs to sqlite-utils / datasette if you're interested

Some updates:

I notice that there is an option in the CLI where we can explicitly set immutable mode when spinning up the server

console Options: -i, --immutable PATH Database files to open in immutable mode

Then, the question is - how can I disable immutable mode in the deployed instance on Vercel?