See discussion in https://github.com/simonw/datasette/issues/2208

:books: Documentation preview :books:: https://datasette--2209.org.readthedocs.build/en/2209/

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/utils/asgi.py#L137-L139

In GET requests you can do ?foo=1&foo=2 - you can do the same in POST requests, but the dict() call here eliminates those duplicates.

You can't even try calling post_body() and implement your own custom parsing because of: - #2204

https://dev.to/hugovk/help-test-python-312-beta-1508/

:books: Documentation preview :books:: https://sqlite-utils--591.org.readthedocs.build/en/591/

Hi, datasette version: "0.26.2" extensions: spatialite: "4.4.0-RC0" sqlite version: "3.22.0"

I have a table of airports with latitude and longitude columns. I've added spatialite (with KNN support). After creating the db using csvs-to-sqlit, I run these commands to setup the spatialite tables:

``` conn.execute('SELECT InitSpatialMetadata(1)')

conn.execute("SELECT AddGeometryColumn('airports', 'point_geom', 4326, 'POINT', 2);")

conn.execute('''UPDATE airports SET point_geom = GeomFromText('POINT('||"longitude"||' '||"latitude"||')',4326);''')

conn.execute("SELECT CreateSpatialIndex('airports', 'point_geom');") ```

I'm attempting to create a canned query and have this in my metadata.json file: "find_airports_nearest_to_point":{ "sql":"SELECT a.pos AS rank, b.id, b.name, b.country, b.latitude AS latitude, b.longitude AS longitude, a.distance / 1000.0 AS dist_km FROM KNN AS a JOIN airports AS b ON (b.rowid = a.fid) WHERE f_table_name = \"airports\" AND ref_geometry = MakePoint( :Long , :Lat ) AND max_items = 10;"} which doesn't seem to perform the templating of the name parameters correctly and I get no results.

Have also tired: MakePoint( || :Long || , || :Lat || ) which returns this error: near "||": syntax error

However I cannot seem to find the correct combination of named parameter syntax (:Lat) or sqlite concatenation operator to make it work. Any ideas if using named parameters inside functions is supported?

Thanks Darren

I upgraded my dependencies, then ran into this problem running datasette inspect:

env/lib/python3.9/site-packages/datasette/cli.py", line 6, in <module> from click_default_group import DefaultGroup ModuleNotFoundError: No module named 'click_default_group'

Turns out the released version of datasette still depends on click-default-group-wheel, so click-default-group doesn't get installed/recognized:

$ virtualenv venv $ source venv/bin/activate $ pip install datasette $ pip list | grep click-default-group click-default-group 1.2.4 click-default-group-wheel 1.2.3 $ python -c "from click_default_group import DefaultGroup" Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'click_default_group' $ pip install --force-reinstall click-default-group ... ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. datasette 0.64.4 requires click-default-group-wheel>=1.2.2, which is not installed. Successfully installed click-8.1.7 click-default-group-1.2.4

Refs: - #2180

This plugin hook is feature complete - including documentation and tests.

I'm not going to land it in Datasette main until we've used it at least once though, which should happen promptly in development for Datasette Cloud.

:books: Documentation preview :books:: https://datasette--2181.org.readthedocs.build/en/2181/

refs 2157, specifically this comment

Passing in --internal my_internal.db over and over again can get repetitive.

This PR adds a new configurable env variable DATASETTE_INTERNAL_DB_PATH. If it's defined, then it takes place as the path to the internal database. Users can still overwrite this behavior by passing in their own --internal internal.db flag.

In draft mode for now, needs tests and documentation.

Side note: Maybe we can have a sections in the docs that lists all the "configuration environment variables" that Datasette respects? I did a quick grep and found:

• DATASETTE_LOAD_PLUGINS
• DATASETTE_SECRETS

:books: Documentation preview :books:: https://datasette--2174.org.readthedocs.build/en/2174/

Now available as a wheel: - https://github.com/click-contrib/click-default-group/issues/21

:books: Documentation preview :books:: https://datasette--2173.org.readthedocs.build/en/2173/

I used a custom SQL function in a migration script and then realized that it should be de-registered before the end of the script to avoid leaking into the calling code.

The current _internal database is used by Datasette core to cache info about databases/tables/columns/foreign keys of databases in a Datasette instance. It's a temporary database created at startup, that can only be seen by the root user. See an example _internal DB here, after logging in as root.

The current _internal database has a few rough edges:

• It's part of datasette.databases, so many plugins have to specifically exclude _internal from their queries examples here
• It's only used by Datasette core and can't be used by plugins or 3rd parties
• It's created from scratch at startup and stored in memory. Why is fine, the performance is great, but persistent storage would be nice.

Additionally, it would be really nice if plugins could use this _internal database to store their own configuration, secrets, and settings. For example:

• datasette-auth-tokens creates a _datasette_auth_tokens table to store auth token metadata. This could be moved into the _internal database to avoid writing to the gues database
• datasette-socrata creates a socrata_imports table, which also can be in _internal
• datasette-upload-csvs creates a _csv_progress_ table, which can be in _internal
• datasette-write-ui wants to have the ability for users to toggle whether a table appears editable, which can be either in datasette.yaml or on-the-fly by storing config in _internal

In general, these are specific features that Datasette plugins would have access to if there was a central internal database they could read/write to:

• Dynamic configuration. Changing the datasette.yaml file works, but can be tedious to restart the server every time. Plugins can define their own configuration table in _internal, and could read/write to it to store configuration based on user actions (cell menu click, API access, etc.)
• Caching. If a plugin or Datasette Core needs to cache some expensive computation, they can store it inside _internal (possibly as a temporary table) instead of managing their own caching solution.
• Audit logs. If a plugin performs some sensitive operations, they can log usage info to _internal for others to audit later.
• Long running process status. Many plugins (datasette-upload-csvs, datasette-litestream, datasette-socrata) perform tasks that run for a really long time, and want to give continue status updates to the user. They can store this info inside_internal
• Safer authentication. Passwords and authentication plugins usually store credentials/hashed secrets in configuration files or environment variables, which can be difficult to handle. Now, they can store them in _internal

Proposal

• We remove _internal from datasette.databases property.
• We add new datasette.get_internal_db() method that returns the _internal database, for plugins to use
• We add a new --internal internal.db flag. If provided, then the _internal DB will be sourced from that file, and further updates will be persisted to that file (instead of an in-memory database)
• When creating internal.db, create a new _datasette_internal table to mark it a an "datasette internal database"
• In datasette serve, we check for the existence of the _datasette_internal table. If it exists, we assume the user provided that file in error and raise an error. This is to limit the chance that someone accidentally publishes their internal database to the internet. We could optionally add a --unsafe-allow-internal flag (or database plugin) that allows someone to do this if they really want to.

New features unlocked with this

These features don't really need a standardized _internal table per-say (plugins could currently configure their own long-time storage features if they really wanted to), but it would make it much simpler to create these kinds of features with a persistent application database.

• datasette-comments : A plugin for commenting on rows or specific values in a database. Comment contents + threads + email notification info can be stored in _internal
• Bookmarks: "Bookmarking" an SQL query could be stored in _internal, or a URL link shortener
• Webhooks: If a plugin wants to either consume a webhook or create a new one, they can store hashed credentials/API endpoints in _internal

If I have a self-hosted instance of Datasette up and running, I'd like to be able to the use the CLI to publish databases to that instance, not only Google or Heroku. Ideally there'd be a url parameter or something similar to which one could point the publish command to their instance.

Bumps the python-packages group with 3 updates: sphinx, furo and blacken-docs.

Updates sphinx from 7.1.2 to 7.2.3

Updates furo from 2023.7.26 to 2023.8.19

Updates blacken-docs from 1.15.0 to 1.16.0

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

:books: Documentation preview :books:: https://datasette--2152.org.readthedocs.build/en/2152/

It's EOL now, as of 2023-06-27 (two days ago): https://devguide.python.org/versions/

Split off from #274

Metadata should an optional "config" section at root, table or database level.

The TableView and RowView and DatabaseView and BaseView classes could all have a .config("key") method which knows how to resolve the hierarchy of configs.

This will allow individual tables (or databases) to set their own config settings for things like sql_time_limit_ms

Extremely detailed story of how I got to this point:

• https://github.com/simonw/llm/issues/162

Steps to reproduce (only if that pragma is on though): bash python -c ' import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute(""" CREATE TABLE "logs" ( [id] INTEGER PRIMARY KEY, [model] TEXT, [prompt] TEXT, [system] TEXT, [prompt_json] TEXT, [options_json] TEXT, [response] TEXT, [response_json] TEXT, [reply_to_id] INTEGER, [chat_id] INTEGER REFERENCES [log]([id]), [duration_ms] INTEGER, [datetime_utc] TEXT ); """) db["logs"].add_foreign_key("reply_to_id", "logs", "id") ' This succeeds in some environments, fails in others.

I got this error trying to drop a column from a table that was part of a SQL view:

error in view plugins: no such table: main.pypi_releases

Upon further investigation I found that this pattern seemed to fix it: python def transform_the_table(conn): # Run this in a transaction: with conn: # We have to read all the views first, because we need to drop and recreate them db = sqlite_utils.Database(conn) views = {v.name: v.schema for v in db.views if table.lower() in v.schema.lower()} for view in views.keys(): db[view].drop() db[table].transform( types=types, rename=rename, drop=drop, column_order=[p[0] for p in order_pairs], ) # Now recreate the views for name, schema in views.items(): db.create_view(name, schema) So grab a copy of any view that might reference this table, start a transaction, drop those views, run the transform, recreate the views again.

I wonder if this should become an option in sqlite-utils? Maybe a recreate_views=True argument for table.tranform(...)? Should it be opt-in or opt-out?

Originally posted by @simonw in https://github.com/simonw/datasette-edit-schema/issues/35#issuecomment-1683370548

https://docs.datasette.io/en/latest/authentication.html#other-permissions-in-metadata says the following:

For all other permissions, you can use one or more "permissions" blocks in your metadata.

To grant access to the permissions debug tool to all signed in users you can grant permissions-debug to any actor with an id matching the wildcard * by adding this a the root of your metadata:

yaml permissions: debug-menu: id: '*'

I tried this.

My metadata.yml file looks like:

yaml permissions: debug-menu: id: '*' permissions-debug: id: '*' plugins: datasette-auth-passwords: myuser_password_hash: $env: "PASSWORD_HASH_MYUSER"

And then I run

zsh datasette -m metadata.yml tiddlywiki.db --root

And I open a session for the "root" user of datasette with the link given.

I open a private browser session and log in as "myuser" from http://127.0.0.1:8001/-/login

Then I check http://127.0.0.1:8001/-/actor which confirms that I am logged in as the "myuser" actor

json { "actor": { "id": "myuser" } }

In the session where I am logged in as "myuser" I then try to go to http://127.0.0.1:8001/-/permissions

But all I get there as the logged in user "myuser" is

Forbidden

Permission denied

And then if I check the http://127.0.0.1:8001/-/permissions as the datasette "root" user from another browser session, I see:

permissions-debug checked at 2023-08-06T16:22:58.997841 ✗ (used default)

Actor: {"id": "myuser"}

It seems that in spite of having tried to give the permissions-debug permission to the "myuser" user in my metadata.yml file, datasette does not agree that "myuser" has permission permissions-debug..

What do I need to do differently so that my "myuser" user is able to access http://127.0.0.1:8001/-/permissions ?

I think this has been a bug for a long time.

https://latest.datasette.io/ currently shows:

Those colors are based on a hash of the database name. But when you click through to https://latest.datasette.io/fixtures

It's red on all sub-pages too.

This idea first came up here: - https://github.com/simonw/datasette/issues/2112#issuecomment-1652751140

If datasette.render_template(...) takes an optional Context subclass as an alternative to a context dictionary, I could then use dataclasses to define the context made available to specific templates - which then gives me something I can use to help document what they are.

Also refs: - https://github.com/simonw/datasette/issues/1510

118 commits! https://github.com/simonw/datasette/compare/1.0a2...26be9f0445b753fb84c802c356b0791a72269f25

• 2109

• 2109

Bumps furo from 2023.3.27 to 2023.5.20.

You can trigger a rebase of this PR by commenting @dependabot rebase.

:books: Documentation preview :books:: https://datasette--2077.org.readthedocs.build/en/2077/

Note Automatic rebases have been disabled on this pull request as it has been open for over 30 days.

A minor refactor, but it is a good starting point for this new branch. Refs: - #2109

As seen in LLM and now in sqlite-utils too: - https://github.com/simonw/sqlite-utils/issues/570

Useful for developing plugins, see tutorial at https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html

Splitting off an issue for prepare_connection() since Alex got the PR in seconds before I shipped 3.34!

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646686424

I'm going to start by adding the register_command hook using the exact same pattern as Datasette and LLM.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643450

I use this nginx config: ``` location /datasette-llm { return 302 /datasette-llm/; }

location /datasette-llm/ {
  proxy_set_header Upgrade           $http_upgrade;
  proxy_set_header Connection        "Upgrade";
  proxy_http_version 1.1;
  proxy_set_header X-Real-IP         $remote_addr;
  proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto https;
  proxy_set_header X-Forwarded-Host  $http_host;
  proxy_set_header Host              $host;
  proxy_max_temp_file_size           0;
  proxy_pass                         http://127.0.0.1:8001/datasette-llm/;
  proxy_redirect                     http:// https://;
  proxy_buffering off;
  proxy_request_buffering off;
  proxy_set_header Origin            '';
  client_max_body_size 0;
  auth_basic                         "datasette-llm";
  auth_basic_user_file               /etc/nginx/custom-userdb;
}

Then I start datasette with this command: datasette serve --setting base_url /datasette-llm/ $(llm logs path) ``` Everything else works right, except the links in "This data as json, CSV". They get an extra URL element "datasette-llm" like this:

https://192.168.1.3:5432/datasette-llm/datasette-llm/logs.json?sql=select+*+from+_llm_migrations

https://192.168.1.3:5432/datasette-llm/datasette-llm/logs.csv?sql=select+*+from+_llm_migrations&_size=max

When I remove that extra "datasette-llm" from the URL, those links work too.

Documented context plus protective unit tests. Goal is that custom templates built for 1.x will not break without a 2.x release.

Refs:

• 2078

TODO:

• [x] Teach router layer how to handle this
• [x] Use it for something

:books: Documentation preview :books:: https://datasette--2080.org.readthedocs.build/en/2080/

https://github.com/simonw/sqlite-utils/actions/runs/5039064797/jobs/9036965488

Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.7.16/x64/bin/flake8", line 8, in <module> sys.exit(main()) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/cli.py", line 22, in main app.run(argv) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py", line 363, in run self._run(argv) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py", line 350, in _run self.initialize(argv) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py", line 330, in initialize self.find_plugins(config_finder) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py", line 153, in find_plugins self.check_plugins = plugin_manager.Checkers(local_plugins.extension) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 357, in __init__ self.namespace, local_plugins=local_plugins File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 238, in __init__ self._load_entrypoint_plugins() File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 254, in _load_entrypoint_plugins eps = importlib_metadata.entry_points().get(self.namespace, ()) AttributeError: 'EntryPoints' object has no attribute 'get' Error: Process completed with exit code 1.

Refs: - #545

:books: Documentation preview :books:: https://sqlite-utils--549.org.readthedocs.build/en/549/

:books: Documentation preview :books:: https://sqlite-utils--537.org.readthedocs.build/en/537/

I got this error:

File "/Users/simon/Dropbox/Development/openai-to-sqlite/openai_to_sqlite/cli.py", line 27, in embeddings rows, _ = rows_from_file(input) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/simon/.local/share/virtualenvs/openai-to-sqlite-jt4obeb2/lib/python3.11/site-packages/sqlite_utils/utils.py", line 305, in rows_from_file first_bytes = buffered.peek(2048).strip() ^^^^^^^^^^^^^^^^^^^ From this code: ```python

@cli.command() @click.argument( "db_path", type=click.Path(file_okay=True, dir_okay=False, allow_dash=False), ) @click.option( "-i", "--input", type=click.File("r"), default="-", ) def embeddings(db_path, input): "Store embeddings for one or more text documents" click.echo("Here is some output") db = sqlite_utils.Database(db_path) rows, _ = rows_from_file(input) print(list(rows)) `` The error went away when I changed it totype=click.File("rb")`.

This should either be called out in the documentation or rows_from_file() should be fixed to handle text-mode files in addition to binary files.

As there's the possibility of hiding tables - I've run into the need of hiding specific columns - data that's either not relevant for public or can't be shown due to privacy reasons.

Working with nested data in SQL is tricky, can I make it easier with a view or canned query?

FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError: [Errno 2] No such file or directory

From https://github.com/simonw/datasette/actions/runs/4348548218/jobs/7597208191

``` =================================== FAILURES =================================== ____ test_install_requirements _______

run_module = <MagicMock name='run_module' id='139768358191936'>

@mock.patch("datasette.cli.run_module")
def test_install_requirements(run_module):
    runner = CliRunner()

  with runner.isolated_filesystem():

/home/runner/work/datasette/datasette/tests/test_cli.py:184:

/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/contextlib.py:119: in enter return next(self.gen)

self = <click.testing.CliRunner object at 0x7f1e5bfb9490>, temp_dir = None

@contextlib.contextmanager
def isolated_filesystem(
    self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None
) -> t.Iterator[str]:
    """A context manager that creates a temporary directory and
    changes the current working directory to it. This isolates tests
    that affect the contents of the CWD to prevent them from
    interfering with each other.

    :param temp_dir: Create the temporary directory under this
        directory. If given, the created directory is not removed
        when exiting.

    .. versionchanged:: 8.0
        Added the ``temp_dir`` parameter.
    """

  cwd = os.getcwd()

E FileNotFoundError: [Errno 2] No such file or directory

/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/click/testing.py:466: FileNotFoundError ``` Not sure why it only affected the "Calculate test coverage" one.

I'm using ChatGPT to figure out enough AppleScript to get at my notes data.

sqlite foreign keys can be compound, but that is not as well supported by datasette as single column foreign keys.

in particular,

1. in a table view, there is not a link from the row to the referenced row if the foreign key is compound
2. in a row view, there is no listing of tables and rows that refer to the focal row if those referencing foreign keys are compound.

Both of these issues are discussed in #1099.

This PR only fixes the second one, because it's not clear what the right UX is for the first issue.

Some things that might not be desirable about this approach.

1. it changes the external API, by changing column => columns and other_column => other_columns (see inline comment for more discussion.
2. There are various places where the plural foreign keys have to be checked for length and discarded or transformed to singular.

A deployment of #1967 to ReadTheDocs just failed like this: https://readthedocs.org/projects/datasette/builds/19045460/

``` Running Sphinx v6.0.0 making output directory... done building [mo]: targets for 0 po files that are out of date building [html]: targets for 28 source files that are out of date updating environment: [new config] 28 added, 0 changed, 0 removed reading sources... [ 3%] authentication reading sources... [ 7%] binary_data reading sources... [ 10%] changelog

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 299, in next_line self.line = self.input_lines[self.line_offset] File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 1136, in getitem return self.data[i] IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 226, in run self.next_line() File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 302, in next_line raise EOFError EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/cmd/build.py", line 281, in build_main app.build(args.force_all, args.filenames) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/application.py", line 344, in build self.builder.build_update() File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/builders/init.py", line 310, in build_update self.build(to_build, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/builders/init.py", line 326, in build updated_docnames = set(self.read()) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/builders/init.py", line 433, in read self._read_serial(docnames) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/builders/init.py", line 454, in _read_serial self.read_doc(docname) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/builders/init.py", line 510, in read_doc publisher.publish() File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/core.py", line 224, in publish self.document = self.reader.read(self.source, self.parser, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/io.py", line 103, in read self.parse() File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/readers/init.py", line 76, in parse self.parser.parse(self.input, document) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/parsers.py", line 78, in parse self.statemachine.run(inputlines, document, inliner=self.inliner) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 169, in run results = StateMachineWS.run(self, input_lines, input_offset, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 233, in run context, next_state, result = self.check_line( File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 445, in check_line return method(match, context, next_state) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 3024, in text self.section(title.lstrip(), source, style, lineno + 1, messages) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 325, in section self.new_subsection(title, lineno, messages) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 391, in new_subsection newabsoffset = self.nested_parse( File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 279, in nested_parse state_machine.run(block, input_offset, memo=self.memo, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 195, in run results = StateMachineWS.run(self, input_lines, input_offset) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 233, in run context, next_state, result = self.check_line( File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 445, in check_line return method(match, context, next_state) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2785, in underline self.section(title, source, style, lineno - 1, messages) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 325, in section self.new_subsection(title, lineno, messages) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 391, in new_subsection newabsoffset = self.nested_parse( File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 279, in nested_parse state_machine.run(block, input_offset, memo=self.memo, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 195, in run results = StateMachineWS.run(self, input_lines, input_offset) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 233, in run context, next_state, result = self.check_line( File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 445, in check_line return method(match, context, next_state) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 1273, in bullet i, blank_finish = self.list_item(match.end()) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 1295, in list_item self.nested_parse(indented, input_offset=line_offset, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 279, in nested_parse state_machine.run(block, input_offset, memo=self.memo, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 195, in run results = StateMachineWS.run(self, input_lines, input_offset) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 239, in run result = state.eof(context) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2725, in eof self.blank(None, context, None) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2716, in blank paragraph, literalnext = self.paragraph( File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 416, in paragraph textnodes, messages = self.inline_text(text, lineno) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 425, in inline_text nodes, messages = self.inliner.parse(text, lineno, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 649, in parse before, inlines, remaining, sysmessages = method(self, match, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 792, in interpreted_or_phrase_ref nodelist, messages = self.interpreted(rawsource, escaped, role, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 889, in interpreted nodes, messages2 = role_fn(role, rawsource, text, lineno, self) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/ext/extlinks.py", line 101, in role title = caption % part TypeError: not all arguments converted during string formatting

Exception occurred: File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/ext/extlinks.py", line 101, in role title = caption % part TypeError: not all arguments converted during string formatting The full traceback has been saved in /tmp/sphinx-err-kq7ylgqo.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks! ```

It's used by datasette-geojson here: https://github.com/eyeseast/datasette-geojson/commit/902bf135a5a33a0dc8264673d00a59a67cb05152

https://latest.datasette.io/fixtures/facetable.json doesn't currently include the CREATE TABLE statement for the page, even though it's available on the HTML version at https://latest.datasette.io/fixtures/facetable

Decision: as an initial fix I'm going to de-duplicate those keys by using tags__array etc - with a _2 on the end if that key is already used.

I'll open a separate issue to redesign this better for Datasette 1.0.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/625#issuecomment-996130862

Refs #1636

• [x] Documentation
• [ ] Implementation
• [ ] Validate metadata to check there are no nonsensical permissions (like debug-menu set at the table level)
• [ ] Tests

:books: Documentation preview :books:: https://datasette--1938.org.readthedocs.build/en/1938/

Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions

This method allows multiple permissions to be checked at onced. It raises a datasette.Forbidden exception if any of the checks are denied before one of them is explicitly granted.

This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns True or not a single one of them returns False:

That's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1675#issuecomment-1074177827

The RowTableShared class is making this a whole lot more complicated.

I'm going to split the RowView view out into an entirely separate views/row.py module.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1715#issuecomment-1108875068

https://github.com/simonw/datasette/actions/runs/3587402085/jobs/6038106719v

Warning: "service_account_key" has been deprecated. Please switch to using google-github-actions/auth which supports both Workload Identity Federation and Service Account Key JSON authentication. For more details, see https://github.com/google-github-actions/setup-gcloud#authorization Error: google-github-actions/setup-gcloud failed with: failed to execute command `gcloud --quiet auth activate-service-account *** --key-file -`: /opt/hostedtoolcache/gcloud/275.0.0/x64/lib/googlecloudsdk/core/console/console_io.py:544: SyntaxWarning: "is" with a literal. Did you mean "=="? if answer is None or (answer is '' and default is not None): ERROR: gcloud failed to load: module 'collections' has no attribute 'MutableMapping'

I'm committed enough to the 1.0 work now that I'm ready for the main branch to reflect that instead.

If I need to make any dot-releases against 0.63 I can do those from a branch.

When I do sqlite-utils enable-fts my.db table_name column_name (with or without --fts5), I get an FTS4 virtual table instead of the expected FTS5.

FTS5 is however available and Python/SQLite versions do not seem to be the issue. I can manually create the FTS5 virtual table, and then Datasette also works with it from this same Python environment.

>>> sqlite3.version 2.6.0 >>> sqlite3.sqlite_version 3.39.4

PRAGMA compile_options; includes ENABLE_FTS5.

sqlite-utils, version 3.30.

Any ideas what's happening and how to fix?

https://github.com/simonw/sqlite-utils/actions/runs/3472012235 failed on Python 3.11:

Truncated output: sqlite_utils/db.py:2467: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True sqlite_utils/db.py:2467: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase sqlite_utils/db.py:2530: error: Incompatible default for argument "where" (default has type "None", argument has type "str") [assignment] sqlite_utils/db.py:2530: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True sqlite_utils/db.py:2530: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase sqlite_utils/db.py:2658: error: Argument 1 to "count_where" of "Queryable" has incompatible type "Optional[str]"; expected "str" [arg-type] Found 23 errors in 1 file (checked 51 source files) Best look at https://github.com/hauntsaninja/no_implicit_optional

The CLI equivalent of the /-/create-token page.

Had some design thoughts here: https://github.com/simonw/datasette/issues/1852#issuecomment-1291272280

I liked this option the most:

--setting allow_create_tokens off

In building this Datasette feature on top of sqlite-utils I thought it might be useful to expose the number of rows that had been affected by a bulk insert or update - the cursor.rowcount:

• https://github.com/simonw/datasette/issues/1866

This isn't currently exposed by sqlite-utils.

1. My https://list.saferdisinfectants.org/disinfectants/listN page (linked from https://SaferDisinfectants.org ) has been running beautifully for a year and a half, including a GitHub Actions workflow that's been routinely updating the database.

2. I received a recent report that the list page is down. I don't know when it went down, but the content is replaced with: "render_template() called before await ds.invoke_startup()"

3. The local datasette repo runs without incident.

4. The site is hosted on vercel, linked to my github repo. Perhaps some vercel changes were made, but not by anyone on our side. Here is a screenshot of the current project settings:

Here a screenshot of the latest deployment status:

This is my repository: https://github.com/mroswell/list-N (I notice: datasette==0.59 in my requirements.txt file)

Because it's been long while since I actively worked on this or any other datasette project, I forget a lot of what I knew at one point. Perhaps some configuration file could be missing? Or perhaps I just need to know the right incantation to add to that vercel settings page.

Help is welcome as the nonprofit org is soon hosting its annual conference, and we'd love to have the page working again.

Most of the release notes are already written: - https://github.com/simonw/datasette/releases/tag/0.63a0 - https://github.com/simonw/datasette/releases/tag/0.63a1

inspect_data should already include the hash and the db file size, so this PR takes advantage of using those instead of always recalculating. should help a lot on startup with large DBs.

closes #1834

Added in: - #483

I don't know how to fix this in Sphinx: I'm getting this: https://sqlite-utils.datasette.io/en/latest/cli.html#cli-install

The insert –convert and query –functions options

But I want it to display insert --convert and not insert –convert there.

Here's the code: https://github.com/simonw/sqlite-utils/blob/85247038f70d7eb2f3e272cfeaa4c44459cafba8/docs/cli.rst#L2125

When using the convert function on a JSON column, the result of the conversion function must be a string. If the return value is either a dict (object) or a list (array), the convert call will error out with an unhelpful user defined function exception.

It makes sense that since the original column value was a string and required conversion to data structures, the result should be converted back into a JSON string as well. However, other functions auto-convert to JSON string representation, so the fact that convert doesn't could be surprising.

At least the documentation should note this requirement, because the sqlite error messages won't readily reveal the issue.

Jf only sqlite's JSON column type meant something :)

Failing on Python 3.10 at the moment: https://github.com/simonw/datasette/actions/runs/3323629947/jobs/5494340302

Following: - #1829

https://latest.datasette.io/_internal looks like this:

But those queries and tables are private too, and should also show the padlock icon.

Refs #1829

• [x] Fix table page
• [x] Fix database page
• [x] Fix query page
• [x] Fix row page
• [x] Tests
• [x] Updated documentation for check_visibility method, to cover the new permissions= keyword argument

Also this fix is currently only applied on the table page - needs to be applied on database, row and query pages too.

:books: Documentation preview :books:: https://datasette--1842.org.readthedocs.build/en/1842/

Had a really interesting conversation today about how hard it is to get from "I installed Datasette" to "I've done something useful with it": https://news.ycombinator.com/item?id=33216789#33218590

Spending some time focusing on that first-run experience feels very worthwhile.

Hello, I've been trying to run Datasette with the configuration directory mode with a structure such as this one:

plain some-directory/ example.sqlite3 another-example.db one-more.custom [...]

(In my scenario I can't just change the filename extension without other problems arising)

Now databases with the .sqlite3 or the custom filename extension are ignored by Datasette in this case. I'm aware that the docs state that a .db extension is required, but I was wondering if there is a reason for restricting this or any workaround available? When I run datasette example.sqlite3 or datasette one-more.custom the databases are served by Datasette without a problem.

This is technically a Sandstorm-specific fix (as external links do not work inside the grain frame), however, I think it is an improvement to the upstream project, so I wanted to propose it here rather than patching it in our package.

There's much opinions on the Internet about whether external links should open in a new tab by default or not, but I'd argue very few people who might click a "powered by" link intend to complete their interaction with the source page (a Datasette). And furthermore, users may be working within various queries or loading visualizations (navigating away when trying to plot a million GPS coordinates pretty much just resets your progress!), so linking away within the tab might be a frustrating or destructive act to one's work, even inadvertently.

original report: https://github.com/ocdtrekkie/datasette-sandstorm/issues/1

:books: Documentation preview :books:: https://datasette--1838.org.readthedocs.build/en/1838/

Refs #1822

:books: Documentation preview :books:: https://datasette--1823.org.readthedocs.build/en/1823/

This can be a documented wrapper around with db.conn:.

Open Datasette stable doc at https://docs.datasette.io/en/stable/plugin_hooks.html?highlight=render_cell#render-cell-row-value-column-table-database-datasette

render_cell plugin hook method signature is render_cell(row, value, column, table, database, datasette), the example shown inline uses render_cell(value).

This is a good idea, and one that needs to happen before Datasette 1.0:

While you are adding features, would you be future-proofing your APIs if you switched over some arguments over to keyword-only arguments or would that be too disruptive?

Thinking out loud:

async def render_template( self, templates, *, context=None, plugin_context=None, request=None, view_name=None ): Originally posted by @jefftriplett in https://github.com/simonw/datasette/issues/1817#issuecomment-1256781274

If a query hits the timeout it shows a message like:

SQL query took too long. The time limit is controlled by the sql_time_limit_ms configuration option.

But the query is lost. Hitting the browser back button shows the query before the one that errored.

It would be nice if the query that errored was preserved for more tweaking. This would make it similar to how "invalid syntax" works since #1346 / #619.

Bumps furo from 2022.6.21 to 2022.9.15.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

:books: Documentation preview :books:: https://datasette--1812.org.readthedocs.build/en/1812/

Ran the auth command in the same directory I'd previously set up an auth.json file for twitter-to-sqlite and it was completely overwritten. Not the biggest issue, but still unexpected. Ideally, for me, the keys would just be added to the existing file, but getting a warning and a chance to back out would be a good solution as well.

https://iscinumpy.dev/post/bound-version-constraints/ has convinced me not to use upper bound dependencies unless I'm certain they are needed.

Relevant PR: - https://github.com/simonw/datasette/pull/1799

Also:

https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/setup.py#L45-L46

https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/setup.py#L48-L49

https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/setup.py#L51-L55

https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/setup.py#L57-L59

https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/setup.py#L75-L78

https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/setup.py#L81-L82

Updates the requirements on jinja2 to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

https://sqlite-utils.datasette.io/en/stable/cli.html#csv-files-without-a-header-row

The first row of any CSV or TSV file is expected to contain the names of the columns in that file.

If your file does not include this row, you can use the --no-headers option to specify that the tool should not use that fist row as headers.

If you do this, the table will be created with column names called untitled_1 and untitled_2 and so on. You can then rename them using the sqlite-utils transform ... --rename command.

It would be nice to be able to specify the column names when importing CSV/TSV without a header row, via an extra command line option.

(renaming a column of a large table can take a long time, which makes it an inconvenient workaround)

sqlite "IS" is equivalent to SQL "IS NOT DISTINCT FROM"

closes #423

Refs: - #1789

:books: Documentation preview :books:: https://datasette--1792.org.readthedocs.build/en/1792/

I'm not sure which option is best, instead of -a -all.

Working on the Dockerfile in #1249 made me wish for automated tests - to confirm that it boots up correctly, can run SpatiaLite and doesn't have weird bugs like the /db page hanging.

These could run in CI too, but maybe only if the Dockerfile is updated.

Bumps black from 22.1.0 to 22.3.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I am building a database, in which one of the tables has a many-to-many relationship to itself. As far as I can see, this is not (yet) possible using .m2m() in sqlite-utils. This may be a bit of a niche use case, so feel free to close this issue if you feel it would introduce too much complexity compared to the benefits.

Example: suppose I have a table of people, and I want to store the information that John and Mary have two children, Michael and Suzy. It would be neat if I could do something like this:

```python from sqlite_utils import Database

db = Database(memory=True) db["people"].insert({"name": "John"}, pk="name").m2m( "people", [{"name": "Michael"}, {"name": "Suzy"}], m2m_table="parent_child", pk="name" ) db["people"].insert({"name": "Mary"}, pk="name").m2m( "people", [{"name": "Michael"}, {"name": "Suzy"}], m2m_table="parent_child", pk="name" ) ```

But if I do that, the many-to-many table parent_child has only one column: CREATE TABLE [parent_child] ( [people_id] TEXT REFERENCES [people]([name]), PRIMARY KEY ([people_id], [people_id]) )

This could be solved by adding one or two keyword_arguments to .m2m(), e.g. .m2m(..., left_name=None, right_name=None) or .m2m(..., names=(None, None)).

https://sqlite-utils.datasette.io/en/stable/cli-reference.html#insert

It looks like Python keywords are being incorrectly highlighted here.

I've used it in a couple of external places now:

• https://github.com/simonw/datasette-socrata/blob/32fb256a461bf0e790eca10bdc7dd9d96c20f7c4/datasette_socrata/init.py#L264-L280
• https://github.com/simonw/datasette-lite/blob/caa8eade10f0321c64f9f65c4561186f02d57c5b/webworker.js#L55-L64

Refs: - https://github.com/simonw/datasette-lite/issues/32

To allow users customise the facet content as they would prefer such as sorting of facet results. ordering of suggested facets etc.

Windows 10 Python 3.9.6

When I was running a full text search through the Python library, I noticed that the query was being run on a different full text search table than the one I was trying to search.

I took a look at the following function

https://github.com/simonw/sqlite-utils/blob/841ad44bacaff05ec79ef78166d12e80c82ba6d7/sqlite_utils/db.py#L2213

and noticed:

python sql LIKE '%VIRTUAL TABLE%USING FTS%content=%{table}%'

My database contains tables with similar names and %{table}% was matching another table that ended differently in its name. I have included a sample test that shows this occurring:

I search for Marsupials in db["books"] and The Clue of the Broken Blade is returned.

This occurs since the search for Marsupials was "successfully" done against db["booksb"] and rowid 1 is returned. "The Clue of the Broken Blade" has a rowid of 1 in db["books"] and this is what is returned from the search.

```python def test_fts_search_with_similar_table_names(fresh_db): db = Database(memory=True) db["books"].insert_all( [ { "title": "The Clue of the Broken Blade", "author": "Franklin W. Dixon", }, { "title": "Habits of Australian Marsupials", "author": "Marlee Hawkins", }, ] ) db["booksb"].insert( { "title": "Habits of Australian Marsupials", "author": "Marlee Hawkins", } )

db["booksb"].enable_fts(["title", "author"])
db["books"].enable_fts(["title", "author"])


query = "Marsupials"

assert [
        {   "rowid": 1,
            "title": "Habits of Australian Marsupials",
            "author": "Marlee Hawkins",
        },
    ] == list(db["books"].search(query))

```

Currently for canned queries, you have to inline SQL in your metadata.yaml like so:

yaml databases: fixtures: queries: neighborhood_search: sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood title: Search neighborhoods

This works fine, but for a few reasons, I usually have my canned queries already written in separate .sql files. I'd like to instead re-use those instead of re-writing it.

So, I'd like to see a new "sql_file" key that works like so:

metadata.yaml:

yaml databases: fixtures: queries: neighborhood_search: sql_file: neighborhood_search.sql title: Search neighborhoods neighborhood_search.sql: sql select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood

Both of these would work in the exact same way, where Datasette would instead open + include neighborhood_search.sql on startup.

A few reasons why I'd like to keep my canned queries SQL separate from metadata.yaml:

• Keeping SQL in standalone SQL files means syntax highlighting and other text editor integrations in my code
• Multiline strings in yaml, while functional, are a tad cumbersome and are hard to edit
• Works well with other tools (can pipe .sql files into the sqlite3 CLI, or use with other SQLite clients easier)
• Typically my canned queries are quite long compared to everything else in my metadata.yaml, so I'd love to separate it where possible

Let me know if this is a feature you'd like to see, I can try to send up a PR if this sounds right!

I was going to vendor baseconv.py, but then I reconsidered - what if there are plugins out there that expect import baseconv to work because they have depended on Datasette?

I used https://cs.github.com/ and as far as I can tell there aren't any!

So I'm going to remove that dependency and work out a smarter way to do this - probably by providing a utility function within Datasette itself.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1733#issuecomment-1115258737

https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_trace=1 is not showing the trace JSON at the bottom of the page.

Confirmed that trace_debug is true on https://global-power-plants.datasettes.com/-/settings

Possibly related:

• https://github.com/simonw/datasette-total-page-time/issues/1

fixes #1224

This PR ensures all databases found in a configuration directory that match the files in inspect-data.json will be set to immutable as outlined in https://docs.datasette.io/en/latest/settings.html#configuration-directory-mode

specifically on building the datasette instance it checks: - if immutables is an empty tuple - as passed by the cli code - if immutables is the default function value None - when it's not explicitly set

And correctly builds the immutable database list from the inspect-data[file] keys.

Note for this to work the inspect-data.json file must contain file paths which are relative to the configuration directory otherwise the file paths won't match and the dbs won't be set to immutable.

I couldn't find an easy way to test this due to the way make_app_client works, happy to take directions on adding a test for this.

I've updated the relevant docs as well, i.e. use the inspect cli cmd from the config directory path to create the relevant file cd $config_dir datasette inspect *.db --inspect-file=inspect-data.json https://docs.datasette.io/en/latest/performance.html#using-datasette-inspect

Binary: 2,427,344 bytes would be nicer - even better, include a tooltip showing that size translated using this function: https://github.com/simonw/datasette/blob/138e4d9a53e3982137294ba383303c3a848cfca4/datasette/utils/init.py#L837-L846

I'm trying to make a small plugin that depends on static assets, by following the guide here. I made a plugins/ directory with datasette_nteract_data_explorer.py.

I am trying to follow the example of datasette_vega, and serving static assets. I created a statics/ directory within plugins/ to serve my JS and CSS.

https://github.com/simonw/datasette-vega/blob/00de059ab1ef77394ba9f9547abfacf966c479c4/datasette_vega/init.py#L13

Unfortunately, datasette doesn't seem to be able to find my assets.

Input:

bash datasette ~/Library/Safari/History.db --plugins-dir=plugins/

Output:

I suspect this issue might go away if I move away from "one-off" plugin mode, but it's been a while since I created a new python package so I'm not sure how much work there is to go between "one off" and "packaged for PyPI". I'd like to try to avoid needing to repackage a new tar.gz file and or reinstall my library repeatedly when developing new python code.

1. Is there a way to serve a static assets when using the plugins/ directory method instead of installing plugins as a new python package?
2. If not, is there a way I can work on developing a plugin without creating and repackaging tar.gz files after every change, or is that the recommended path?

Thanks for your help!

Any JSON that is newline-delimited and has whitespace (newlines) between the start of a JSON object and an attribute fails due to a parse error.

e.g. given the valid JSONL:

{ "attribute": "value" } { "attribute": "value2" }

I would expect that sqlite-utils insert --nl my.db mytable file.jsonl would properly import the data into mytable. However, the following error is thrown instead:

json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)

It makes sense that since the file is intended to be newline separated, the thing being parsed is "{" (which obviously fails), however the default newline-separated output of jq isn't compact. Using jq -c avoids this problem, but the fix is unintuitive and undocumented.

Proposed solutions: 1. Default to a "loose" newline-separated parse; this could be implemented internally as [the equivalent of] a jq -c filter ahead of the insert step. 2. Catch the JSONDecodeError (or pre-empt it in the case of a record === "{\n") and give the user a "it looks like your json isn't actually newline-delimited; try running it through jq -c instead" error message.

It might just have been too early in the morning when I was playing with this, but running pipes of data through sqlite-utils without the 'knack' of it led to some false starts.

... highlights how inconsistent the way the capturing works is. Especially as_format which can be None or "" or .json or json or not used at all in the case of TableView.

https://github.com/simonw/datasette/blob/764738dfcb16cd98b0987d443f59d5baa9d3c332/tests/test_routes.py#L12-L36

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1666#issuecomment-1073039670

Part of: - #1660

I ran into some bugs earlier with URL routing - having more robust testing around this (especially since they are defined using regular expressions) would be really useful.

• A utility function that resolves a path against a list of reflexes and returns the match
• Make the routes and regular expressions available from a private Datasette method
• Add tests that exercise them

Related: - #1660

If hashed_url mode is turned on and crossdb is also turned on, then queries to _memory should have a hash_id.

One way that it could work is to have the _memory hash be a hash of all the individual databases.

Otherwise, crossdb queries can get quit out of data if using aggressive caching.

Did a search on the issues here and didn't find anything related to what I want.

I want to have information that is on the database level of the JSON like title, source and source_url, and use it on the index page.

I tried some small tweaks on the python and html files, but failed to get that result.

Is there a way? Thanks!

The new JSON API in a stable, documented form.

Currently, I have a bunch of code that looks like this:

python subjects = db["subjects"] if db["subjects"].exists() else db["subjects"].create({ ... }) It would be neat if sqlite-utils could simplify that by supporting CREATE TABLE IF NOT EXISTS, so that I'd be able to write, e.g.

python subjects = db["subjects"].create({...}, if_not_exists=True)

I built this for shapefile-to-sqlite but it would be useful in sqlite-utils too:

https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L16-L19

python SPATIALITE_PATHS = ( "/usr/lib/x86_64-linux-gnu/mod_spatialite.so", "/usr/local/lib/mod_spatialite.dylib", )

https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L105-L109

python def find_spatialite(): for path in SPATIALITE_PATHS: if os.path.exists(path): return path return None

In conjunction with #135 - this would do the same thing as --load-extension=path-to-spatialite (see #134)

Hi, it would be great to add a parameter that enables the load of a sqlite extension you need.

Something like "-ext modspatialite".

In this way your great tool would be even more comfortable and powerful.

Thank you very much