Does what it says and nothing else. This is the same set of paths as Datasette uses.

:books: Documentation preview :books:: https://sqlite-utils--536.org.readthedocs.build/en/536/

This also passes in the extension path when specified in GIS methods. Wherever we know an extension path, we use db.init_spatialite(find_spatialite() or load_extension).

:books: Documentation preview :books:: https://sqlite-utils--531.org.readthedocs.build/en/531/

Closes #1983

All documentation here. Edits welcome.

:books: Documentation preview :books:: https://datasette--1996.org.readthedocs.build/en/1996/

These are in the codebase but not in documentation. I think everything else in this class is documented.

```python class Urls: ... def row(self, database, table, row_path, format=None): path = f"{self.table(database, table)}/{row_path}" if format is not None: path = path_with_format(path=path, format=format) return PrefixedUrlString(path)

def row_blob(self, database, table, row_path, column):
    return self.table(database, table) + "/{}.blob?_blob_column={}".format(
        row_path, urllib.parse.quote_plus(column)
    )

```

Two methods will get canned queries for a Datasette instance:

Datasette.get_canned_queries will return all canned queries for a database that an actor can see.

Datasette.get_canned_query will return a single canned query by name.

Code is here: https://github.com/simonw/datasette/blob/main/datasette/app.py#L503

This will be the official way to access metadata from plugins.

Posting this while I'm thinking about it: I mentioned at the end of this thread that I'm usually doing datasette --get to export canned queries.

I used to use a tool called datafreeze to do scripted exports, but that project looks dead now. The ergonomics of it are pretty nice, though, and the Freezefile.yml structure is actually not too far from Datasette's canned queries.

This is related to the idea for datasette query (#1356) but I think it's a distinct feature. It's most likely a plugin, but I want to raise it here because it's probably something other people have thought about.

Ran inspect on a spatialite database and got these warnings:

ERROR: conn=<sqlite3.Connection object at 0x119e46110>, sql = 'select count(*) from [SpatialIndex]', params = None: no such module: VirtualSpatialIndex ERROR: conn=<sqlite3.Connection object at 0x119e46110>, sql = 'select count(*) from [ElementaryGeometries]', params = None: no such module: VirtualElementary ERROR: conn=<sqlite3.Connection object at 0x119e46110>, sql = 'select count(*) from [KNN]', params = None: no such module: VirtualKNN

It still worked, but probably want to catch this.

I started working on using config_dir with my datasette-query-files plugin and realized Datasette doesn't actually hold onto the config_dir argument. It gets used in __init__ but then forgotten. It would be nice to be able to use it in plugins, though.

Here's the reference issue: https://github.com/eyeseast/datasette-query-files/issues/4

Closes #1764

Small change that adds self.config_dir = config_dir to Datasette.__init__. This will let plugins also use config_dir, if available.

I started sketching out a plugin to add a datasette query subcommand to export data from the command line. This is based on discussions in #1356 and #1605. Before I get too far down this rabbit hole, I figure it's worth getting some feedback here (unless this should happen in Discussions). Here's what I'm thinking:

At its most basic, it will write the results of a query to STDOUT.

sh datasette query -d data.db 'select * from data' > results.json

This isn't much improvement over using sqlite-utils. To make better use of datasette and its ecosystem, run datasette query using a canned query defined in a metadata.yml file.

For example, using the metadata file from alltheplaces-datasette:

sh cd alltheplaces-datasette datasette query -d alltheplaces.db -m metadata.yml count_by_spider

That query would be good to get as CSV, and we can auto-discover metadata and databases in the current directory:

sh cd alltheplaces-datasette datasette query count_by_spider -f csv

In this case, count_by_spider is a canned query defined on the alltheplaces database. If the same query is defined on multiple databases or its otherwise unclear which database query should use, pass the -d or --database option.

If a query takes parameters, I can pass them in at runtime, using the --param or -p option:

sh datasette query -d data.db -p value something 'select * from neighborhoods where some_column = :value'

I'm very interested in feedback on this, including whether it should be a plugin or in Datasette core. (I don't have a strong opinion about this, but I'm prototyping it as a plugin to start.)

I end up with these in my local directory:

.hypothesis/
Pipfile
Pipfile.lock
pyproject.toml

Might as well gitignore them.

Closes #418

This adds four files to .gitignore:

.hypothesis/
Pipfile
Pipfile.lock
pyproject.toml

Those are all generated in the course of development and testing.

This is a fairly new feature -- SQLite version 3.31.0 (2020-01-22) -- that I, admittedly, haven't gotten to work yet. But it looks incredibly useful: https://dgl.cx/2020/06/sqlite-json-support

I'm not sure if this is an option on add-column or a separate command like add-generated-column. Either way, it needs an argument to populate it. It could be something like this:

sh sqlite-utils add-column data.db table-name generated --as 'json_extract(data, "$.field")' --virtual

More here: https://www.sqlite.org/gencol.html

Closes #398

This adds SpatiaLite helpers to the CLI.

```sh

init spatialite when creating a database

sqlite-utils create database.db --enable-wal --init-spatialite

add geometry columns

needs a database, table, geometry column name, type, with optional SRID and not-null

this will throw an error if the table doesn't already exist

sqlite-utils add-geometry-column database.db table-name geometry --srid 4326 --not-null

spatial index an existing table/column

this will throw an error it the table and column don't exist

sqlite-utils create-spatial-index database.db table-name geometry ```

Docs and tests are included.

Now that #385 is merged, add CLI versions of those methods.

```sh

init spatialite

sqlite-utils init-spatialite database.db

or maybe/also

sqlite-utils create database.db --enable-wal --spatialite

add geometry columns

needs a database, table, geometry column name, type, with optional SRID and not-null

this needs to create a table if it doesn't already exist

sqlite-utils add-geometry-column database.db table-name geometry --srid 4326 --not-null

spatial index an existing table/column

sqlite-utils create-spatial-index database.db table-name geometry ```

Should be mostly straightforward. The one thing worth highlighting in docs is that geometry columns can only be added to existing tables. Trying to add a geometry column to a table that doesn't exist yet might mean you have a schema like {"rowid": int, "geometry": bytes}. Might be worth nudging people to explicitly create a table first, then add geometry columns.

Refs #79

This PR adds three new Spatialite-related methods to Database and Table:

• Database.init_spatialite loads the Spatialite extension and initializes it
• Table.add_geometry_column adds a geometry column
• Table.create_spatial_index creates a spatial index

Has tests and documentation. Feedback very welcome.

They're generated by Spatialite and not very interesting in most cases.

Following our conversation last week, want to note this here before I forget.

I've had a couple situations where I'd like to do a bunch of updates in an async event loop, but I run into SQLite's issues with concurrent writes. This feels like something sqlite-utils could help with.

PeeWee ORM has a SQLite write queue that might be a good model. It's using threads or gevent, but I think that approach would translate well enough to asyncio.

Happy to help with this, too.

Ran into an issue while trying to build a plugin to render GeoJSON. I'm using pytest's tmp_path fixture, which is a pathlib.Path, to get a temporary database path. I was getting a weird error involving writes, but I was doing reads. Turns out it's the internal database trying to insert a Path where it wants a string.

My test looked like this:

```python @pytest.mark.asyncio async def test_render_feature_collection(tmp_path): database = tmp_path / "test.db" datasette = Datasette([database])

# this will break with a path
await datasette.refresh_schemas()

# build a url
url = datasette.urls.table(database.stem, TABLE_NAME, format="geojson")

response = await datasette.client.get(url)
fc = response.json()

assert 200 == response.status_code

```

I only ran into this while running tests, because passing in database paths from the CLI uses strings, but it's a weird error and probably something other people have run into.

The fix is easy enough: Convert the path to a string and everything works. So this:

```python @pytest.mark.asyncio async def test_render_feature_collection(tmp_path): database = tmp_path / "test.db" datasette = Datasette([str(database)])

# this is fine now
await datasette.refresh_schemas()

```

This could (probably, haven't tested) be fixed here by calling str(db.path) or by doing that conversion earlier.

Fixes #1365

This is the simplest possible fix, with a test that will fail without it. There are a bunch of places where db.path is getting converted to and from a Path type, so this fix errs on the side of calling str(db.path) right before it's inserted.

I have a project going where it would be useful to do some spatial processing in SQLite (instead of large files) and then output GeoJSON. So my workflow would be something like this:

1. Read Shapefiles, GeoJSON, CSVs into a SQLite database
2. Join, filter, prune as needed
3. Export GeoJSON for just the stuff I need at that moment, while still having a database of things that will be useful later

I'm wondering if this is worth adding to SQLite-utils itself (GeoJSON, at least), or if it's better to make a counterpart to the ecosystem of *-to-sqlite tools, say a suite of sqlite-to-* things. Or would it be crazy to have a plugin system?