when you are running from docker, you always will want to run as mode=ro because the same thing that is causing duplication in the inspect layer will cause duplication in the final container read/write layer when datasette serve runs.

@simonw, should i open up a new issue for investigating the differences between "immutable=1" and "mode=ro" and possibly switching to "mode=ro". Or would you like to keep that conversation in this issue?

yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in immutable mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!

running a test of that now.

this completely addressed #1480

when i hack the connect method to open non mutable files with "mode=ro" and not "immutable=1" https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/database.py#L79

then:

bash 870 B RUN /bin/sh -c datasette inspect nlrb.db --inspect-file inspect-data.json

the datasette inspect layer is only the size of the json file!

yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in immutable mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!

running a test of that now.

and if we try immutable mode, which is how things are opened by datasette inspect we duplicate the files!!!

```python

test_sql_immutable.py

import sqlite3 import sys

db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}?immutable=1', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ```

fascinatingly, telling python to open sqlite in read only mode makes this layer have a size of 0

```python

test_sql_ro.py

import sqlite3 import sys

db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}?mode=ro', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ```

that's quite weird because setting the file permissions to read only didn't do anything. (on reflection, that chmod isn't doing anything because the dockerfile commands are run as root)

okay, some progress!! running some sql against a database file causes that file to get duplicated even if it doesn't apparently change the file.

make a little test script like this:

```python

test_sql.py

import sqlite3 import sys

db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ```

then

docker RUN python test_sql.py nlrb.db

produced a layer that's the same size as nlrb.db!!

it's not that the inspect command is somehow changing the db files. if i set them to only read-only, the "inspect" layer still has the same very large size.

i thought it was maybe to do with reading through all the files, but that does not seem to be the case

if i make a little test file like:

```python

test_read.py

import hashlib import sys import pathlib

HASH_BLOCK_SIZE = 1024 * 1024

def inspect_hash(path): """Calculate the hash of a database, efficiently.""" m = hashlib.sha256() with path.open("rb") as fp: while True: data = fp.read(HASH_BLOCK_SIZE) if not data: break m.update(data)

return m.hexdigest()

inspect_hash(pathlib.Path(sys.argv[1])) ```

then a line in the Dockerfile like

docker RUN python test_read.py nlrb.db && echo "[]" > /etc/inspect.json

just produes a layer of 3B