issue_comments: 1708777964

Here's the thinking: setting theDATASETTE_INTERNAL environment variable for your entire machine will cause any time you run datasette x.db to result in a process that shares the same internal.db database as other instances.

You might run more than one instance at once (I often have 4 or 5 going). This would currently break, because they would over-write each other's catalog tables:

https://github.com/simonw/datasette/blob/e4abae3fd7a828625d00c35c316852ffbaa5ef2f/datasette/utils/internal_db.py#L5-L62

The breaking wouldn't be obvious because the catalog tables aren't used by any features yet, but it's still bad.

This convinced us that actually we should move those catalog_ tables OUT of internal.db. The _internal database will be reserved for plugins that want to use it for caching, storing progress, etc.

I think we move them to an in-memory _catalog database which is excluded from ds.databases (like _internal is ) but can be accessed using datasette.get_catalog_database() - similar to datasette.get_internal_database().

So each instance of Datasette gets its own truly private _catalog, which is in-memory and so gets cleared at the end of each process.

An interesting thing that came up about a shared _internal database is that it provides opportunities for things like a notes plugin which allows you to attach notes to any row in any table in a database... where those notes become available to multiple Datasette instances that you might launch on the same laptop.