issue_comments

That's a great idea. I'd ruled that out because working with the different operating system versions of those is tricky, but if watchdog can handle those differences for me this could be a really good option.

I'm going to move this over to a draft pull request.

MVP for this feature: just do it once on startup, don't scan for new files every X seconds.

OK, I have a plan. I'm going to try and implement this is a core Datasette feature (no plugins) with the following design:

• You can tell Datasette "load any databases you find in this directory" by passing the --dir=path/to/dir option to datasette that are valid SQLite files and will attach them to Datasette
• Every 10 seconds Datasette will re-scan those directories to see if any new files have been added
• That 10s will be the default for a new --config directory_scan_s:10 config option. You can set this to 0 to disable scanning entirely, at which point Datasette will only run the scan once on startup.

To check if a file is valid SQLite, Datasette will first check if the first few bytes of the file are b"SQLite format 3\x00". If they are, it will open a connection to the file and attempt to run select * from sqlite_master against it. If that runs without any errors it will assume the file is usable and connect it.

I'm going to add two methods to the Datasette class to help support this work (and to enable exciting new plugin opportunities in the future):

• datasette.add_database(name, db) - adds a new named database to the list of connected databases. db will be a Database() object, which may prove useful in the future for things like #670 and could also allow some plugins to provide in-memory SQLite databases.
• datasette.remove_database(name)

A neat ability of Datasette Library would be if it can work against other files that have been dropped into the folder. In particular: if a user drops a CSV file into the folder, how about automatically converting that CSV file to SQLite using sqlite-utils?

This would allow Datasette to be easily used as a "data library" (like a data warehouse but less expectation of big data querying technology such as Presto).

One of the things I learned at the NICAR CAR 2019 conference in Newport Beach is that there is a very real need for some kind of easily accessible data library at most newsrooms.