issue_comments

I just remembered that there's one other place that this could fit: as a Datasette "insert" plugin.

This is vaporware at the moment, but the idea is that Datasette itself could grow a mechanism for importing data, that's driven by plugins.

Out of the box Datasette would be able to import CSV and CSV files, similar to sqlite-utils insert ... --csv - but plugins would then be able to add support for additional format such as GeoJSON or - in this case - Parquet.

The neat thing about having it as a Datasette plugin is that one plugin would enable three different ways of importing data:

1. Via a new datasette insert ... CLI option (similar to sqlite-utils)
2. Via a web form upload interface, where authenticated Datasette users would be able to upload files
3. Via an API interface, where files could be programatically submitted to a running Datasette server

I started fleshing out this idea quite a while ago but didn't make much concrete progress, maybe I should revisit it:

• https://github.com/simonw/datasette/issues/1160

Thanks for your reply @simonw . Tbh, my first attempt was actually the parquet-to-sqlite package but I already had Makefiles that relied on SQLite-utils and it was less intrusive to my workflow. Maybe I'll revisit that decision. FYI: there's a [sqlite-parquet-vtable](https://github.com/cldellow/sqlite-parquet-vtable)

I don't think plugins make much sense either. Probably defeats the purpose of simplicity: simple database along with a pip-able package.

Some other recent projects (like trying to get this library to work in JupyterLite) have made me much more cautious about adding new dependencies, especially dependencies like pyarrow which require custom C/Rust extensions.

There are a few ways this could work though:

• Have this as an optional dependency feature - so it only works if the user installs pyarrow as well
• Implement this as a separate tool, parquet-to-sqlite - which could itself depend on sqlite-utils
• Add a concept of "plugins" to sqlite-utils, similar to how those work in Datasette: https://docs.datasette.io/en/stable/plugins.html

My favourite option is parquet-to-sqlite because that can be built without any additional changes to sqlite-utils at all!

I find the concept of plugins for sqlite-utils interesting. I've so far not had quite enough potential use-cases to convince me this is worthwhile (especially since it should be very easy to build out separate tools entirely), but I'm ready to be convinced that a plugin mechanism would be worthwhile.