issue_comments

I'm not sure how I missed this issue but it's almost a year later and I'm finally taking a look at your Parquet work.

This is yet more evidence that allowing plugins to provide their own custom Database objects would be a good idea.

I started exploring what Datasette would like on PostgreSQL in #670 - my concern was that I would need to add a large amount of database abstraction code which would dramatically increase the complexity of the core project, but my thinking now is that it might be tractable - Datasette doesn't actually construct SQL in complex ways anywhere outside of the TableView class so abstracting away just that bit should be feasible.

I think the startup() plugin hook at https://docs.datasette.io/en/stable/plugin_hooks.html#startup-datasette should be able to fit this. You can write a plugin which uses that hook to execute CREATE VIRTUAL TABLE against one or more databases when Datasette first starts running.

Would that work here?

Hi,

I've completed some changes to my fork of datasette that allows it to automatically create the parquet virtual table when you supply it with a filename that has the ".parquet" extension.

I had to figure out how to make the "CREATE VIRTUAL TABLE" statement only be applied to the fake in memory parquet database and not to any others that were also being loaded. Thus it supports mixed mode databases e.g

datasette my_test.parquet normal_sqlite_file.db --load-extension=libparquet.so --load-extensio n=mod_spatialite.so

Please see my changes here:
https://github.com/dazzag24/datasette/commit/8e18394353114f17291fd1857073b1e0485a1faf

Thanks

I have successfully tested datasette using a parquet VIRTUAL TABLE. In the first terminal:

datasette airports.db --load-extension=libparquet

In another terminal I load the same sqlite db file using the sqlite3 cli client.

$ sqlite3 airports.db

and then load the parquet extension and create the virtual table.

sqlite> .load /home/darreng/metars/libparquet sqlite> CREATE VIRTUAL TABLE mytable USING parquet('/home/xx/data.parquet');

Now the parquet virtual table is usable by the datasette web UI.

Its not an ideal solution but is a proof that datasette works the parquet extension.