issue_comments

We've grown a bunch of plugin hooks over the past two years: https://datasette.readthedocs.io/en/latest/plugins.html#plugin-hooks

Since the plugin system will never be 100% "finished", I'm closing this in favor of the label: https://github.com/simonw/datasette/labels/plugins

Here's a link demonstrating my new plugin: https://datasette-cluster-map-demo.now.sh/polar-bears-455fe3a/USGS_WC_eartags_output_files_2009-2011-Status

I released everything we have so far in Datasette 0.20 and built and released an example plugin, datasette-cluster-map. Here's my blog entry about it: https://simonwillison.net/2018/Apr/20/datasette-plugins/

I added a mechanism for plugins to serve static files and define custom CSS and JS URLs in #214 - see new documentation on http://datasette.readthedocs.io/en/latest/plugins.html#static-assets and http://datasette.readthedocs.io/en/latest/plugins.html#extra-css-urls

I just shipped Datasette 0.19 with where I'm at so far: https://github.com/simonw/datasette/releases/tag/0.19

I think that's OK. The two plugins I've implemented so far (prepare_connection and prepare_jinja2_environment) both make sense if they can only be defined once-per-plugin. For the moment I'll assume I can define future hooks to work well with the same limitation.

The syntactic sugar idea in #220 can help here too.

Annoyingly, the following only results in the last of the two prepare_connection hooks being registered:

``` from datasette import hookimpl import pint import random

ureg = pint.UnitRegistry()

@hookimpl def prepare_connection(conn): def convert_units(amount, from_, to_): "select convert_units(100, 'm', 'ft');" return (amount * ureg(from_)).to(to_).to_tuple()[0] conn.create_function('convert_units', 3, convert_units)

@hookimpl def prepare_connection(conn): conn.create_function('random_integer', 2, random.randint) ```

I should check if it's possible to have two template registration function plugins in a single plugin module. If it isn't maybe I should use class plugins instead of module plugins.

OK, from that prototype in f2720b0c6b7172ebe8820 it looks like pluggy provides a solid path forward.

Next steps:

• [x] Build a demo plugin that uses setuptools entrypoints to register with the datasette plugin manager via pluggy
• [x] Figure out a mechanism for registering plugins without first needing to publish them to PyPI. Can I load plugins from a special plugins/ directory similar to the --template-dir=templates/ option already supported by Datasette? #211

Slight code design problem... when I tried installing my branch in a fresh virtual environment I got this error, because setup.py now depends on pluggy (from importing __version__):

File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/setup.py", line 2, in <module> from datasette import __version__ File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/datasette/__init__.py", line 2, in <module> from .hookspecs import hookimpl # noqa File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/datasette/hookspecs.py", line 1, in <module> from pluggy import HookimplMarker ModuleNotFoundError: No module named 'pluggy'

Looks like I've run into point 6 on https://packaging.python.org/guides/single-sourcing-package-version/ :

I created https://github.com/simonw/datasette-plugin-demos which is now published to PyPI and can be installed with pip install datasette-plugin-demos - I've confirmed that if you DO install it my Datasette plugins branch picks up the plugins, and select random_integer(1, 4) works as it should.

Once I've got the plugins mechanism stable and people start releasing plugins it would be useful to have a dedicated Trove classifier on PyPI for Datasette plugins - Framework :: Datasette for example.

This would help me build a Datasette equivalent of the http://plugincompat.herokuapp.com/ site, which works by scanning PyPI for items with the Framework :: Pytest classifier:

https://github.com/pytest-dev/plugincompat/blob/8bdf1a6fb82807091ece0c68c196103ee8270194/update_index.py#L52-L53

It looks like the mechanism for requesting new PyPI classifiers is to file a ticket against warehouse, like these ones: https://github.com/pypa/warehouse/issues/3570 and https://github.com/pypa/warehouse/issues/2881

Here's a demo of the convert_units() SQL function I prototyped in f2720b0c6b7172ebe88

Tox is a good example of a project that uses pluggy in the way I want to use it (function hooks rather than classes): https://github.com/tox-dev/tox/blob/master/tox/hookspecs.py

Datasette 1.0 will be the release of Datasette that attempts to provide a stable plugin API: https://github.com/simonw/datasette/milestone/7

There's a lot of work to be done before then, but as a starting point I'm going to support two very simple extension mechanisms:

• Template system plugins - where the hook gets passed the Jinja environment and can freely register new template tags and filters
• SQLite connection plugins - where the hook gets passed a new SQLite connection and can register custom SQLite functions

The template system hook will go near here:

https://github.com/simonw/datasette/blob/efbb4e83374a2c795e436c72fa79f70da72309b8/datasette/app.py#L1225-L1228

The SQLite connection hook will go near here:

https://github.com/simonw/datasette/blob/efbb4e83374a2c795e436c72fa79f70da72309b8/datasette/app.py#L1094-L1098

These two feel simple enough that I'm not worried that I might design an API that I later regret.

I started a thread on Twitter asking people for good examples of Python projects with a strong plugin ecosystem: https://twitter.com/simonw/status/985377670388105216

The most impressive example that came back was pytest - which now has nearly 400 plugins: https://plugincompat.herokuapp.com/

The pytest plugin infrastructure is available as an independent package called pluggy - which appears to offer everything I need for Datasette. I'm going to give that a go and see how well it works: https://pluggy.readthedocs.io/en/latest/

Oh thanks, that definitely looks like an interesting option.

I'd also suggest taking a look at stevedore, which has a ton of tools for doing plugin stuff. I've had good luck with it in the past.

http://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins Is pretty good

For visualizations, Google Maps should be made available as a plugin. The default visualizations can use Leaflet and Open Street Map, but there's no reason to not make Google Maps available as a plugin, especially if the plugin can provide a mechanism for configuring the necessary API key.

I'm particularly excited in the Google Maps heatmap visualization https://developers.google.com/maps/documentation/javascript/heatmaplayer as seen on http://mochimachine.org/wasteland/

Plugins should be able to interact with the build step. This would give plugins an opportunity to modify the SQL databases and help prepare them for serving - for example, a full-text search plugin might create additional FTS tables, or a mapping plugin might pre-calculate a bunch of geohashes for tables that have latitude/longitude values. Plugins could really take advantage of the immutable nature of the dataset here.

The plugin system can also allow alternative providers for the publish command - e.g. maybe hook up hyper.sh as an option for publishing containers.