Thanks for the review and the code pointers @simonw - I've made the suggested edits, fixed the renamed variable, and confirmed that the panels still render on the table and database views.

Ok @hydrosquall a couple things before this PR should be good to go:

Thank you @asg017 ! I've pushed both suggested changes onto this branch.

Not sure how difficult it'll be to inject it server-side

If we are OK with having a build system, it would free me up to do do many things! We could make datasette-manager.js a server-side rendered file as a "template" instead of having it as a static JS file, but I'm not sure it's worth the extra jump in complexity / loss of syntax highlighting in the JS file.

In the short-term, I could see an intermediary solution where a unit test in the preferred language was able to read both version.py and datasette-manager.js, and make sure that the strings versions are in sync. (This assumes that we want the manager and datasette's versions to be synced, and not decoupled). Since the version is not changing very often, a "manual sync" might be good enough.

In terms of how to integrate this into Datasette, a few options I can see working:

This sounds good to me. I'm not sure how to add a settings flag, but will be interested to see the PR that adds support for it.

I'm also curious to see how "plugins for a plugin' would work

I'm comfortable to wait until we have a realistic usecase for this. In the short term, I think we could give plugins a way to grant access to a "public API of other plugins", and also ask to be notified when plugins with other names have loaded, but don't picture the datasette manager getting more involved than that.

here's a list of Simon's Datasette plugins that use "extra_js_urls()"

Neat, thanks for compiling this list! Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?

I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.)

I look forward to trying this out 👍

Thanks @cldellow for the thoughtful comments! These are all things that I'll keep in mind as we figure out how/if this API is actually used by plugin authors once it's actually out in the world.

Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.

Ah, I understand now! Thanks for explaining.

Hey @cldellow , thanks for the thoughtful feedback and describing the "lazy facets" feature!

It sounds like the postTask API might be relevant for the types of network request scheduling you have in mind.

Addressing your points inline below:

It might also be nice if the plugins could return Promises.

Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like makeColumnActions or makeAboveTablePanelConfigs supported returning a promise of arrays instead only returning plain arrays?

I think what you're describing can be achievable, but I want to make sure I do so in a way that addresses your need / keeps the complexity of the plugin core system at a level this is approachable .

I have a hunch that what you're describing might be achievable without adding Promises to the API with something like

fetch('/api/with-custom-facets').then(myFacets => { // reusing the go() idiom go(manager, myFacets); })

but I'd like to confirm if that's the case before investigating adding support.

bulletproof plugin registration code that is robust against the order in which the script tags load

Yes, I think what you wrote looks right to me! While it looks a little bit verbose compared to the second example, I'm hoping we can mitigate the cost of that during this API incubation phase by making it an easy-to-copy paste code snippet.

I haven't heard of the GA queing pattern before, thanks for the example. I won't have time to implement of proof of concept in the next few weeks, but I took some time to think through the pros/cons to decide whether we may want to add this in a future release:

I can see that this approach brings advantages

• Plugin developers don't need to know the name of the datasette initialization event to start their plugin
• Pushing a function to an array probably is easier (definitely more concise) than adding a document event listener
• One less event listener sitting in memory

It also has some minor costs

• A malicious plugin could choose to (or accidentally) mess with the order of the queue if multiple scripts are lined up
• Some risk in encouraging people to mutate global state
• (not a cost, more a moot point): changing this API may not make a meaningful difference if we're discussing whether people enter 2 vs 5 lines of code, especially if those lines are encapsulated by a function we provide (maybe something that's available on the window provided by Datasette as an inline script tag).

Javascript Plugin Docs (alpha)

Motivation

The Datasette JS Plugin API allows developers to add interactive features to the UI, without having to modify the Python source code.

Setup

No external/NPM dependencies are needed.

Plugin behavior is coordinated by the Datasette manager. Every page has 1 manager.

There are 2 ways to add your plugin to the manager.

1. Read window.__DATASETTE__ if the manager was already loaded.

js const manager = window.__DATASETTE__;

1. Wait for the datasette_init event to fire if your code was loaded before the manager is ready.

```js document.addEventListener("datasette_init", function (evt) { const { detail: manager } = evt;

// register plugin here }); ```

1. Add plugin to the manager by calling manager.registerPlugin in a JS file. Each plugin will supply 1 or more hooks with

2. unique name (YOUR_PLUGIN_NAME)

3. a numeric version (starting at 0.1),
4. configuration value, the details vary by hook. (In this example, getColumnActions takes a function)

js manager.registerPlugin("YOUR_PLUGIN_NAME", { version: 0.1, makeColumnActions: (columnMeta) => { return [ { label: "Copy name to clipboard", // evt = native click event onClick: (evt) => copyToClipboard(columnMeta.column), } ]; }, });

There are 2 plugin hooks available to manager.registerPlugin:

• makeColumnActions - Add items to the cog menu for headers on datasette table pages
• makeAboveTablePanelConfigs - Add items to "tabbed" panel above the <table/> on pages that use the Datasette table template.

While there are additional properties on the manager, but it's not advised to depend on them directly as the shape is subject to change.

1. To make your JS file available as a Datasette plugin from the Python side, you can add a python file resembling this to your plugins directory. Note that you could host your JS file anywhere, it doesn't have to be served from the Datasette statics folder.

I welcome ideas for more hooks, or feedback on the current design!

Examples

See the example plugins file for additional examples.

Hooks API Guide

makeAboveTablePanelConfigs

Provide a function with a list of panel objects. Each panel object should contain

1. A unique string id
2. A string label for the tab
3. A render function. The first argument is reference to an HTML Element.

Example:

js manager.registerPlugin("panel-plugin-graphs", { version: 0.1, makeAboveTablePanelConfigs: () => { return [ { id: 'first-panel', label: "My new panel", render: node => { const description = document.createElement('p'); description.innerText = 'Hello world'; node.appendChild(description); } } ]; }, });

makeColumnActions

Provide a function that returns a list of action objects. Each action object has

1. A string label for the menu dropdown label
2. An onClick render function.

Example:

```js manager.registerPlugin("column-name-plugin", { version: 0.1, getColumnActions: (columnMeta) => {

  // Info about selected column. 
  const { columnName, columnNotNull, columnType, isPk } = columnMeta;

  return [
    {
      label: "Copy name to clipboard",
      onClick: (evt) => copyToClipboard(column),
    }
  ];
},

}); ```

The getColumnActions callback has access to an object with metadata about the clicked column. These fields include:

• columnName: string (name of the column)
• columnNotNull: boolean
• columnType: sqlite datatype enum (text, number, etc)
• isPk: Whether this is the primary key: boolean

You can use this column metadata to customize the action config objects (for example, handling different summaries for text vs number columns).

Thanks for the thoughtful review and generous examples @asg017 ! I'll make the changes you suggested soon. Bonus thoughts inlined below.

comments

These were very much appreciated, it's important to a plugin system that details like this feel right! I'll address them in batch later in the week.

I know TypeScript can be a little controversial

FWIW I am in favor of doing Typescript - I just wanted to keep the initial set of files in this PR as simple as possible to review. Really appreciate you scaffolding this initial set of types + I think it would be a welcome addition to maintain a set of types.d.ts files.

I'm entertaining the idea of writing the actual source code in Typescript as long as the compiled output is readable b/c it can be tricky to keep the types and plain JS files in sync. Curious if you have encountered projects that are good at preventing drift.

Maybe they should have more "action-y" names

This is a great observation. I'm inclined towards something like make* or build* since to me add* make me think the thing the method is attached to is being mutated, but I agree that any of these may be clearer than the current get* setup. I'll go through and update these.

Maybe we can make it easier to do pure-js datasette plugins?

I really like this idea! It'll be easier to get contributors if they don't have to touch the python side at all.

And then do the PERMITTED_VIEWS filtering in JS rather than Python.

One cost of doing this is that pages that won't use the JS would still have to load the unused code (given that I'm not sending up anything complex like lazy loading). But hopefully the manager core size is close to negligible, and it won't be a big deal.

Research notes

• I stuck to the "minimal dependencies" ethos of datasette (no React, Typescript, JS linting, etc).
• Main threads on JS plugin development
• Main: sketch of pluggy-inspired system: https://github.com/simonw/datasette/issues/983
• Main: provide locations in Datasette HTML that are designed for multiple plugins to safely cooperate with each other (starting with the panel, but eventually could extend to "search boxes"): https://github.com/simonw/datasette/issues/1191
• Main: HTML hooks for JS plugin authors: https://github.com/simonw/datasette/issues/987
• Prior threads on JS plugins in Datasette for future design directions
• Idea: pass useful strings to JS plugins: https://github.com/simonw/datasette/issues/1565
• Idea: help with plugin dependency loading: https://github.com/simonw/datasette/issues/1542 . (IMO - the plugin providing the dependency can emit an event once it's done. Other plugins can listen for it, or ask the manager to inform them when the dependency is available).
• Idea: help plugins to manage state in shareable URLs (plugins shouldn't have to interact with the URL directly, should have some basic insulation from clobbering each others' keys): https://github.com/simonw/datasette/issues/1144
• Articles on plugins reviewed
• https://css-tricks.com/designing-a-javascript-plugin-system/
• Plugin/Extension systems reviewed (mostly JS).
• Yarn: https://yarnpkg.com/advanced/plugin-tutorial
• Tappable https://github.com/webpack/tapable (used by Auto, webpack)
• Pluggy: https://pluggy.readthedocs.io/en/stable/
• VSCode: https://code.visualstudio.com/api/get-started/your-first-extension
• Chrome: https://developer.chrome.com/docs/extensions/reference/
• Figma/Figjam Widget: https://www.figma.com/widget-docs/
• Datadog Apps: Programming Model
• Storybook: https://storybook.js.org/docs/react/addons/addons-api

Notes from 1:1 - it is possible to pass in URL params into a ObservableHQ notebook: https://observablehq.com/@bherbertlc/pass-values-as-url-parameters

My current workaround is to use this library from a python script instead of as a CLI tool.

This lets me set the foreign key constraint at table creation time, instead of trying to modify an existing table. docs

I found this stackoverflow helpful, as it explained that Sqlite doesn't support modifying existing tables directly.

A related historical problem:

https://github.com/tekartik/sqflite/issues/525#issuecomment-714500720

I wonder if the version of Sqlite or Python for Intel chip have defensive mode disabled by default, whereas M1 chips versions have it enabled.

Hi @ryascott, thanks for sharing this! How did you upgrade your sqlite3 version? I'm running into this issue (also on an m1) with

Python ==3.10.7 sqlite3.sqlite_version==3.37.0 sqlite-utils==3.30

Unfortunately, 3.10.8 isn't listed in pyenv so I'm unable to install it.

For me, the trigger is trying to use the add-foreign-key command on its own:

bash sqlite-utils add-foreign-key library.db book_creators creator_id creators id

Some stackoverflow searching suggests that brew installing sqlite may fix it ( https://stackoverflow.com/questions/26345972/how-do-i-upgrade-the-sqlite-version-used-by-pythons-sqlite3-module-on-mac ), but I don't want to risk breaking the version of sqlite used by some other system, I'd only like to upgrade sqlite3 inside my current virtual environment.

Happy birthday to datasette and thank you Simon for your continued effort on this project!

I use datasette (python) as a fast layer on top of search for github projects using https://github.com/dogsheep/github-to-sqlite , and use the JSON API it provides to serve sample data to make Vega-Lite graphing workshop examples that don't require authentication/API keys. It's awesome to have a full SQL API support working without needing to develop any custom API middleware for both filtering and grouping.

I've also enjoyed using it as a teaching tool for working with public dataset in civic data workshops and as a platform for making visualization plugins . I

I'm especially excited about datasette-lite, as it will let people participate in future editions of this workshop without having to install anything to make use of their own tables :)

I've been looking into how to to get this data out of Github (especially now there are "secondary rate limits" without an advertised allowance separate from the regular rate limits.

I've had decent success with the Airbyte github extractor (aside from one data quality issue https://github.com/airbytehq/airbyte/pull/15420 ). Airbyte splits data extraction between the GraphQL and REST endpoints depending on the resource type, but they're very comprehensive.

https://github.com/airbytehq/airbyte/blob/306a75ef5370728e0912cf52a1a898a530db0c90/airbyte-integrations/connectors/source-github/source_github/streams.py#L22-L122

Before this, I tried a few solutions in my own custom wrapper mentioned in this thread + its children https://github.com/PyGithub/PyGithub/issues/1989 , but they weren't working as expected.

Thank you! I went through the cookiecutter template, and published my first package here: https://github.com/hydrosquall/datasette-nteract-data-explorer

Is there a way to serve a static assets when using the plugins/ directory method instead of installing plugins as a new python package?

As a workaround, I found I can serve my statics from a non-plugin specific folder using the --static CLI flag.

bash datasette ~/Library/Safari/History.db \ --plugins-dir=plugins/ \ --static assets:dist/

It's not ideal because it means I'll change the cache pattern path depending on how the plugin is running (via pip install or as a one off script), but it's usable as a workaround.