issue_comments

That's fair. The best idea I can think of is that if a plugin wanted to limit intensive queries, it could add LIMITs or something. A hook that gives you visibility of queries and maybe the option to reject felt a little more limited than the existing plugin hooks, so I was trying to think of what else one might want to do while looking at to-be-run queries.

But without a real motivating example, I see why you don't want to add that.

I'm less convinced by the "rewrite the query in some way" optional idea. What kind of use-cases can you imagine for that?

My hunch is that it's much more likely to cause weird breakages than it is to allow for useful plugin extensions, but I'm willing to be convinced otherwise.

Interestingly enough there's actually a mechanism that looks like that a bit already: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/views/database.py#L496-L508

That validate_sql_select() function is defined here: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/init.py#L256-L265

Against these constants:

https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/init.py#L223-L253

Which isn't a million miles away from your suggestion to have a hook that can say if the query should be executed or not.

Thanks for all this! You're totally right that the ASGI option is doable, if a bit low level and coupled to the current URI design. I'm totally fine with that being the final answer.

process_view is interesting and in the general direction of what I had in mind.

A somewhat less powerful idea: Is there value in giving a hook for just the query that's about to be run? Maybe I'm thinking a little narrowly about this problem I decided I wanted to solve, but I could see other uses for a hook of the sketch below:

def prepare_query(database, table, query): """Modify query that is about to be run in some way. Return the (possibly rewritten) query to run, or None to disallow running the query""" (Maybe you actually want to return a tuple so there can be an error message when you disallow, or something.)

Maybe it's too narrowly useful and some of the other pieces of datasette obviate some of these ideas, but off the cuff I could imagine using it to: * Require a LIMIT. Either fail the query or add the limit if it's not there. * Do logging, like my usecase. * Do other analysis on whether you want to allow the query to run; a linter? query complexity?

Definitely feel free to say no, or not now. This is all me just playing around with what datasette and its plugin architecture can do with toy ideas, so don't let me push you to commit to a hook you don't feel confident fits well in the design.

Are you talking just about queries submitted to /database?sql= using the interface on https://latest.datasette.io/fixtures?sql=select+*+from+facetable or are you interested in queries that are run to power other pages like https://latest.datasette.io/fixtures/facetable as well? I'll assume the former.

There are a few ways you could solve this at the moment.

The easiest would be with a piece of ASGI middleware that looks for URLs matching /dbname?sql=... and logs those. I played with a version of that a few years ago: https://simonwillison.net/2019/Dec/16/logging-sqlite-asgi-middleware/ - see also https://github.com/simonw/asgi-log-to-sqlite

Then you can load that middleware from a plugin using https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette

That feels a bit delicate because it's relying on the URL design not changing, but I'm happy to confirm that URL is going to stay the same for Datasette 1.0 and I have no plans to change it ever.

There's also a tracing mechanism built into Datasette itself that you could hook into. The internals of that are documented here: https://docs.datasette.io/en/stable/internals.html#datasette-tracer - but I don't yet consider it a 100% stable API. I don't plan to change it but I won't promise not to either.

I used that mechanism in this plugin: https://datasette.io/plugins/datasette-pretty-traces - demonstrated here: https://latest-with-plugins.datasette.io/github/commits?_trace=1

The hackiest way to do this would be to patch Datasette itself and try to replace the query_view. This definitely isn't a documented, stable API though and would be very likely to break at arbitrary points in the future.

So my recommendation for the moment is the ASGI middleware option.

I'm definitely open to suggestions for plugin hooks that might make this kind of thing easier.

One idea I've been mulling is whether there should be a plugin hook that files on arbitrary views - similar to Django's process_view mechanism: https://docs.djangoproject.com/en/4.2/topics/http/middleware/#process-view

That would allow people to setup code that runs before or after any of the default views in Datasette.

I'm not yet 100% sold on the idea, because I worry about implementing it in a way that guarantees plugins won't break on future releases. But I'm open to considering it.