issue_comments

I released a plugin that implements an early version of this a while ago: https://github.com/simonw/datasette-search-all

I'm still thinking about how a plugin / separate tool could work that builds a single FTS index over the content from multiple tables such that you can run relevance-calculated queries against those multiple tables.

Since that's going to be a plugin / separate tool too, I'm closing this issue.

@simonw I made this primitive search that I've put in my Datasette project's custom templates directory: https://gist.github.com/chrismp/e064b41f08208a6f9a93150a23cf7e03

I'll take your suggestion and do this all in Javascript. Would I need to make a static/ folder in my datasette project's root directory and make a custom index.html template that pulls from static/js/search-all-fts.js? Or would you suggest another way?

In my case, I have an ever-growing number of databases and tables within them. Most tables have FTS enabled. I cannot predict the names of future tables and databases, nor can I predict the names of the columns for which I wish to enable FTS.

For my purposes, I was thinking of writing up something that sends these two GET requests to each of my databases' tables.

http://my-server.com/database-name/table-name.json?_search=mySearchString http://my-server.com/database-name/table-name.json

In the resulting JSON strings, I'd check the value of the key filtered_table_rows_count. If the value is 0 in the first URL's result, or if values from both requests are the same, that means FTS is either disabled for the table or it has no rows matching the search query.

Is this feasible within the datasette library, or would it require some type of plugin? Or maybe you know of a better way of accomplishing this goal. Maybe I overlooked something.

Hmmm... you know I think I actually need this for my https://github.com/simonw/fara-datasette project...

https://fara.datasettes.com/fara?sql=select+1+as+sortme%2C+%27FARA_All_RegistrantDocs%27+as+%27table%27%2C+rowid+as+id%2C+Document_Type+%7C%7C+%27+%27+%7C%7C+Short_Form_Name+%7C%7C+%27+for+%27+%7C%7C+Registrant_Name+as+title%0D%0Afrom+FARA_All_RegistrantDocs%0D%0A++where+rowid+in+%28select+rowid+from+FARA_All_RegistrantDocs_fts+where+FARA_All_RegistrantDocs_fts+match+%3Asearch%0D%0A++++order+by+rank+limit+10%0D%0A%29%0D%0Aunion%0D%0Aselect+2+as+sortme%2C+%27FARA_All_ForeignPrincipals%27+as+%27table%27%2C+rowid+as+id%2C+Foreign_Principal+%7C%7C+%27+for+%27+%7C%7C+%22Country%2FLocation_Represented%22+as+title%0D%0Afrom+FARA_All_ForeignPrincipals%0D%0A++where+rowid+in+%28select+rowid+from+FARA_All_ForeignPrincipals_fts+where+FARA_All_ForeignPrincipals_fts+match+%3Asearch%0D%0A++++order+by+rank+limit+10%0D%0A%29%0D%0Aunion%0D%0Aselect+3+as+sortme%2C+%27FARA_All_Registrants%27+as+%27table%27%2C+rowid+as+id%2C+Name+%7C%7C+%27+on+%27+%7C%7C+Registration_Date+%7C%7C+%27+in+%27+%7C%7C+City+as+title%0D%0Afrom+FARA_All_Registrants%0D%0A++where+rowid+in+%28select+rowid+from+FARA_All_Registrants_fts+where+FARA_All_Registrants_fts+match+%3Asearch%0D%0A++++order+by+rank+limit+10%0D%0A%29%0D%0Aunion%0D%0Aselect+4+as+sortme%2C+%27FARA_All_ShortForms%27+as+%27table%27%2C+rowid+as+id%2C+Short_Form_First_Name+%7C%7C+%27+%27+%7C%7C+Short_Form_Last_Name+%7C%7C+%27+in+%27+%7C%7C+City+as+title%0D%0Afrom+FARA_All_ShortForms%0D%0A++where+rowid+in+%28select+rowid+from+FARA_All_ShortForms_fts+where+FARA_All_ShortForms_fts+match+%3Asearch%0D%0A++++order+by+rank+limit+10%0D%0A%29%0D%0A%0D%0A&search=cohen

Eh, I'm not concerned with a relevance score right now. I think I'd be fine with a search whose results show links to data tables with at least one result.

You can use a database query against sqlite_master to find all the FTS tables:

http://sf-trees.datasettes.com/trees-b64f0cb?sql=select+name+from+sqlite_master+where+name+like+%22%25_fts%22+and+type+%3D+%22table%22

select name from sqlite_master where name like "%_fts" and type = "table"

You could then construct a crafty UNION query to get results back from all of those tables at once:

http://sf-trees.datasettes.com/trees-b64f0cb?sql=select+%27PlantType_value_fts%27%2C+rowid%2C+value+from+PlantType_value_fts+where+PlantType_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qCareAssistant_value_fts%27%2C+rowid%2C+value+from+qCareAssistant_value_fts+where+qCareAssistant_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qCaretaker_value_fts%27%2C+rowid%2C+value+from+qCaretaker_value_fts+where+qCaretaker_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qLegalStatus_value_fts%27%2C+rowid%2C+value+from+qLegalStatus_value_fts+where+qLegalStatus_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qSiteInfo_value_fts%27%2C+rowid%2C+value+from+qSiteInfo_value_fts+where+qSiteInfo_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qSpecies_value_fts%27%2C+rowid%2C+value+from+qSpecies_value_fts+where+qSpecies_value_fts+match+%3Asearch&search=fi%2A

(I'm searching for fi* here to demonstrate wildcards)

The problem, as discussed earlier, is relevance: there's no way to compare the scores you're getting across different tables, so you won't be able to order by anything.

Very cool, thank you.

Using http://search-24ways.herokuapp.com as an example, let's say I want to search all FTS columns in all tables in all databases for the word "web."

Here's a link to the query I'd need to run to search "web" on FTS columns in articles table of the 24ways database.

And here's a link to the JSON version of the above result. I'd like to get the JSON result of that query for each FTS table of each database in my datasette project.

Is it possible in Javascript to automate the construction of query URLs like the one I linked, but for every FTS table in my datasette project?

One interesting way to approach this could be to do it entirely in JavaScript. I've had a lot of success building small apps on top of Datasette's JavaScript API - I wrote up one example here: https://24ways.org/2018/fast-autocomplete-search-for-your-website/

Once #272 is done I'll be adding a plugin hook that allows plugins to define entirely new pages within the Datasette application, which may also be a good way to work on this.

I'd like to start working on this. I've made a custom template for index.html that contains a form that contains a search input. But I'm not sure where to go from here. When user enters a search term, I'd like for that term to go into a function I'll make that will search all tables with full text search enabled.

Can I make additional custom Python scripts for this or must I edit datasette's files directly?

Thinking about this more, I'd probably have to make a template page to go along with this, right? I'm guessing there's no way to add an all-databases-all-tables search to datasette's "home page" except by copying the "home page" template and editing it?

It does not, but that's a really great idea for a feature.

One challenge here is that FTS ranking calculations take overall table statistics into account, which means it's usually not possible to combine rankings from different tables in a sensible way. But that doesn't mean it's not possible to return grouped results.

I think this makes a lot of sense as a plugin.