issue_comments

I released a plugin that implements an early version of this a while ago: https://github.com/simonw/datasette-search-all

I'm still thinking about how a plugin / separate tool could work that builds a single FTS index over the content from multiple tables such that you can run relevance-calculated queries against those multiple tables.

Since that's going to be a plugin / separate tool too, I'm closing this issue.

Hmmm... you know I think I actually need this for my https://github.com/simonw/fara-datasette project...

https://fara.datasettes.com/fara?sql=select+1+as+sortme%2C+%27FARA_All_RegistrantDocs%27+as+%27table%27%2C+rowid+as+id%2C+Document_Type+%7C%7C+%27+%27+%7C%7C+Short_Form_Name+%7C%7C+%27+for+%27+%7C%7C+Registrant_Name+as+title%0D%0Afrom+FARA_All_RegistrantDocs%0D%0A++where+rowid+in+%28select+rowid+from+FARA_All_RegistrantDocs_fts+where+FARA_All_RegistrantDocs_fts+match+%3Asearch%0D%0A++++order+by+rank+limit+10%0D%0A%29%0D%0Aunion%0D%0Aselect+2+as+sortme%2C+%27FARA_All_ForeignPrincipals%27+as+%27table%27%2C+rowid+as+id%2C+Foreign_Principal+%7C%7C+%27+for+%27+%7C%7C+%22Country%2FLocation_Represented%22+as+title%0D%0Afrom+FARA_All_ForeignPrincipals%0D%0A++where+rowid+in+%28select+rowid+from+FARA_All_ForeignPrincipals_fts+where+FARA_All_ForeignPrincipals_fts+match+%3Asearch%0D%0A++++order+by+rank+limit+10%0D%0A%29%0D%0Aunion%0D%0Aselect+3+as+sortme%2C+%27FARA_All_Registrants%27+as+%27table%27%2C+rowid+as+id%2C+Name+%7C%7C+%27+on+%27+%7C%7C+Registration_Date+%7C%7C+%27+in+%27+%7C%7C+City+as+title%0D%0Afrom+FARA_All_Registrants%0D%0A++where+rowid+in+%28select+rowid+from+FARA_All_Registrants_fts+where+FARA_All_Registrants_fts+match+%3Asearch%0D%0A++++order+by+rank+limit+10%0D%0A%29%0D%0Aunion%0D%0Aselect+4+as+sortme%2C+%27FARA_All_ShortForms%27+as+%27table%27%2C+rowid+as+id%2C+Short_Form_First_Name+%7C%7C+%27+%27+%7C%7C+Short_Form_Last_Name+%7C%7C+%27+in+%27+%7C%7C+City+as+title%0D%0Afrom+FARA_All_ShortForms%0D%0A++where+rowid+in+%28select+rowid+from+FARA_All_ShortForms_fts+where+FARA_All_ShortForms_fts+match+%3Asearch%0D%0A++++order+by+rank+limit+10%0D%0A%29%0D%0A%0D%0A&search=cohen

You can use a database query against sqlite_master to find all the FTS tables:

http://sf-trees.datasettes.com/trees-b64f0cb?sql=select+name+from+sqlite_master+where+name+like+%22%25_fts%22+and+type+%3D+%22table%22

select name from sqlite_master where name like "%_fts" and type = "table"

You could then construct a crafty UNION query to get results back from all of those tables at once:

http://sf-trees.datasettes.com/trees-b64f0cb?sql=select+%27PlantType_value_fts%27%2C+rowid%2C+value+from+PlantType_value_fts+where+PlantType_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qCareAssistant_value_fts%27%2C+rowid%2C+value+from+qCareAssistant_value_fts+where+qCareAssistant_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qCaretaker_value_fts%27%2C+rowid%2C+value+from+qCaretaker_value_fts+where+qCaretaker_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qLegalStatus_value_fts%27%2C+rowid%2C+value+from+qLegalStatus_value_fts+where+qLegalStatus_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qSiteInfo_value_fts%27%2C+rowid%2C+value+from+qSiteInfo_value_fts+where+qSiteInfo_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qSpecies_value_fts%27%2C+rowid%2C+value+from+qSpecies_value_fts+where+qSpecies_value_fts+match+%3Asearch&search=fi%2A

(I'm searching for fi* here to demonstrate wildcards)

The problem, as discussed earlier, is relevance: there's no way to compare the scores you're getting across different tables, so you won't be able to order by anything.

One interesting way to approach this could be to do it entirely in JavaScript. I've had a lot of success building small apps on top of Datasette's JavaScript API - I wrote up one example here: https://24ways.org/2018/fast-autocomplete-search-for-your-website/

Once #272 is done I'll be adding a plugin hook that allows plugins to define entirely new pages within the Datasette application, which may also be a good way to work on this.

It does not, but that's a really great idea for a feature.

One challenge here is that FTS ranking calculations take overall table statistics into account, which means it's usually not possible to combine rankings from different tables in a sensible way. But that doesn't mean it's not possible to return grouped results.

I think this makes a lot of sense as a plugin.