issues

Table actions

2 rows where repo = 140912432 and user = 36287 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date), closed_at (date)

state 2

type 1

repo 1

  • sqlite-utils · 2 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
907795562 MDU6SXNzdWU5MDc3OTU1NjI= 265 Using enable_fts before search term prabhur 36287 open 0     1 2021-06-01T01:43:34Z 2023-04-01T17:27:18Z   NONE  

Many thanks for the sqlite-utils suite of utilities. Has made my life much much easier. I used this to create a table and enable FTS. All works fine. The datasette utility detects FTS and shows a text box. Searching for a term using that interface works well.

However, when I start to use features by following https://www.sqlite.org/fts5.html section "3. Full-text Query Syntax" I seem to run into issues that I suspect is due to escape_fts wrapper function.

As an example, if i search for the term "^குகை"on the text box in datasette it produces 140 results. However, when i tweak the query produced by datasette to not use "escape_fts" it produces 5 results.

Similarly, when I try to restrict the search to a single column in FTS using a spec like {title : ^குகை} it returns no rows. The same thing pulls results when used without escape_fts. The text in the table is in Tamil language and the search term is a Tamil word.

... where posts_fts match escape_fts(:search) vs

... where posts_fts match (:search)

Any ideas why? How can I get the benefits of both escaping as well as utilizing different facets of providing / controlling search terms? Thanks.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/265/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
836963850 MDU6SXNzdWU4MzY5NjM4NTA= 249 Full text search possibly broken? prabhur 36287 closed 0     2 2021-03-21T02:03:44Z 2021-03-21T02:43:32Z 2021-03-21T02:43:32Z NONE  

I'm not quite sure if this is an issue with sqlite-utils or datasette.

Background I was previously using sqlite-utils version < 3.6. I have a bunch of csv files that have some data scraped from a website.

sqlite-utils create-table mydb.db post \ posted_date text \ url text \ title text \ raw_text text \ --not-null posted_date \ --not-null url \ --pk=url FTS is enabled via sqlite-utils enable-fts ./mydb.db post title raw_text

Data is loaded to the table via sqlite-utils insert ./mydb.db post ${filename} --csv

Note that the data contains text in my language Tamil.

Loading happens just fine. datasette serves the db file just fine. It recognizes FTS and shows the "search" box. However, none of the queries work. Whatever text I supply, it always returns 0 rows. I literally copy paste words from the row listing on the screen and paste it on the search box.

Interestingly, only thing I can remember is switching to sqlite-utils 3.6. I had to do this because the prior version had an issue with column size.

I have attached one of the csv files that can be loaded to the table. Substitute "${filename}" with that file for the sqlite-utils insert command. posts_20200417-20201231.csv.zip

Interestingly, the FTS based search from datasette worked just fine before this version upgrade. That is, the queries returned results. I will try to downgrade just to see if the theory is correct.

I appreciate any help here. Thanks.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/249/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);