issues

Table actions

2 rows where repo = 140912432 and user = 1231935 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date), closed_at (date)

state 2

type 1

repo 1

  • sqlite-utils · 2 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1720096994 I_kwDOCGYnMM5mhpji 554 `IndexError` when doing `.insert(..., pk='id')` after `insert_all` xavdid 1231935 open 0     1 2023-05-22T17:13:02Z 2023-05-22T17:18:33Z   NONE  

I believe this is related to https://github.com/simonw/sqlite-utils/issues/98.

When pk is specified by table A's insert call, it throws an index error if a different table has written a row with a higher rowid than exists in the first table. Here's a basic example:

```py from sqlite_utils import Database

def test_pk_for_insert(fresh_db): user = {"id": "abc", "name": "david"}

fresh_db["users"].insert(user, pk="id")

fresh_db["comments"].insert_all(
    [
        {"id": "def", "text": "ok"},
        {"id": "ghi", "text": "great"},
    ],
)

fresh_db["users"].insert(
    user,
    ignore=True,
    # BUG: when specifying pk on the second insert call 
    # db.py goes into a block it doesn't expect and we get the error
    pk="id",
)

if name == "main": db = Database("bug.db") if db["users"].exists(): raise ValueError( "bug only shows on a new database - remove bug.db before running the script" ) test_pk_for_insert(db) ```

The error is:

py File "/Users/david/projects/reddit-to-sqlite/.venv/lib/python3.11/site-packages/sqlite_utils/db.py", line 2960, in insert_chunk row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^ IndexError: list index out of range

The issue is in this block:

https://github.com/simonw/sqlite-utils/blob/2747257a3334d55e890b40ec58fada57ae8cfbfd/sqlite_utils/db.py#L2954-L2958

relevant locals are:

  • pk: 'id'
  • result.lastrowid: 2

What's most interesting is the comment # self.last_rowid will be 0 if a "INSERT OR IGNORE" happened, which doesn't seem to be the case here.

sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/554/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1695428235 I_kwDOCGYnMM5lDi6L 538 `table.upsert_all` fails to write rows when `not_null` is present xavdid 1231935 closed 0     9 2023-05-04T07:30:38Z 2023-05-08T20:06:35Z 2023-05-08T19:27:02Z NONE  

I found an odd bug today, where calls to table.upsert_all don't write rows if you include the not_null kwarg.

Repro Example

```py from sqlite_utils import Database

db = Database("upsert-test.db")

db["comments"].upsert_all( [{"id": 1, "name": "david"}], pk="id", not_null=["name"], )

assert list(db["comments"].rows) # err! ```

The schema is correctly created:

sql CREATE TABLE [comments] ( [id] INTEGER PRIMARY KEY, [name] TEXT NOT NULL )

But no rows are created. Removing either the not_null kwargs works as expected, as does an insert_all call.

Version Info

  • Python: 3.11.0
  • sqlite-utils: 3.30
  • sqlite: 3.39.5 2022-10-14
 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);