html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122490,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122490,IC_kwDOCGYnMM5kUmk6,22429695,2023-08-17T23:45:44Z,2023-08-18T00:46:07Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`92.85%`** and project coverage change: **`-0.07%`** :warning: > Comparison is base [(`1dc6b5a`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/1dc6b5aa644a92d3654f7068110ed7930989ce71?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.82% compared to head [(`2915050`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.76%. > Report is 1 commits behind head on main.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #584 +/- ## ========================================== - Coverage 95.82% 95.76% -0.07% ========================================== Files 8 8 Lines 2829 2834 +5 ========================================== + Hits 2711 2714 +3 - Misses 118 120 +2 ``` | [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <92.85%> (-0.15%)` | :arrow_down: |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683164661,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683164661,IC_kwDOCGYnMM5kUw31,9599,2023-08-18T00:45:53Z,2023-08-18T00:45:53Z,OWNER,"More updated documentation: - https://sqlite-utils--584.org.readthedocs.build/en/584/reference.html#sqlite_utils.db.Table.transform - https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#python-api-transform-add-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145819,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145819,IC_kwDOCGYnMM5kUsRb,9599,2023-08-18T00:17:26Z,2023-08-18T00:17:26Z,OWNER,Updated documentation: https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#adding-foreign-key-constraints,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145110,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145110,IC_kwDOCGYnMM5kUsGW,9599,2023-08-18T00:16:28Z,2023-08-18T00:16:48Z,OWNER,"One last piece of documentation: need to document the new option to `table.transform()` and `table.transform_sql()`: https://github.com/simonw/sqlite-utils/blob/0771ac61fe5c2aca74075b20b1a99b9bd4c65661/sqlite_utils/db.py#L1706-L1708 I should write tests for them too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683143723,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683143723,IC_kwDOCGYnMM5kUrwr,9599,2023-08-18T00:14:52Z,2023-08-18T00:14:52Z,OWNER,"Another docs update: this bit in here https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-multiple-foreign-key-constraints-at-once talks about how `.add_foreign_keys()` is a performance optimization to avoid having to run VACUUM a bunch of separate times: > The final step in adding a new foreign key to a SQLite database is to run `VACUUM`, to ensure the new foreign key is available in future introspection queries. > > `VACUUM` against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using `table.add_foreign_key(...)` these can quickly add up. > > Instead, you can use `db.add_foreign_keys(...)` to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a `table`, `column`, `other_table` and `other_column`. That doesn't apply any more - the new mechanism using `.transform()` works completely differently, so this issue around running VACUUM no longer applies.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683139304,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683139304,IC_kwDOCGYnMM5kUqro,9599,2023-08-18T00:09:56Z,2023-08-18T00:09:56Z,OWNER,"Upgrading `flake8` locally replicated the error: ``` pip install -U flake8 flake8 ``` ``` ./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96 ./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683138953,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683138953,IC_kwDOCGYnMM5kUqmJ,9599,2023-08-18T00:09:20Z,2023-08-18T00:09:20Z,OWNER,"Weird, I'm getting a `flake8` problem in CI which doesn't occur on my laptop: ``` ./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96 ./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683137259,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683137259,IC_kwDOCGYnMM5kUqLr,9599,2023-08-18T00:06:59Z,2023-08-18T00:06:59Z,OWNER,"The docs still describe the old trick, I need to update that: https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122767,IC_kwDOCGYnMM5kUmpP,9599,2023-08-17T23:46:09Z,2023-08-17T23:46:09Z,OWNER,"Oops, `mypy` failures: ``` sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type ""Tuple[Any, ...]"", variable has type ""Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]"") [assignment] sqlite_utils/db.py:1164: error: Need type annotation for ""by_table"" (hint: ""by_table: Dict[, ] = ..."") [var-annotated] sqlite_utils/db.py:1169: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""transform"" [union-attr] sqlite_utils/db.py:1813: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type] sqlite_utils/db.py:1824: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type] Found 5 errors in 1 file (checked 56 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683118376,IC_kwDOCGYnMM5kUlko,9599,2023-08-17T23:41:10Z,2023-08-17T23:41:10Z,OWNER,"The problem here is that the table created by this line: ```python fresh_db.create_table(""breeds"", {""name"": str}) ``` Has this schema: ```sql CREATE TABLE [breeds] ( [name] TEXT ); ``` SQLite creates an invisible `rowid` column for it automatically. On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added: ```sql (Pdb) print(fresh_db.schema) CREATE TABLE [dogs] ( [name] TEXT , [breed_id] INTEGER, FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid]) ); CREATE TABLE [breeds] ( [name] TEXT ); ``` But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884 Here's what the debugger reveals about this code: ```python for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): raise AlterError( ""No such column: {}.{}"".format(fk.other_table, fk.other_column) ) ``` ``` (Pdb) fk ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid') (Pdb) self[fk.other_table].columns [Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683114719,IC_kwDOCGYnMM5kUkrf,9599,2023-08-17T23:36:02Z,2023-08-17T23:36:02Z,OWNER,"Just these three lines recreate the problem: ```python from sqlite_utils import Database fresh_db = Database(memory=True) fresh_db.create_table(""dogs"", {""name"": str}) fresh_db.create_table(""breeds"", {""name"": str}) fresh_db[""dogs""].add_column(""breed_id"", fk=""breeds"") ``` Traceback: ``` Traceback (most recent call last): File """", line 1, in File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2170, in add_column self.add_foreign_key(col_name, fk, fk_col) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2273, in add_foreign_key self.db.add_foreign_keys([(self.name, column, other_table, other_column)]) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1169, in add_foreign_keys self[table].transform(add_foreign_keys=fks) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1728, in transform sqls = self.transform_sql( ^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1896, in transform_sql self.db.create_table_sql( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 882, in create_table_sql raise AlterError( sqlite_utils.db.AlterError: No such column: breeds.rowid ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112857,IC_kwDOCGYnMM5kUkOZ,9599,2023-08-17T23:33:58Z,2023-08-17T23:33:58Z,OWNER,"Full test: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112298,IC_kwDOCGYnMM5kUkFq,9599,2023-08-17T23:33:14Z,2023-08-17T23:33:14Z,OWNER,"Just one failing test left: ``` # Soundness check foreign_keys point to existing tables for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): > raise AlterError( ""No such column: {}.{}"".format(fk.other_table, fk.other_column) ) E sqlite_utils.db.AlterError: No such column: breeds.rowid sqlite_utils/db.py:882: AlterError ==== short test summary info ==== FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid ==== 1 failed, 378 deselected in 0.49s ==== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,