home / github

Menu
  • Search all tables
  • GraphQL API

issues

Table actions
  • GraphQL API for issues

110 rows where repo = 140912432 and type = "pull" sorted by updated_at descending

✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: milestone, comments, author_association, draft, created_at (date), updated_at (date), closed_at (date)

state 2

  • closed 106
  • open 4

type 1

  • pull · 110 ✖

repo 1

  • sqlite-utils · 110 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
2001006157 PR_kwDOCGYnMM5f2OZC 604 Add more STRICT table support tkhattra 16437338 closed 0     4 2023-11-19T19:38:53Z 2023-12-08T05:17:20Z 2023-12-08T05:05:27Z CONTRIBUTOR simonw/sqlite-utils/pulls/604
  • https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776

Make table.transform() preserve STRICT mode.


:books: Documentation preview :books:: https://sqlite-utils--604.org.readthedocs.build/en/604/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/604/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1884335789 PR_kwDOCGYnMM5Zs0KB 591 Test against Python 3.12 preview simonw 9599 closed 0     3 2023-09-06T16:10:00Z 2023-11-04T00:58:03Z 2023-11-04T00:58:02Z OWNER simonw/sqlite-utils/pulls/591

https://dev.to/hugovk/help-test-python-312-beta-1508/


:books: Documentation preview :books:: https://sqlite-utils--591.org.readthedocs.build/en/591/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/591/reactions",
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 1,
    "eyes": 0
}
0  
1919296686 PR_kwDOCGYnMM5bifPC 596 Fixes mapping for time fields related to mysql, closes #522 nezhar 4420927 closed 0     1 2023-09-29T13:41:48Z 2023-11-04T00:49:50Z 2023-11-04T00:49:50Z CONTRIBUTOR simonw/sqlite-utils/pulls/596

Adds COLUMN_TYPE_MAPPING for TIME fields that are mapped as datetime.timedelta for MySQL and json represantation for datetime.timedelta in order to fix #522


:books: Documentation preview :books:: https://sqlite-utils--596.org.readthedocs.build/en/596/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/596/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1926729132 PR_kwDOCGYnMM5b7Z_y 598 Fixed issue #433 - CLI eats cursor spookylukey 62745 closed 0     2 2023-10-04T18:06:58Z 2023-11-04T00:46:55Z 2023-11-04T00:40:30Z CONTRIBUTOR simonw/sqlite-utils/pulls/598

The issue is that underlying iterator is not fully consumed within the body of the with file_progress() block. Instead, that block creates generator expressions like docs = (dict(zip(headers, row)) for row in reader)

These iterables are consumed later, outside the with file_progress() block, which consumes the underlying iterator, and in turn updates the progress bar.

This means that the ProgressBar.__exit__ method gets called before the last time the ProgressBar.update method gets called. The result is that the code to make the cursor invisible (inside the update() method) is called after the cleanup code to make it visible (in the __exit__ method).

The fix is to move consumption of the docs iterators within the progress bar block. (

(An additional fix, to make ProgressBar more robust against this kind of misuse, would to make it refusing to update after its __exit__ method had been called, just like files cannot be read() after they are closed. That requires a in the click library).

Note that Github diff obscures the simplicity of this diff, it's just indenting a block of code.


:books: Documentation preview :books:: https://sqlite-utils--598.org.readthedocs.build/en/598/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/598/reactions",
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 1,
    "eyes": 0
}
0  
1977004379 PR_kwDOCGYnMM5elFZf 600 Add spatialite arm64 linux path MikeCoats 37802088 closed 0     5 2023-11-03T22:23:26Z 2023-11-04T00:34:33Z 2023-11-04T00:31:49Z CONTRIBUTOR simonw/sqlite-utils/pulls/600

According to both Debian and Ubuntu, the correct “target triple” for arm64 is aarch64-linux-gnu, so we should be looking in /usr/lib/aarch64-linux-gnu for mod_spatialite.so.

I can confirm that on both of my Debian arm64 SBCs, libsqlite3-mod-spatialite installs to that path.

$ ls -l /usr/lib/*/*spatial* lrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so -> mod_spatialite.so.7.1.0 lrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7 -> mod_spatialite.so.7.1.0 -rw-r--r-- 1 root root 7348584 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7.1.0

This is a set of before and after snippets of pytest’s output for this PR.

Before

$ pytest tests/test_get.py ...... [ 73%] tests/test_gis.py ssssssssssss [ 75%] tests/test_hypothesis.py .... [ 75%]

After

$ pytest tests/test_get.py ...... [ 73%] tests/test_gis.py ............ [ 75%] tests/test_hypothesis.py .... [ 75%]

Issue: #599


:books: Documentation preview :books:: https://sqlite-utils--600.org.readthedocs.build/en/600/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/600/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1886783150 PR_kwDOCGYnMM5Z1H1d 593 .transform() now preserves rowid values, refs #592 simonw 9599 closed 0     1 2023-09-08T01:02:28Z 2023-09-10T17:44:59Z 2023-09-09T00:45:30Z OWNER simonw/sqlite-utils/pulls/593

Refs: - #592

  • [x] Tests against weird shaped tables

I need to test that this works against:

  • rowid tables
  • Tables that have a column called rowid even though they are not rowid tables

:books: Documentation preview :books:: https://sqlite-utils--593.org.readthedocs.build/en/593/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/593/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1855838223 PR_kwDOCGYnMM5YM-I3 584 .transform() instead of modifying sqlite_master for add_foreign_keys simonw 9599 closed 0     13 2023-08-17T23:32:45Z 2023-08-18T00:48:13Z 2023-08-18T00:48:08Z OWNER simonw/sqlite-utils/pulls/584

Refs: - #577


:books: Documentation preview :books:: https://sqlite-utils--584.org.readthedocs.build/en/584/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1816917522 PR_kwDOCGYnMM5WJ6Jm 573 feat: Implement a prepare_connection plugin hook asg017 15178711 closed 0     4 2023-07-22T22:48:44Z 2023-07-22T22:59:09Z 2023-07-22T22:59:09Z CONTRIBUTOR simonw/sqlite-utils/pulls/573

Just like the Datasette prepare_connection hook, this PR adds a similar hook for the sqlite-utils plugin system.

The sole argument is conn, since I don't believe a database or datasette argument would be relevant here.

I want to do this so I can release sqlite-utils plugins for my SQLite extensions, similar to the Datasette plugins I've release for them.

An example plugin: https://gist.github.com/asg017/d7cdf0d56e2be87efda28cebee27fa3c

```bash $ sqlite-utils install https://gist.github.com/asg017/d7cdf0d56e2be87efda28cebee27fa3c/archive/5f5ad549a40860787629c69ca120a08c32519e99.zip

$ sqlite-utils memory 'select hello("alex") as response' [{"response": "Hello, alex!"}] ``` Refs: - #574


:books: Documentation preview :books:: https://sqlite-utils--573.org.readthedocs.build/en/573/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/573/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1773458985 PR_kwDOCGYnMM5T2mMb 560 Use sqlean if available in environment simonw 9599 closed 0     10 2023-06-25T19:48:48Z 2023-06-26T08:21:00Z 2023-06-25T23:25:51Z OWNER simonw/sqlite-utils/pulls/560

Refs: - #559


:books: Documentation preview :books:: https://sqlite-utils--560.org.readthedocs.build/en/560/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1718635018 PR_kwDOCGYnMM5Q9lY4 553 Reformatted CLI examples in docs simonw 9599 closed 0     2 2023-05-21T20:44:34Z 2023-05-21T20:57:27Z 2023-05-21T20:57:23Z OWNER simonw/sqlite-utils/pulls/553

Refs: - #551


:books: Documentation preview :books:: https://sqlite-utils--553.org.readthedocs.build/en/553/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/553/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1718586377 PR_kwDOCGYnMM5Q9cAv 549 TUI powered by Trogon simonw 9599 closed 0     3 2023-05-21T17:55:42Z 2023-05-21T18:42:00Z 2023-05-21T18:41:56Z OWNER simonw/sqlite-utils/pulls/549

Refs: - #545


:books: Documentation preview :books:: https://sqlite-utils--549.org.readthedocs.build/en/549/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/549/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1718550688 PR_kwDOCGYnMM5Q9VH0 546 Analyze tables options: --common-limit, --no-most, --no-least simonw 9599 closed 0     2 2023-05-21T15:54:39Z 2023-05-21T16:19:30Z 2023-05-21T16:19:30Z OWNER simonw/sqlite-utils/pulls/546

Refs #544

  • [x] Documentation for CLI options
  • [x] Documentation for new Python API parameters: most_common: bool and least_common: bool
  • [x] Tests for CLI
  • [x] Tests for Python API
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/546/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1665200812 PR_kwDOCGYnMM5OKveS 537 Support self-referencing FKs in `Table.create` numist 544011 closed 0     3 2023-04-12T20:26:59Z 2023-05-08T22:45:33Z 2023-05-08T21:10:01Z CONTRIBUTOR simonw/sqlite-utils/pulls/537

:books: Documentation preview :books:: https://sqlite-utils--537.org.readthedocs.build/en/537/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/537/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1576990618 PR_kwDOCGYnMM5JkkED 526 Fix repeated calls to `Table.convert()` mcarpenter 167893 closed 0     0 2023-02-09T00:14:49Z 2023-05-08T21:56:05Z 2023-05-08T21:53:58Z CONTRIBUTOR simonw/sqlite-utils/pulls/526

Fixes #525. All tests pass.

There's perhaps a better way to name lambdas? There could be a collision if a caller passes a function with name like lambda_123456.

SQLite documentation is a little, ah, lite on function name specs. If there is a character that can be used in place of underscore in a SQLite function name that is not permitted in a Python function identifier then that could be a good way to prevent accidental collisions. (I tried dash, colon, dot, no joy).

Otherwise, there is little chance of this happening and if it should happen the risk is mitigated by now throwing an exception in the case of a (name, arity) collision without replace=True.


:books: Documentation preview :books:: https://sqlite-utils--526.org.readthedocs.build/en/526/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/526/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1465194930 PR_kwDOCGYnMM5DvZxa 515 upsert new rows with constraints, fixes #514 cldellow 193185 closed 0     1 2022-11-26T16:15:21Z 2023-05-08T21:27:11Z 2023-05-08T21:27:10Z NONE simonw/sqlite-utils/pulls/515

This fixes #514 by making the initial insert for upserts include all columns, so that new rows can be added to tables with non-pkey columns that have constraints.

(aside: I'm not a python programmer. pip? pipenv? venv? These are mystical incantations to me. The process to set up this repo for local development and testing was so easy. Thank you for the excellent contributing documentation!)


:books: Documentation preview :books:: https://sqlite-utils--515.org.readthedocs.build/en/515/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/515/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1505568103 PR_kwDOCGYnMM5F609a 519 Fixes breaking DEFAULT values rhoboro 13819005 closed 0     1 2022-12-21T01:27:52Z 2023-05-08T21:13:37Z 2023-05-08T21:13:37Z CONTRIBUTOR simonw/sqlite-utils/pulls/519

Fixes #509, Fixes #336

Thanks for the great library! I fixed a bug that sqlite-utils transform breaks DEFAULT values. All tests already present passed with no changes, and I added some tests for this PR.

In #509 case, fixed here.

```shell $ sqlite3 test.db << EOF CREATE TABLE mytable ( col1 TEXT DEFAULT 'foo', col2 TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) ) EOF

$ sqlite3 test.db "SELECT sql FROM sqlite_master WHERE name = 'mytable';" CREATE TABLE mytable ( col1 TEXT DEFAULT 'foo', col2 TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) )

$ sqlite3 test.db "INSERT INTO mytable DEFAULT VALUES; SELECT * FROM mytable;" foo|2022-12-21 01:15:39.669

$ sqlite-utils transform test.db mytable --rename col1 renamedcol1 $ sqlite3 test.db "SELECT sql FROM sqlite_master WHERE name = 'mytable';" CREATE TABLE "mytable" ( [renamedcol1] TEXT DEFAULT 'foo', [col2] TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) # ← Non-String Value )

$ sqlite3 test.db "INSERT INTO mytable DEFAULT VALUES; SELECT * FROM mytable;" foo|2022-12-21 01:15:39.669 foo|2022-12-21 01:15:56.432 ```

And #336 case also fixed. Special values are described here.

3.2. The DEFAULT clause ... A default value may also be one of the special case-independent keywords CURRENT_TIME, CURRENT_DATE or CURRENT_TIMESTAMP.

```shell $ echo 'create table bar (baz text, created_at timestamp default CURRENT_TIMESTAMP)' | sqlite3 foo.db $ sqlite3 foo.db SQLite version 3.39.5 2022-10-14 20:58:05 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE bar (baz text, created_at timestamp default CURRENT_TIMESTAMP); sqlite> .exit

$ sqlite-utils transform foo.db bar --column-order baz $ sqlite3 foo.db SQLite version 3.39.5 2022-10-14 20:58:05 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE IF NOT EXISTS "bar" ( [baz] TEXT, [created_at] FLOAT DEFAULT CURRENT_TIMESTAMP ); sqlite> .exit

$ sqlite-utils transform foo.db bar --column-order baz $ sqlite3 foo.db SQLite version 3.39.5 2022-10-14 20:58:05 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE IF NOT EXISTS "bar" ( [baz] TEXT, [created_at] FLOAT DEFAULT CURRENT_TIMESTAMP # ← Non-String Value ); ```


:books: Documentation preview :books:: https://sqlite-utils--519.org.readthedocs.build/en/519/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/519/reactions",
    "total_count": 3,
    "+1": 3,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1578793661 PR_kwDOCGYnMM5Jqn1u 528 Enable `Table.convert()` on falsey values mcarpenter 167893 closed 0     1 2023-02-10T00:04:09Z 2023-05-08T21:08:23Z 2023-05-08T21:08:23Z CONTRIBUTOR simonw/sqlite-utils/pulls/528

Fixes #527


:books: Documentation preview :books:: https://sqlite-utils--528.org.readthedocs.build/en/528/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/528/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1659525418 PR_kwDOCGYnMM5N35VZ 536 Add paths for homebrew on Apple silicon eyeseast 25778 closed 0     1 2023-04-08T13:34:21Z 2023-04-13T01:44:43Z 2023-04-13T01:44:43Z CONTRIBUTOR simonw/sqlite-utils/pulls/536

Does what it says and nothing else. This is the same set of paths as Datasette uses.


:books: Documentation preview :books:: https://sqlite-utils--536.org.readthedocs.build/en/536/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/536/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1620164673 PR_kwDOCGYnMM5L08O8 531 Add paths for homebrew on Apple silicon eyeseast 25778 closed 0     4 2023-03-11T22:27:52Z 2023-04-09T01:49:44Z 2023-04-09T01:49:43Z CONTRIBUTOR simonw/sqlite-utils/pulls/531

This also passes in the extension path when specified in GIS methods. Wherever we know an extension path, we use db.init_spatialite(find_spatialite() or load_extension).


:books: Documentation preview :books:: https://sqlite-utils--531.org.readthedocs.build/en/531/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/531/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
743384829 MDExOlB1bGxSZXF1ZXN0NTIxMjg3OTk0 203 changes to allow for compound foreign keys drkane 1049910 open 0     7 2020-11-16T00:30:10Z 2023-01-25T18:47:18Z   FIRST_TIME_CONTRIBUTOR simonw/sqlite-utils/pulls/203

Add support for compound foreign keys, as per issue #117

Not sure if this is the right approach. In particular I'm unsure about:

  • the new ForeignKey class, which replaces the namedtuple in order to ensure that column and other_column are forced into tuples. The class does the job, but doesn't feel very elegant.
  • I haven't rewritten guess_foreign_table to take account of multiple columns, so it just checks for the first column in the foreign key definition. This isn't ideal.
  • I haven't added any ability to the CLI to add compound foreign keys, it's only in the python API at the moment.

The PR also contains a minor related change that columns and tables are always quoted in foreign key definitions.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/203/reactions",
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1000275035 PR_kwDOCGYnMM4r7n-9 327 Extract expand: Support JSON Arrays phaer 101753 closed 0     0 2021-09-19T10:34:30Z 2022-12-29T09:05:36Z 2022-12-29T09:05:36Z NONE simonw/sqlite-utils/pulls/327

Hi,

I needed to extract data in JSON Arrays to normalize data imports. I've quickly hacked the following together based on #241 which refers to #239 where you, @simonw, wrote:

Could this handle lists of objects too? That would be pretty amazing - if the column has a [{...}, {...}] list in it could turn that into a many-to-many.

They way this works in my work is that many-to-many relationships are created for anything that maps to an dictionary in a list, and many-to-one relations for everything else (assumed to be scalar values). Not sure what the best approach here would be? Are many-to-one relationships are at all useful here?

What do you think about this approach? I could try to add it to the cli interface and documentation if wanted.

Thanks for this awesome piece of software in any case! :sun_with_face:

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/327/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1430563092 PR_kwDOCGYnMM5B6_6K 508 Allow surrogates in parameters chapmanjacobd 7908073 closed 0     2 2022-10-31T22:11:49Z 2022-11-17T15:11:16Z 2022-10-31T22:55:36Z CONTRIBUTOR simonw/sqlite-utils/pulls/508

closes #507

https://dwheeler.com/essays/fixing-unix-linux-filenames.html


:books: Documentation preview :books:: https://sqlite-utils--508.org.readthedocs.build/en/508/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/508/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1405196044 PR_kwDOCGYnMM5AmYzy 499 feat: recreate fts triggers after table transform chapmanjacobd 7908073 open 0     2 2022-10-11T20:35:39Z 2022-10-26T17:54:51Z   CONTRIBUTOR simonw/sqlite-utils/pulls/499

https://github.com/simonw/sqlite-utils/pull/498


:books: Documentation preview :books:: https://sqlite-utils--499.org.readthedocs.build/en/499/

alternatively, self.disable_fts()

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/499/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1404013495 PR_kwDOCGYnMM5AicIh 498 fix: enable-fts permanently save triggers chapmanjacobd 7908073 closed 0     2 2022-10-11T05:10:51Z 2022-10-15T04:33:08Z 2022-10-11T06:34:31Z CONTRIBUTOR simonw/sqlite-utils/pulls/498

I was wondering why my all my databases were giving wild search results. Turns out create_trigger was not sticking!

Running sqlite-utils triggers x.db shows [] after running enable-fts using the python api. Looking at the counts trigger it seems that is the right way to save triggers. triggers show up now


:books: Documentation preview :books:: https://sqlite-utils--498.org.readthedocs.build/en/498/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/498/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1366512990 PR_kwDOCGYnMM4-nBs9 486 progressbar for inserts/upserts of all fileformats, closes #485 MischaU8 99098079 closed 0     7 2022-09-08T14:58:02Z 2022-09-15T20:40:03Z 2022-09-15T20:37:51Z CONTRIBUTOR simonw/sqlite-utils/pulls/486

:books: Documentation preview :books:: https://sqlite-utils--486.org.readthedocs.build/en/486/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1355433619 PR_kwDOCGYnMM4-B7Mc 480 search_sql add include_rank option chapmanjacobd 7908073 closed 0     4 2022-08-30T09:10:29Z 2022-08-31T03:40:35Z 2022-08-31T03:40:35Z CONTRIBUTOR simonw/sqlite-utils/pulls/480

I haven't tested this yet but wanted to get a heads-up whether this kind of change would be useful or if I should just duplicate the function and tweak it within my code


:books: Documentation preview :books:: https://sqlite-utils--480.org.readthedocs.build/en/480/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/480/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1348294436 PR_kwDOCGYnMM49qP2V 468 db[table].create(..., transform=True) and create-table --transform simonw 9599 closed 0   3.29 8355157 6 2022-08-23T17:27:58Z 2022-08-27T23:17:55Z 2022-08-27T23:17:55Z OWNER simonw/sqlite-utils/pulls/468

Work in progress. Still needs documentation and tests (and to cover more cases of things that might have changed).

Refs: - #467


:books: Documentation preview :books:: https://sqlite-utils--468.org.readthedocs.build/en/468/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1309542173 PR_kwDOCGYnMM47pwAb 455 in extract code, check equality with IS instead of = for nulls fgregg 536941 closed 0     3 2022-07-19T13:40:25Z 2022-08-27T14:45:03Z 2022-08-27T14:45:03Z CONTRIBUTOR simonw/sqlite-utils/pulls/455

sqlite "IS" is equivalent to SQL "IS NOT DISTINCT FROM"

closes #423

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/455/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1352953535 PR_kwDOCGYnMM4950Az 473 Support entrypoints for `--load-extension` simonw 9599 closed 0     1 2022-08-27T05:53:59Z 2022-08-27T05:55:52Z 2022-08-27T05:55:47Z OWNER simonw/sqlite-utils/pulls/473

Refs #470


:books: Documentation preview :books:: https://sqlite-utils--473.org.readthedocs.build/en/473/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/473/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1319881016 PR_kwDOCGYnMM48Mmde 457 Link to installation instructions simonw 9599 closed 0   3.29 8355157 2 2022-07-27T17:38:36Z 2022-08-27T03:55:52Z 2022-07-27T17:57:50Z OWNER simonw/sqlite-utils/pulls/457

Also testing https://docs.readthedocs.io/en/stable/pull-requests.html

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/457/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1326087800 PR_kwDOCGYnMM48hI-_ 460 Cross-link CLI to Python docs simonw 9599 closed 0     4 2022-08-02T16:18:28Z 2022-08-18T21:58:10Z 2022-08-18T21:58:07Z OWNER simonw/sqlite-utils/pulls/460

Work in progress, partly to test the ReadTheDocs preview link action.

Refs: - #426


:books: Documentation preview :books:: https://readthedocs-preview--460.org.readthedocs.build/en/460/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/460/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1334416486 PR_kwDOCGYnMM488n6D 463 Use Read the Docs action v1 humitos 244656 closed 0     1 2022-08-10T10:31:47Z 2022-08-18T08:30:14Z 2022-08-17T23:11:16Z CONTRIBUTOR simonw/sqlite-utils/pulls/463

Read the Docs repository was renamed from readthedocs/readthedocs-preview to readthedocs/actions/. Now, the preview action is under readthedocs/actions/preview and is tagged as v1


:books: Documentation preview :books:: https://sqlite-utils--463.org.readthedocs.build/en/463/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/463/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1342357149 PR_kwDOCGYnMM49Wsnq 465 beanbag-docutils>=2.0 simonw 9599 closed 0     2 2022-08-17T22:41:39Z 2022-08-17T23:38:07Z 2022-08-17T23:38:02Z OWNER simonw/sqlite-utils/pulls/465

Refs #464

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/465/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1342374388 PR_kwDOCGYnMM49Wv9T 466 Use Read the Docs action v1 (#463) simonw 9599 closed 0     0 2022-08-17T23:11:50Z 2022-08-17T23:11:54Z 2022-08-17T23:11:54Z OWNER simonw/sqlite-utils/pulls/466

Read the Docs repository was renamed from readthedocs/readthedocs-preview to readthedocs/actions/. Now, the preview action is under readthedocs/actions/preview and is tagged as v1

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/466/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1326391841 PR_kwDOCGYnMM48iLGF 462 Discord badge simonw 9599 closed 0     2 2022-08-02T20:56:04Z 2022-08-02T21:15:57Z 2022-08-02T21:15:52Z OWNER simonw/sqlite-utils/pulls/462

Also testing fix for: - https://github.com/readthedocs/readthedocs-preview/issues/10


:books: Documentation preview :books:: https://sqlite-utils--462.org.readthedocs.build/en/462/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/462/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1299760627 PR_kwDOCGYnMM47JUun 452 Add duplicate table feature davidleejy 1690072 closed 0     1 2022-07-09T20:24:31Z 2022-07-15T21:21:37Z 2022-07-15T21:21:36Z CONTRIBUTOR simonw/sqlite-utils/pulls/452

This PR addresses a feature request raised in issue #449. Specifically this PR adds a functionality that lets users duplicate a table via:

python table_new = db["my_table"].duplicate("new_table")

Test added in file tests/test_duplicate.py.

Happy to make changes to meet maintainers' feedback, if any.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/452/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1244294227 PR_kwDOCGYnMM44P4GG 437 docs to dogs yurivish 114388 closed 0     1 2022-05-22T15:50:33Z 2022-05-30T21:32:41Z 2022-05-30T21:32:41Z CONTRIBUTOR simonw/sqlite-utils/pulls/437

Fixes a typo.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/437/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1223177069 PR_kwDOCGYnMM43LrKB 429 Depend on click-default-group-wheel simonw 9599 closed 0     2 2022-05-02T18:03:10Z 2022-05-02T18:52:42Z 2022-05-02T18:05:00Z OWNER simonw/sqlite-utils/pulls/429

Trying to get this to work with Pyodide.

Refs: https://github.com/simonw/click-default-group-wheel/issues/3

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/429/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1178484369 PR_kwDOCGYnMM405rPe 419 Ignore common generated files eyeseast 25778 closed 0     1 2022-03-23T18:06:22Z 2022-03-24T21:01:44Z 2022-03-24T21:01:44Z CONTRIBUTOR simonw/sqlite-utils/pulls/419

Closes #418

This adds four files to .gitignore:

.hypothesis/
Pipfile
Pipfile.lock
pyproject.toml

Those are all generated in the course of development and testing.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/419/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1149729902 PR_kwDOCGYnMM4zbaJy 410 Correct spelling mistakes (found with codespell) EdwardBetts 3818 closed 0     1 2022-02-24T20:44:18Z 2022-03-06T08:48:29Z 2022-03-01T21:05:29Z CONTRIBUTOR simonw/sqlite-utils/pulls/410   sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/410/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1138948786 PR_kwDOCGYnMM4y3yW0 407 Add SpatiaLite helpers to CLI eyeseast 25778 closed 0     7 2022-02-15T16:50:17Z 2022-02-16T01:49:40Z 2022-02-16T00:58:08Z CONTRIBUTOR simonw/sqlite-utils/pulls/407

Closes #398

This adds SpatiaLite helpers to the CLI.

```sh

init spatialite when creating a database

sqlite-utils create database.db --enable-wal --init-spatialite

add geometry columns

needs a database, table, geometry column name, type, with optional SRID and not-null

this will throw an error if the table doesn't already exist

sqlite-utils add-geometry-column database.db table-name geometry --srid 4326 --not-null

spatial index an existing table/column

this will throw an error it the table and column don't exist

sqlite-utils create-spatial-index database.db table-name geometry ```

Docs and tests are included.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/407/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1102899312 PR_kwDOCGYnMM4w_p22 385 Add new spatialite helper methods eyeseast 25778 closed 0     16 2022-01-14T03:57:30Z 2022-02-05T00:04:26Z 2022-02-04T05:55:10Z CONTRIBUTOR simonw/sqlite-utils/pulls/385

Refs #79

This PR adds three new Spatialite-related methods to Database and Table:

  • Database.init_spatialite loads the Spatialite extension and initializes it
  • Table.add_geometry_column adds a geometry column
  • Table.create_spatial_index creates a spatial index

Has tests and documentation. Feedback very welcome.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/385/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1097477582 PR_kwDOCGYnMM4wtl17 377 `sqlite-utils bulk` command simonw 9599 closed 0   3.21 7558727 3 2022-01-10T05:34:24Z 2022-01-11T02:10:57Z 2022-01-11T02:10:54Z OWNER simonw/sqlite-utils/pulls/377

Refs #375

Still needs:

  • [x] Refactor @insert_upsert_options so that it doesn't duplicate @import_options
  • [x] Tests
  • [x] Documentation
  • [x] Try it against a really big file
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/377/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1097041471 PR_kwDOCGYnMM4wsVM6 367 Initial prototype of .analyze() methods simonw 9599 closed 0   3.21 7558727 2 2022-01-08T21:35:12Z 2022-01-10T19:31:08Z 2022-01-10T19:31:08Z OWNER simonw/sqlite-utils/pulls/367

Refs #366

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/367/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
1  
1094890366 PR_kwDOCGYnMM4wlm3B 361 --lines and --text and --convert and --import simonw 9599 closed 0     15 2022-01-06T01:49:44Z 2022-01-06T06:37:03Z 2022-01-06T06:24:54Z OWNER simonw/sqlite-utils/pulls/361

Refs #356

Still TODO:

  • [x] Get --lines working, with tests
  • [x] Get --text working, with tests
  • [x] Get regular JSON import working with --convert with tests
  • [x] Get --lines working with --convert with tests
  • [x] Get --text working with --convert with tests
  • [x] Get --csv and --tsv import working with --convert with tests
  • [x] Get --nl working with --convert with tests
  • [x] Documentation for all of the above
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1066603133 PR_kwDOCGYnMM4vKAzW 347 Test against pysqlite3 running SQLite 3.37 simonw 9599 open 0     9 2021-11-29T23:17:57Z 2021-12-11T01:02:19Z   OWNER simonw/sqlite-utils/pulls/347

Refs #346 and #344.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1039037439 PR_kwDOCGYnMM4t0uaI 333 Add functionality to read Parquet files. Florents-Tselai 2118708 closed 0     3 2021-10-28T23:43:19Z 2021-11-25T19:47:35Z 2021-11-25T19:47:35Z NONE simonw/sqlite-utils/pulls/333

I needed this for a project of mine, and I thought it'd be useful to have it in sqlite-utils (It's also mentioned in #248 ). The current implementation works (data is read & data types are inferred correctly. I've added a single straightforward test case, but @simonw please let me know if there are any non-obvious flags/combinations I should test too.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/333/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1025726600 PR_kwDOCGYnMM4tKxHD 330 Test against Python 3.10 simonw 9599 closed 0     1 2021-10-13T21:50:22Z 2021-11-15T02:59:29Z 2021-10-13T22:25:05Z OWNER simonw/sqlite-utils/pulls/330   sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/330/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
979612115 MDExOlB1bGxSZXF1ZXN0NzE5OTk4MjI1 322 Add dict type to be mapped as TEXT in sqllite minaeid90 2496189 closed 0     1 2021-08-25T20:54:26Z 2021-11-15T00:27:40Z 2021-11-15T00:27:40Z CONTRIBUTOR simonw/sqlite-utils/pulls/322

the library deal with Postgres type jsonb as dictionary, add dict type as a TEXT for mapping to sqlite

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/322/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
988013247 MDExOlB1bGxSZXF1ZXN0NzI3MDEyOTk2 324 Use python-dateutil package instead of dateutils meatcar 191622 closed 0     1 2021-09-03T18:31:19Z 2021-11-14T23:25:40Z 2021-11-14T23:25:40Z CONTRIBUTOR simonw/sqlite-utils/pulls/324

While working on updating sqlite-utils for NixOS/Nixpkgs, I came a cross the following:

In 5ec6686153e29ae10d4921a1ad4c841f192f20e2, a new dependency was added on dateutils (https://pypi.org/project/dateutils/).

I believe this is unintentional, and instead python-dateutil (https://pypi.org/project/python-dateutil/) was intended.

My reasoning is: - python-dateutil is imported here in recipes.py - The mypy type-python-dateutil dependency in setup.py - python-dateutil is a dependency of dateutils as seen in the output in docs/tutorial.ipynb

Seems like the trailing "s" seems to be the source of confusion 😅

I've swapped the dependencies out, hope this helps.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/324/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1046271107 PR_kwDOCGYnMM4uK5z2 337 Default values for `--attach` and `--param` options urbas 771193 closed 0     1 2021-11-05T21:57:53Z 2021-11-05T22:33:03Z 2021-11-05T22:33:02Z NONE simonw/sqlite-utils/pulls/337

It seems that click 8.x uses None as the default value for multiple=True options.

This change makes the code forward-compatible with click 8.x.

See this build failure for more info: https://hydra.nixos.org/build/156926608

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/337/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
929748885 MDExOlB1bGxSZXF1ZXN0Njc3NTU0OTI5 293 Test against Python 3.10-dev simonw 9599 closed 0     3 2021-06-25T01:40:39Z 2021-10-13T21:49:33Z 2021-10-13T21:49:33Z OWNER simonw/sqlite-utils/pulls/293
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/293/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
991237645 MDExOlB1bGxSZXF1ZXN0NzI5NzMxNDQx 326 Test against 3.10-dev simonw 9599 closed 0     2 2021-09-08T15:01:15Z 2021-10-13T21:49:28Z 2021-10-13T21:49:28Z OWNER simonw/sqlite-utils/pulls/326

This tests against the latest 3.10 RC, https://www.python.org/downloads/release/python-3100rc2/

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/326/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
978537855 MDExOlB1bGxSZXF1ZXN0NzE5MTA5NzA5 321 Ability to insert file contents as text, in addition to blob simonw 9599 closed 0     5 2021-08-24T22:37:18Z 2021-08-24T23:31:17Z 2021-08-24T23:31:13Z OWNER simonw/sqlite-utils/pulls/321

Refs #319.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/321/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
832687563 MDExOlB1bGxSZXF1ZXN0NTkzODA1ODA0 247 FTS quote functionality from datasette DeNeutoy 16001974 closed 0     2 2021-03-16T11:17:34Z 2021-08-18T18:43:12Z 2021-08-18T18:43:12Z CONTRIBUTOR simonw/sqlite-utils/pulls/247

Addresses #246 - this is a bit of a kludge because it doesn't actually validate the FTS string, just makes sure that it will not crash when executed, but I figured that building a query parser is a bit out of the scope of sqlite-utils and if you actually want to use the query language, you probably need to parse that yourself.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/247/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
965143346 MDExOlB1bGxSZXF1ZXN0NzA3NDkwNzg5 312 Add reference page to documentation using Sphinx autodoc simonw 9599 closed 0     10 2021-08-10T16:59:17Z 2021-08-10T23:09:32Z 2021-08-10T23:09:28Z OWNER simonw/sqlite-utils/pulls/312

Refs #311.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/312/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
957536983 MDExOlB1bGxSZXF1ZXN0NzAwOTQ0NjQ0 303 sqlite-utils convert command and db[table].convert(...) method simonw 9599 closed 0     1 2021-08-01T16:52:42Z 2021-08-02T04:47:42Z 2021-08-02T04:47:39Z OWNER simonw/sqlite-utils/pulls/303

Refs #251, #302.

  • [x] Get recipes working
  • [x] Document recipes
  • [x] Implement db[table].convert(...) method
  • [x] Add tests for recipes that use the new Python method
  • [x] Implement db[table].convert(..., multi=True) mechanism
  • [x] Documentation for db[table].convert(...)
  • [x] Refactor sqlite-utils convert to use the new method
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/303/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
922099793 MDExOlB1bGxSZXF1ZXN0NjcxMDE0NzUx 273 sqlite-utils memory command for directly querying CSV/JSON data simonw 9599 closed 0     8 2021-06-16T05:04:58Z 2021-06-18T15:01:17Z 2021-06-18T15:00:52Z OWNER simonw/sqlite-utils/pulls/273

Refs #272. Initial implementation only does CSV data, still needs:

  • [x] Implement --save
  • [x] Add --dump to the documentation
  • [x] Add --attach example to the documentation
  • [x] Replace :memory: in documentation
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/273/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
923612361 MDExOlB1bGxSZXF1ZXN0NjcyMzU5NjA5 277 add -h support closes #276 mcint 601708 closed 0     2 2021-06-17T08:08:26Z 2021-06-18T14:56:59Z 2021-06-18T14:56:59Z CONTRIBUTOR simonw/sqlite-utils/pulls/277

This appears to be the canonical solution.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/277/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
906355849 MDExOlB1bGxSZXF1ZXN0NjU3MzczNzI2 262 Ability to add descending order indexes simonw 9599 closed 0     0 2021-05-29T04:51:04Z 2021-05-29T05:01:42Z 2021-05-29T05:01:39Z OWNER simonw/sqlite-utils/pulls/262

Refs #260

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/262/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
857280617 MDExOlB1bGxSZXF1ZXN0NjE0NzI3MDM2 254 Fix incorrect create-table cli description robjwells 1935268 closed 0     1 2021-04-13T20:03:15Z 2021-05-19T04:43:46Z 2021-05-19T02:57:26Z CONTRIBUTOR simonw/sqlite-utils/pulls/254

The description for create-table was duplicated from create-index.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/254/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
820468864 MDExOlB1bGxSZXF1ZXN0NTgzNDA3OTg5 244 Typo in upsert example j-e-d 387669 closed 0     1 2021-03-02T23:14:14Z 2021-05-19T02:58:21Z 2021-05-19T02:58:21Z CONTRIBUTOR simonw/sqlite-utils/pulls/244

Remove extra [

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/244/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
830803173 MDExOlB1bGxSZXF1ZXN0NTkyMjg5MzI0 245 Correct some typos dbready 1076745 closed 0     1 2021-03-13T04:26:56Z 2021-05-19T02:58:04Z 2021-05-19T02:58:04Z CONTRIBUTOR simonw/sqlite-utils/pulls/245

Noticed a typo in the docs and followed that up with a spellcheck. Had to bite my tongue at some of the British spellings.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/245/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
868191959 MDExOlB1bGxSZXF1ZXN0NjIzNzU1NzIz 258 Fixing insert from JSON containing strings with non-ascii characters … dylan-wu 6586811 closed 0     1 2021-04-26T20:50:00Z 2021-05-19T02:47:44Z 2021-05-19T02:47:44Z CONTRIBUTOR simonw/sqlite-utils/pulls/258

…are escaped aps unicode for lists, tuples, dicts

Fix of #257

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/258/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
816601354 MDExOlB1bGxSZXF1ZXN0NTgwMjM1NDI3 241 Extract expand - work in progress simonw 9599 open 0     0 2021-02-25T16:36:38Z 2021-02-25T16:36:38Z   OWNER simonw/sqlite-utils/pulls/241

Refs #239. Still needs documentation and CLI implementation.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/241/reactions",
    "total_count": 3,
    "+1": 3,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
1  
797159961 MDExOlB1bGxSZXF1ZXN0NTY0MjE1MDEx 225 fix for problem in Table.insert_all on search for columns per chunk of rows nieuwenhoven 261237 closed 0     2 2021-01-29T20:16:07Z 2021-02-14T21:04:13Z 2021-02-14T21:04:13Z NONE simonw/sqlite-utils/pulls/225

Hi,

I ran into a problem when trying to create a database from my Apple Healthkit data using healthkit-to-sqlite. The program crashed because of an invalid insert statement that was generated for table rDistanceCycling.

The actual problem turned out to be in sqlite-utils. Table.insert_all processes the data to be inserted in chunks of rows and checks for every chunk which columns are used, and it will collect all column names in the variable all_columns. The collection of columns is done using a nested list comprehension that is not completely correct.

I'm using a Windows machine and had to make a few adjustments to the tests in order to be able to run them because they had a posix dependency.

Thanks, kind regards,

Frans

```

this is a (condensed) chunk of data from my Apple healthkit export that caused the problem.

the 3 last items in the chunk have additional keys: metadata_HKMetadataKeySyncVersion and metadata_HKMetadataKeySyncIdentifier

chunk = [{'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '7.0.1', 'device': '<<HKDevice: 0x281cf6c70>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:7.0.1>', 'unit': 'km', 'creationDate': '2020-10-10 12:29:09 +0100', 'startDate': '2020-10-10 12:29:06 +0100', 'endDate': '2020-10-10 12:29:07 +0100', 'value': '0.00518016'}, {'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '7.0.1', 'device': '<<HKDevice: 0x281cf6c70>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:7.0.1>', 'unit': 'km', 'creationDate': '2020-10-10 12:29:10 +0100', 'startDate': '2020-10-10 12:29:07 +0100', 'endDate': '2020-10-10 12:29:08 +0100', 'value': '0.00544049'}, {'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '6.2.6', 'device': '<<HKDevice: 0x281cf83e0>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>', 'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:40:50 +0100', 'endDate': '2020-07-15 16:42:49 +0100', 'value': '0.952092', 'metadata_HKMetadataKeySyncVersion': '1', 'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520450.99823:616520569.99360:119'}, {'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '6.2.6', 'device': '<<HKDevice: 0x281cf83e0>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>', 'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:42:49 +0100', 'endDate': '2020-07-15 16:44:51 +0100', 'value': '0.848983', 'metadata_HKMetadataKeySyncVersion': '1', 'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520569.99360:616520691.98826:119'}, {'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '6.2.6', 'device': '<<HKDevice: 0x281cf83e0>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>', 'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:44:51 +0100', 'endDate': '2020-07-15 16:46:50 +0100', 'value': '0.834403', 'metadata_HKMetadataKeySyncVersion': '1', 'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520691.98826:616520810.98305:119'}]

def all_columns_old(): all_columns = [col for col in chunk[0]] all_columns += [column for record in chunk for column in record if column not in all_columns] return all_columns

def all_columns_new(): all_columns = [col for col in chunk[0]] for record in chunk: all_columns += [column for column in record if column not in all_columns] return all_columns

if name == 'main': from pprint import pprint

print('problem: ')
pprint(all_columns_old())
print('\nfix: ')
pprint(all_columns_new())

```

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/225/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
808037010 MDExOlB1bGxSZXF1ZXN0NTczMTQ3MTY4 233 Run tests against Ubuntu, macOS and Windows simonw 9599 closed 0     0 2021-02-14T20:11:02Z 2021-02-14T20:39:54Z 2021-02-14T20:39:54Z OWNER simonw/sqlite-utils/pulls/233

Refs #232

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/233/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
792297010 MDExOlB1bGxSZXF1ZXN0NTYwMjA0MzA2 224 Add fts offset docs. polyrand 37962604 closed 0     2 2021-01-22T20:50:58Z 2021-02-14T19:31:06Z 2021-02-14T19:31:06Z NONE simonw/sqlite-utils/pulls/224

The limit can be passed as a string to the query builder to have an offset. I have tested it using the shorthand limit=f"15, 30", the standard syntax should work too.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/224/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
763320133 MDExOlB1bGxSZXF1ZXN0NTM3NzkxNjc1 208 sqlite-utils analyze-tables command and table.analyze_column() method simonw 9599 closed 0     6 2020-12-12T05:27:49Z 2020-12-13T07:20:16Z 2020-12-13T07:20:12Z OWNER simonw/sqlite-utils/pulls/208

Refs #207

  • [x] Improve design of CLI output
  • [x] Truncate long values in least/most common
  • [x] Add a -c column selection option
  • [x] Tests
  • [x] Documentation
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/208/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
752888228 MDExOlB1bGxSZXF1ZXN0NTI5MDkwNTYw 204 use jsonify_if_need for sql updates mfa 78035 closed 0     1 2020-11-29T10:49:00Z 2020-12-08T17:49:42Z 2020-12-08T17:49:42Z CONTRIBUTOR simonw/sqlite-utils/pulls/204
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/204/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
735663855 MDExOlB1bGxSZXF1ZXN0NTE1MDE0ODgz 195 table.search() improvements plus sqlite-utils search command simonw 9599 closed 0     3 2020-11-03T22:02:08Z 2020-11-06T18:30:49Z 2020-11-06T18:30:42Z OWNER simonw/sqlite-utils/pulls/195

Refs #192. Still needs tests.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/195/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
730693696 MDExOlB1bGxSZXF1ZXN0NTEwOTU2MTM0 190 Progress bar for sqlite-utils insert command simonw 9599 closed 0     0 2020-10-27T18:08:53Z 2020-10-27T18:16:03Z 2020-10-27T18:16:03Z OWNER simonw/sqlite-utils/pulls/190

Refs #173

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/190/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
729818242 MDExOlB1bGxSZXF1ZXN0NTEwMjM1OTA5 189 Allow iterables other than Lists in m2m records adamwolf 35681 closed 0     3 2020-10-26T18:47:44Z 2020-10-27T16:28:37Z 2020-10-27T16:24:21Z CONTRIBUTOR simonw/sqlite-utils/pulls/189

I was playing around with sqlite-utils, creating a Roam Research dogsheep-style importer for Datasette, and ran into a slight snag.

I wanted to use a generator to add an order column in an importer. It looked something like:

``` def order_generator(iterable, attr=None): if attr is None: attr = "order" order: int = 0

for i in iterable:
    i[attr] = order
    order += 1
    yield i

```

When I used this with insert_all and other things, it worked fine--but it didn't work as the records argument to m2m. I dug into it, and sqlite-utils is explicitly checking if the records argument is a list or a tuple. I flipped the check upside down, and now it checks if the argument is a mapping. If it's a mapping, it wraps it in a list, otherwise it leaves it alone.

(I get that it might not really make sense to put the order column on the second table. I changed my import schema a bit, and no longer have a real example, but maybe this change still makes sense.)

The automated tests still pass, but I did not add any new ones.

Let me know what you think! I'm really loving Datasette and its ecosystem; thanks for everything!

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/189/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
716955793 MDExOlB1bGxSZXF1ZXN0NDk5NjAzMzU5 184 Test against Python 3.9 simonw 9599 closed 0     0 2020-10-08T01:37:05Z 2020-10-08T01:44:06Z 2020-10-08T01:44:06Z OWNER simonw/sqlite-utils/pulls/184
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/184/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
709043182 MDExOlB1bGxSZXF1ZXN0NDkzMTYyNzY3 178 Update README.md shakeel 19921 closed 0     1 2020-09-25T15:52:11Z 2020-10-01T14:18:30Z 2020-09-30T20:29:28Z CONTRIBUTOR simonw/sqlite-utils/pulls/178

The sqlite-utils insert releases.db releases - --pk is missing the pk field name, added "id" to fix it.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/178/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
707944044 MDExOlB1bGxSZXF1ZXN0NDkyMjU3NDA1 174 Much, much faster extract() implementation simonw 9599 closed 0     7 2020-09-24T07:52:31Z 2020-09-24T15:44:00Z 2020-09-24T15:43:56Z OWNER simonw/sqlite-utils/pulls/174

Takes my test down from ten minutes to four seconds. Refs #172.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/174/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
557830332 MDExOlB1bGxSZXF1ZXN0MzY5MzQ4MDg0 78 New conversions= feature, refs #77 simonw 9599 closed 0     0 2020-01-31T00:02:33Z 2020-09-22T07:48:29Z 2020-01-31T00:24:31Z OWNER simonw/sqlite-utils/pulls/78
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/78/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
705975133 MDExOlB1bGxSZXF1ZXN0NDkwNjA3OTQ5 161 table.transform() method simonw 9599 closed 0   2.20 5897911 13 2020-09-21T23:16:59Z 2020-09-22T07:48:24Z 2020-09-22T04:20:02Z OWNER simonw/sqlite-utils/pulls/161

Refs #114

  • [x] Ability to change the primary key
  • [x] Support for changing default value for columns
  • [x] Support for changing NOT NULL status of columns
  • [x] Support for copying existing foreign keys and removing them
  • <strike>Support for conversions= parameter</strike>
  • [x] Detailed documentation
  • [x] PRAGMA foreign_keys stuff
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
706092617 MDExOlB1bGxSZXF1ZXN0NDkwNzAzMTcz 166 Keyword only arguments for transform() simonw 9599 closed 0     0 2020-09-22T05:41:44Z 2020-09-22T06:39:11Z 2020-09-22T06:39:11Z OWNER simonw/sqlite-utils/pulls/166

Refs #165

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/166/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
697203800 MDExOlB1bGxSZXF1ZXN0NDgzMTc1NTA5 158 Fix accidental mega long line in docs tomviner 167319 closed 0     1 2020-09-09T22:31:23Z 2020-09-16T06:21:43Z 2020-09-16T06:21:43Z CONTRIBUTOR simonw/sqlite-utils/pulls/158
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/158/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
697030843 MDExOlB1bGxSZXF1ZXN0NDgzMDI3NTg3 156 Typos in tests simonwiles 96218 closed 0     1 2020-09-09T18:00:58Z 2020-09-09T18:24:50Z 2020-09-09T18:21:23Z CONTRIBUTOR simonw/sqlite-utils/pulls/156

One of these is my fault, and the other is one I just happened to come across. They're harmless, but might as well be fixed.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/156/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
688668680 MDExOlB1bGxSZXF1ZXN0NDc1ODc0NDkz 146 Handle case where subsequent records (after first batch) include extra columns simonwiles 96218 closed 0     5 2020-08-30T07:13:58Z 2020-09-08T23:20:37Z 2020-09-08T23:20:37Z CONTRIBUTOR simonw/sqlite-utils/pulls/146

Addresses #145.

I think this should do the job. If it meets with your approval I'll update this PR to include an update to the documentation -- I came across this bug while preparing a PR to update the documentation around batch_size in any event.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/146/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
695360889 MDExOlB1bGxSZXF1ZXN0NDgxNjE2NzA0 151 Tracer mechanism for seeing underlying SQL simonw 9599 closed 0     0 2020-09-07T19:46:43Z 2020-09-07T21:57:00Z 2020-09-07T21:57:00Z OWNER simonw/sqlite-utils/pulls/151

Refs #150. Needs tests and documentation, including for the new db.execute() and db.executescript() methods.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/151/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
688386219 MDExOlB1bGxSZXF1ZXN0NDc1NjY1OTg0 142 insert_all(..., alter=True) should work for new columns introduced after the first 100 records simonwiles 96218 closed 0     3 2020-08-28T22:22:57Z 2020-08-30T07:28:23Z 2020-08-28T22:30:14Z CONTRIBUTOR simonw/sqlite-utils/pulls/142

Closes #139.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/142/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
651844316 MDExOlB1bGxSZXF1ZXN0NDQ1MDIzMzI2 118 Add insert --truncate option tsibley 79913 closed 0     9 2020-07-06T21:58:40Z 2020-07-08T17:26:21Z 2020-07-08T17:26:21Z CONTRIBUTOR simonw/sqlite-utils/pulls/118

Deletes all rows in the table (if it exists) before inserting new rows. SQLite doesn't implement a TRUNCATE TABLE statement but does optimize an unqualified DELETE FROM.

This can be handy if you want to refresh the entire contents of a table but a) don't have a PK (so can't use --replace), b) don't want the table to disappear (even briefly) for other connections, and c) have to handle records that used to exist being deleted.

Ideally the replacement of rows would appear instantaneous to other connections by putting the DELETE + INSERT in a transaction, but this is very difficult without breaking other code as the current transaction handling is inconsistent and non-systematic. There exists the possibility for the DELETE to succeed but the INSERT to fail, leaving an empty table. This is not much worse, however, than the current possibility of one chunked INSERT succeeding and being committed while the next chunked INSERT fails, leaving a partially complete operation.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
652816158 MDExOlB1bGxSZXF1ZXN0NDQ1ODMzOTA4 120 Fix query command's support for DML tsibley 79913 closed 0     1 2020-07-08T01:36:34Z 2020-07-08T05:14:04Z 2020-07-08T05:14:04Z CONTRIBUTOR simonw/sqlite-utils/pulls/120

See commit messages for details. I ran into this while investigating another feature/issue.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/120/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
589801352 MDExOlB1bGxSZXF1ZXN0Mzk1MjU4Njg3 96 Add type conversion for Panda's Timestamp b0b5h4rp13 32605365 closed 0     2 2020-03-29T14:13:09Z 2020-03-31T04:40:49Z 2020-03-31T04:40:48Z CONTRIBUTOR simonw/sqlite-utils/pulls/96

Add type conversion for Panda's Timestamp, if Panda library is present in system (thanks for this project, I was about to do the same thing from scratch)

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/96/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
557892819 MDExOlB1bGxSZXF1ZXN0MzY5Mzk0MDQz 80 on_create mechanism for after table creation simonw 9599 closed 0     5 2020-01-31T03:38:48Z 2020-01-31T05:08:04Z 2020-01-31T05:08:04Z OWNER simonw/sqlite-utils/pulls/80

I need this for geojson-to-sqlite, in particular https://github.com/simonw/geojson-to-sqlite/issues/6

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/80/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
546078359 MDExOlB1bGxSZXF1ZXN0MzU5ODIyNzcz 75 Explicitly include tests and docs in sdist jayvdb 15092 closed 0     1 2020-01-07T04:53:20Z 2020-01-31T00:21:27Z 2020-01-31T00:21:27Z CONTRIBUTOR simonw/sqlite-utils/pulls/75

Also exclude 'tests' from runtime installation.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/75/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
529376481 MDExOlB1bGxSZXF1ZXN0MzQ2MjY0OTI2 67 Run tests against 3.5 too simonw 9599 closed 0     2 2019-11-27T14:20:35Z 2019-12-31T01:29:44Z 2019-12-31T01:29:43Z OWNER simonw/sqlite-utils/pulls/67
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/67/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
543738004 MDExOlB1bGxSZXF1ZXN0MzU3OTkyNTg4 72 Fixed implementation of upsert simonw 9599 closed 0     0 2019-12-30T05:08:05Z 2019-12-30T05:29:24Z 2019-12-30T05:29:24Z OWNER simonw/sqlite-utils/pulls/72

Refs #66

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/72/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
519039316 MDExOlB1bGxSZXF1ZXN0MzM3ODUzMzk0 65 Release 1.12.1 simonw 9599 closed 0     0 2019-11-07T04:51:29Z 2019-11-07T04:58:48Z 2019-11-07T04:58:47Z OWNER simonw/sqlite-utils/pulls/65
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/65/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
519032008 MDExOlB1bGxSZXF1ZXN0MzM3ODQ3NTcz 64 test_insert_upsert_all_empty_list simonw 9599 closed 0     0 2019-11-07T04:24:45Z 2019-11-07T04:32:38Z 2019-11-07T04:32:38Z OWNER simonw/sqlite-utils/pulls/64
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/64/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
487987958 MDExOlB1bGxSZXF1ZXN0MzEzMTA1NjM0 57 Add triggers while enabling FTS amjith 49260 closed 0     4 2019-09-02T04:23:40Z 2019-09-03T01:03:59Z 2019-09-02T23:42:29Z CONTRIBUTOR simonw/sqlite-utils/pulls/57

This adds the option for a user to set up triggers in the database to keep their FTS table in sync with the parent table.

Ref: https://sqlite.org/fts5.html#external_content_and_contentless_tables

I would prefer to make the creation of triggers the default behavior, but that will break existing usage where people have been calling populate_fts after inserting new rows.

I am happy to make changes to the PR as you see fit.

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/57/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
488341021 MDExOlB1bGxSZXF1ZXN0MzEzMzgzMzE3 60 db.triggers and table.triggers introspection simonw 9599 closed 0     0 2019-09-03T00:04:32Z 2019-09-03T00:09:42Z 2019-09-03T00:09:42Z OWNER simonw/sqlite-utils/pulls/60

Closes #59

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/60/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
487847945 MDExOlB1bGxSZXF1ZXN0MzEzMDA3NDgz 56 Escape the table name in populate_fts and search. amjith 49260 closed 0     2 2019-09-01T06:29:05Z 2019-09-02T17:23:21Z 2019-09-02T17:23:21Z CONTRIBUTOR simonw/sqlite-utils/pulls/56

The table names weren't escaped using double quotes in the populate_fts method.

Reproducible case: ```

import sqlite_utils db = sqlite_utils.Database("abc.db") db["http://example.com"].insert_all([ ... {"id": 1, "age": 4, "name": "Cleo"}, ... {"id": 2, "age": 2, "name": "Pancakes"} ... ], pk="id")

<Table http://example.com (id, age, name)> >>> db["http://example.com"].enable_fts(["name"]) Traceback (most recent call last): File "", line 1, in <module> db["http://example.com"].enable_fts(["name"]) File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/sqlite_utils/db.py", l ine 705, in enable_fts self.populate_fts(columns) File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/sqlite_utils/db.py", l ine 715, in populate_fts self.db.conn.executescript(sql) sqlite3.OperationalError: unrecognized token: ":" >>> ```
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/56/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
481887482 MDExOlB1bGxSZXF1ZXN0MzA4MjkyNDQ3 55 Ability to introspect and run queries against views simonw 9599 closed 0     1 2019-08-17T13:40:56Z 2019-08-23T12:19:42Z 2019-08-23T12:19:42Z OWNER simonw/sqlite-utils/pulls/55

See #54

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/55/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
476436920 MDExOlB1bGxSZXF1ZXN0MzAzOTkwNjgz 53 Work in progress: m2m() method for creating many-to-many records simonw 9599 closed 0     0 2019-08-03T10:03:56Z 2019-08-04T03:38:10Z 2019-08-04T03:37:33Z OWNER simonw/sqlite-utils/pulls/53
  • [x] table.insert({"name": "Barry"}).m2m("tags", lookup={"tag": "Coworker"})
  • [x] Explicit table name .m2m("humans", ..., m2m_table="relationships")
  • [x] Automatically use an existing m2m table if a single obvious candidate exists (a table with two foreign keys in the correct directions)
  • [x] Require the explicit m2m_table= argument if multiple candidates for the m2m table exist
  • [x] Documentation

Refs #23

sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/53/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
467862459 MDExOlB1bGxSZXF1ZXN0Mjk3NDEyNDY0 38 table.update() method simonw 9599 closed 0     2 2019-07-14T17:03:49Z 2019-07-28T15:43:51Z 2019-07-28T15:43:51Z OWNER simonw/sqlite-utils/pulls/38

Refs #35

Still to do:

  • [x] Unit tests
  • [x] Switch to using .get()
  • [x] Better exceptions, plus unit tests for what happens if pk does not exist
  • [x] Documentation
  • [x] Ensure compound primary keys work properly
  • [x] alter=True support
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/38/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
473733752 MDExOlB1bGxSZXF1ZXN0MzAxODI0MDk3 51 Fix for too many SQL variables, closes #50 simonw 9599 closed 0     1 2019-07-28T11:30:30Z 2019-07-28T11:59:32Z 2019-07-28T11:59:32Z OWNER simonw/sqlite-utils/pulls/51
sqlite-utils 140912432 pull    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/51/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);
Powered by Datasette · Queries took 81.271ms · About: github-to-sqlite
  • Sort ascending
  • Sort descending
  • Facet by this
  • Hide this column
  • Show all columns
  • Show not-blank rows