Bumps the python-packages group with 1 update: black.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

:books: Documentation preview :books:: https://datasette--2206.org.readthedocs.build/en/2206/

See discussion in https://github.com/simonw/datasette/issues/2208

:books: Documentation preview :books:: https://datasette--2209.org.readthedocs.build/en/2209/

Opening db files in immutable mode sometimes leads to the file being mutated, which causes duplication in the docker image layers: see #1836, #1480

That this happens in "immutable" mode is surprising, because the sqlite docs say that setting this should open the database as read only.

https://www.sqlite.org/c3ref/open.html

immutable: The immutable parameter is a boolean query parameter that indicates that the database file is stored on read-only media. When immutable is set, SQLite assumes that the database file cannot be changed, even by a process with higher privilege, and so the database is opened read-only and all locking and change detection is disabled. Caution: Setting the immutable property on a database file that does in fact change can result in incorrect query results and/or SQLITE_CORRUPT errors. See also: SQLITE_IOCAP_IMMUTABLE.

Perhaps this is a bug in sqlite?

:books: Documentation preview :books:: https://datasette--1870.org.readthedocs.build/en/1870/

Update hupper's entry point so that click commands are processed properly.

Fixes #2123

:books: Documentation preview :books:: https://datasette--2155.org.readthedocs.build/en/2155/

WIP

This PR builds on #5 to continue implementing gmail import support.

Building on @UtahDave's work, these commits add a few performance and bug fixes:

• Decreased memory overhead for import by manually parsing mbox headers.
• Fixed error where some messages in the mbox would yield a row with NULL in all columns.

I will send more commits to fix any errors I encounter as I run the importer on my personal takeout data.

refs 2157, specifically this comment

Passing in --internal my_internal.db over and over again can get repetitive.

This PR adds a new configurable env variable DATASETTE_INTERNAL_DB_PATH. If it's defined, then it takes place as the path to the internal database. Users can still overwrite this behavior by passing in their own --internal internal.db flag.

In draft mode for now, needs tests and documentation.

Side note: Maybe we can have a sections in the docs that lists all the "configuration environment variables" that Datasette respects? I did a quick grep and found:

• DATASETTE_LOAD_PLUGINS
• DATASETTE_SECRETS

:books: Documentation preview :books:: https://datasette--2174.org.readthedocs.build/en/2174/

• build: use poetry for package management, asdf for python version
• build: cleanup poetry config, add keywords, ignore dist
• ci: migrate circleci to gh actions
• fix: dup method definition

1. The code uses two different ways of writing encoding formats, mac_roman and macroman. It is uncertain whether there are any typo errors.
2. When there are Chinese characters in the content, exporting it results in garbled code. Changing it to utf8 can fix the issue.

Closes #2095.

:books: Documentation preview :books:: https://datasette--2159.org.readthedocs.build/en/2159/

This adds a brand link to the top navbar if 'brand' key is populated in metadata.json. The link will be either '#' or use the contents of 'brand_url' in metadata.json for href.

I was able to get this done on my own site by replacing templates/_crumbs.html with a custom version, but I thought it would be nice to incorporate this in the tool directly.

:books: Documentation preview :books:: https://datasette--2158.org.readthedocs.build/en/2158/

still testing, hope it works!

:books: Documentation preview :books:: https://datasette--2151.org.readthedocs.build/en/2151/

:books: Documentation preview :books:: https://datasette--2100.org.readthedocs.build/en/2100/

Refactored out the SQL for extracting aesthetic scores to use osxphotos -- adds compatbility for Big Sur via osxphotos which has been updated for new table names in Big Sur. Have not yet refactored the SQL for extracting labels which is still compatible with Big Sur.

This PR changes the display of facets to hopefully make them more readable.

Before | After ---|--- |

Gotta tell it where to look

G'day! I got confused while debugging, earlier today. That's on me, but it does strike me a little repetition in the metadata documentation might help those flicking around it rather than reading it from top to bottom. No worries if you think otherwise.

:books: Documentation preview :books:: https://datasette--2096.org.readthedocs.build/en/2096/

Just like facet's suggest() is trapping QueryInterrupted for facet columns, we also need to trap get_row_count(), which can reach timeout if database tables are big enough.

I've included get_columns() inside the block as that's just another query, despite it's a really cheap one and might never raise the exception.

:books: Documentation preview :books:: https://datasette--2082.org.readthedocs.build/en/2082/

:books: Documentation preview :books:: https://datasette--2076.org.readthedocs.build/en/2076/

serving multiple database files and getting tired by the default sort, changes so the sort order puts the latest changed databases to be on top of the list so don't have to scroll down, lazy as i am ;)

:books: Documentation preview :books:: https://datasette--2074.org.readthedocs.build/en/2074/

This PR adds a new section about packaging a plugin using poetry within the "Writing plugins" page of the documentation.

:books: Documentation preview :books:: https://datasette--2061.org.readthedocs.build/en/2061/

Ruff supports over 500 lint rules and can be used to replace Flake8 (plus dozens of plugins), isort, pydocstyle, yesqa, eradicate, pyupgrade, and autoflake, all while executing (in Rust) tens or hundreds of times faster than any individual tool.

The ruff Action uses minimal steps to run in ~5 seconds, rapidly providing intuitive GitHub Annotations to contributors.

:books: Documentation preview :books:: https://datasette--2056.org.readthedocs.build/en/2056/

this var app isn't actually used? unless init it does some side-effect outside of the event loop, idon't think it's necessary.

Feel free to ignore this PR if the deleted line actually does something.

:books: Documentation preview :books:: https://datasette--2034.org.readthedocs.build/en/2034/

I added a virtual column (no storage overhead) to the output that easily links back to the source. It works nicely out of the box with datasette:

I got bit a bit by https://github.com/simonw/sqlite-utils/issues/411, so I went with a manual table_xinfo and creating the table via execute. Happy to adjust if that issue moves, but this seems like it works.

I also added my best-guess instructions for local development on this package. I'm shooting in the dark, so feel free to replace with how you work on it locally.

This is a version of #2031 for the 0.64.x branch.

:books: Documentation preview :books:: https://datasette--2044.org.readthedocs.build/en/2044/

Unlike the table view, the single row view does not resolve foreign key references into labels. This patch extracts the foreign key reference expansion code from TableView.data() into a standalone function that is then called by both TableView.data() and RowView.data().

:books: Documentation preview :books:: https://datasette--2031.org.readthedocs.build/en/2031/

...while maintaining given order.

Fixes #1975 (if I'm understanding correctly).

:books: Documentation preview :books:: https://datasette--2026.org.readthedocs.build/en/2026/

Fixes #2016

:books: Documentation preview :books:: https://datasette--2025.org.readthedocs.build/en/2025/

Refs #417. Run it like this:

 datasette -d ~/Library

Uses a new plugin hook - available_databases()

This modernizes some uses of double quotes for string literals to use only single quotes, fixes simonw/datasette#2001

While developing it, I manually enabled the stricter mode by using the code snippet at https://gist.github.com/cldellow/85bba507c314b127f85563869cd94820

I think that code snippet isn't generally safe/portable, so I haven't tried to automate it in the tests.

:books: Documentation preview :books:: https://datasette--2004.org.readthedocs.build/en/2004/

The presence of inner.* causes SQLite to materialize a row with all the columns. Those columns will be discarded later.

Instead, we can select only the column we'll use. This lets SQLite's optimizer realize that the other columns in the CTE definition aren't needed.

On a test table with 278K rows, 98K of which had an array, this speeds up the facet calculation from 4 sec to 1 sec.

:books: Documentation preview :books:: https://datasette--2008.org.readthedocs.build/en/2008/

Add support for compound foreign keys, as per issue #117

Not sure if this is the right approach. In particular I'm unsure about:

• the new ForeignKey class, which replaces the namedtuple in order to ensure that column and other_column are forced into tuples. The class does the job, but doesn't feel very elegant.
• I haven't rewritten guess_foreign_table to take account of multiple columns, so it just checks for the first column in the foreign key definition. This isn't ideal.
• I haven't added any ability to the CLI to add compound foreign keys, it's only in the python API at the moment.

The PR also contains a minor related change that columns and tables are always quoted in foreign key definitions.

sqlite foreign keys can be compound, but that is not as well supported by datasette as single column foreign keys.

in particular,

1. in a table view, there is not a link from the row to the referenced row if the foreign key is compound
2. in a row view, there is no listing of tables and rows that refer to the focal row if those referencing foreign keys are compound.

Both of these issues are discussed in #1099.

This PR only fixes the second one, because it's not clear what the right UX is for the first issue.

Some things that might not be desirable about this approach.

1. it changes the external API, by changing column => columns and other_column => other_columns (see inline comment for more discussion.
2. There are various places where the plural foreign keys have to be checked for length and discarded or transformed to singular.

Closes #1983

All documentation here. Edits welcome.

:books: Documentation preview :books:: https://datasette--1996.org.readthedocs.build/en/1996/

Not sure when this changed (iOS 16 maybe?), but the WorkoutStatistics now has a whole bunch of information about workouts, e.g. for runs it contains the distance (as a <WorkoutStatistics type="HKQuantityTypeIdentifierDistanceWalkingRunning ...> element).

Adding it as another column at leat allows me to pull these out (using SQLite's JSON support). I'm running with this patch on my own data now.

Given the code here, it seems like in the past this file was named "tweet.js". In recent exports, it's named "tweets.js". The archive importer needs to be modified to take this into account. Existing logic is reused for importing this table. (However, the resulting table name will be different, matching the different file name -- archive_tweets, rather than archive_tweet).

Previously, twitter-to-sqlite only supported OAuth1 authentication, and the token must be on behalf of a user. However, Twitter also supports application-only bearer tokens, documented here: https://developer.twitter.com/en/docs/authentication/oauth-2-0/bearer-tokens This PR adds support to twitter-to-sqlite for using application-only bearer tokens. To use, the auth.json file just needs to contain a "bearer_token" key instead of "api_key", "api_secret_key", etc.

• add to checkin_details view new column for a calculated photo links
• supported multiple links split by newline
• create events table if there's no events in the history to avoid SQL errors

Fixes #9.

https://github.com/simonw/sqlite-utils/pull/498

:books: Documentation preview :books:: https://sqlite-utils--499.org.readthedocs.build/en/499/

alternatively, self.disable_fts()

Refs #1822

:books: Documentation preview :books:: https://datasette--1823.org.readthedocs.build/en/1823/

a quick first pass at implementing the feature requested in https://github.com/dogsheep/github-to-sqlite/issues/64

testing instructions:

$ github-to-sqlite events events.db user/khimaros

if the specified user is the authenticated user, it will also include private events.

caveat: pagination appears to be broken (i don't see next in the response JSON from GitHub)

New --organization flag to signify all given "usernames" are private orgs. Adapts API URL to the organization path instead.

Not the best implementation, but a first draft to talk around

Fixes #75 (badly, no tests, overly vague, untested)

This allows _json to accept a wildcard for when there are many JSON columns that the user wants to convert. I hope this is useful. I've tested it on our datasette and haven't ran into any issues. I imagine on a large set of results, there could be some performance issues, but it will probably be negligible for most use cases.

On a side note, I ran into an issue where I had to upgrade black on my system beyond the pinned version in setup.py. Here is the upstream issue <https://github.com/psf/black/issues/2964 . I didn't include this in the PR yet since I didn't look into the issue too far, but I can if you would like.

Tested with:

 hacker-news-to-sqlite user hacker-news.db fernand0

Result: Traceback (most recent call last): File "/home/ftricas/.pyenv/versions/3.10.6/bin/hacker-news-to-sqlite", line 8, in <module> sys.exit(cli()) File "/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/hacker_news_to_sqlite/cli.py", line 27, in user submitted = user.pop("submitted", None) or [] AttributeError: 'NoneType' object has no attribute 'pop'

There is a problem of style with the patch (but not sure what to do) because with the new inicialization ( submitted = []) the part

 or []

is not needed. Maybe there is a more adequate way of doing this.

Recent versions of setuptools can run most things from setup.cfg so one can have a simpler version that does not require executing code on install.

The bulk of the changes were automated by running https://pypi.org/project/setup-py-upgrade/ with a few minor edits for the bits that it can not auto convert (the initial get_long_description() and get_version() can not be automatically converted)

Looks like a previous commit missed a photo-to-sqlite→ dogsheep-photos replacement.

Fix typo

I'm not sure if this is due to a new format or just because my ageinfo file is blank, but trying to import an archive would crash when it got to that file. This PR adds a guard clause in the ageinfo transformer and sets a default value that doesn't throw an exception. Seems likely to be the same issue mentioned by danp in https://github.com/dogsheep/twitter-to-sqlite/issues/54, my ageinfo file looks the same. Added that same ageinfo file to the test archive as well to help confirm my workaround didn't break anything.

Let me know if you want any changes!

Twitter pushes you heavily towards v2 projects instead of v1 apps – I know the README mentions v1 API compatibility at the top, but I still nearly got turned around here.

Some repos organize network dependents by a Package toggle. This PR adds the ability to page through those options and scrape underlying dependents.

Refs #1625

Refs #1191

Next steps:

• [ ] Get comfortable that this pattern is the right way to go
• [ ] Implement it for all of the other pages, not just the table page
• [ ] Add a new set of plugin tests that exercise ALL of these new hook locations
• [ ] Document, then ship

Refs #346 and #344.

Right now, datasette routes can only be a 2-tuple of (regex, view_fn).

If it was possible for datasette to handle extra options, like standard Django does, it would add flexibility for plugin authors.

For example, if extra options were enabled, then it would be easy to make a single table the home page (#1284). This plugin would accomplish it.

```python from datasette import hookimpl from datasette.views.table import TableView

@hookimpl def register_routes(datasette): return [ (r"^/$", TableView.as_view(datasette), {'db_name': 'DB_NAME', 'table': 'TABLE_NAME'}) ] ```

File name from google takeout has no space. The code only runs without error if filename is "MyActivity.json" and not "My Activity.json". Is it a new change by Google?

When parsing the Evernote <note> elements, the script will now also parse any nested <tag> elements, writing them out into a separate sqlite table.

Here is an example of how to query the data after the script has run: select notes.*, (select group_concat(tag) from notes_tags where notes_tags.note_id=notes.id) as tags from notes;

My .enex source file is 3+ years old so I am assuming the structure hasn't changed. Interestingly, my notebook names show up in the tags list where the tag name is prefixed with notebook_, so this could maybe help work around the first limitation mentioned in the evernote-to-sqlite blog post.

Adds a method for retrieving all the teams within an organisation and all the members in those teams. The latter is stored as a join table team_members beteween teams and users.

Description

Proposing a solution to the API limitation for merged_by in pull_requests. Specifically the following called out in the readme:

Note that the merged_by column on the pull_requests table will only be populated for pull requests that are loaded using the --pull-request option - the GitHub API does not return this field for pull requests that are loaded in bulk.

This approach might cause larger repos to hit rate limits called out in https://github.com/dogsheep/github-to-sqlite/issues/51 but seems to work well in the repos I tested and included below.

Old Behavior

• Had to list out the pull-requests individually via multiple --pull-request flags

New Behavior

• --merged-by flag for getting 'merge_by' information out of pull-requests without having to specify individual PR numbers.

Testing

Picking some repo that has more than one merger (datasette only has 1 😉 )

$ github-to-sqlite pull-requests ./github.db opnsense/tools --merged-by $ echo "select id, url, merged_by from pull_requests;" | sqlite3 ./github.db 83533612|https://github.com/opnsense/tools/pull/39|1915288 102632885|https://github.com/opnsense/tools/pull/43|1915288 149114810|https://github.com/opnsense/tools/pull/57|1915288 160394495|https://github.com/opnsense/tools/pull/64|1915288 163308408|https://github.com/opnsense/tools/pull/67|1915288 169723264|https://github.com/opnsense/tools/pull/69|1915288 171381422|https://github.com/opnsense/tools/pull/72|1915288 179938195|https://github.com/opnsense/tools/pull/77|1915288 196233824|https://github.com/opnsense/tools/pull/82|1915288 215289964|https://github.com/opnsense/tools/pull/93| 219696100|https://github.com/opnsense/tools/pull/97|1915288 223664843|https://github.com/opnsense/tools/pull/99| 228446172|https://github.com/opnsense/tools/pull/103|1915288 238930434|https://github.com/opnsense/tools/pull/110|1915288 255507110|https://github.com/opnsense/tools/pull/119|1915288 255980675|https://github.com/opnsense/tools/pull/120|1915288 261906770|https://github.com/opnsense/tools/pull/125| 263800503|https://github.com/opnsense/tools/pull/127|1915288 264038685|https://github.com/opnsense/tools/pull/128|1915288 264696704|https://github.com/opnsense/tools/pull/129|1915288 266660547|https://github.com/opnsense/tools/pull/130|1915288 273120409|https://github.com/opnsense/tools/pull/133|1915288 274370803|https://github.com/opnsense/tools/pull/135| 276600629|https://github.com/opnsense/tools/pull/139| 277303655|https://github.com/opnsense/tools/pull/141|1915288 293033714|https://github.com/opnsense/tools/pull/145| 294827649|https://github.com/opnsense/tools/pull/146| 295140008|https://github.com/opnsense/tools/pull/147|1915288 305690829|https://github.com/opnsense/tools/pull/150|9783985 307077931|https://github.com/opnsense/tools/pull/152|1915288 321782100|https://github.com/opnsense/tools/pull/155| 337265672|https://github.com/opnsense/tools/pull/160| 337267484|https://github.com/opnsense/tools/pull/161|1915288 368251763|https://github.com/opnsense/tools/pull/169| 428262505|https://github.com/opnsense/tools/pull/181| 437557011|https://github.com/opnsense/tools/pull/182|1915288 447079893|https://github.com/opnsense/tools/pull/185| 461822092|https://github.com/opnsense/tools/pull/191| 463290142|https://github.com/opnsense/tools/pull/193|1915288 470112962|https://github.com/opnsense/tools/pull/194|1915288 472644649|https://github.com/opnsense/tools/pull/195|1915288 488696898|https://github.com/opnsense/tools/pull/198| 513289902|https://github.com/opnsense/tools/pull/201| 522530265|https://github.com/opnsense/tools/pull/203| 564443347|https://github.com/opnsense/tools/pull/213| 597579516|https://github.com/opnsense/tools/pull/220|1915288 602860357|https://github.com/opnsense/tools/pull/221|1915288 608744738|https://github.com/opnsense/tools/pull/222|1915288 623279673|https://github.com/opnsense/tools/pull/228|1915288 664656182|https://github.com/opnsense/tools/pull/233| 664781786|https://github.com/opnsense/tools/pull/234|1915288 670683636|https://github.com/opnsense/tools/pull/235|1915288 683150764|https://github.com/opnsense/tools/pull/237| 685016233|https://github.com/opnsense/tools/pull/238| 687099825|https://github.com/opnsense/tools/pull/239|1915288 715705652|https://github.com/opnsense/tools/pull/244|1915288 715721248|https://github.com/opnsense/tools/pull/245|1915288 userid are now present for those PRs that were merged.

Without the flag the merged_by behavior remains missing as expected when get PRs bulk:

$ github-to-sqlite pull-requests ./github.db opnsense/tools $ echo "select id, url, merged_by from pull_requests;" | sqlite3 ./github.db 83533612|https://github.com/opnsense/tools/pull/39| 102632885|https://github.com/opnsense/tools/pull/43| 149114810|https://github.com/opnsense/tools/pull/57| 160394495|https://github.com/opnsense/tools/pull/64| 163308408|https://github.com/opnsense/tools/pull/67| 169723264|https://github.com/opnsense/tools/pull/69| 171381422|https://github.com/opnsense/tools/pull/72| 179938195|https://github.com/opnsense/tools/pull/77| 196233824|https://github.com/opnsense/tools/pull/82| 215289964|https://github.com/opnsense/tools/pull/93| 219696100|https://github.com/opnsense/tools/pull/97| 223664843|https://github.com/opnsense/tools/pull/99| 228446172|https://github.com/opnsense/tools/pull/103| 238930434|https://github.com/opnsense/tools/pull/110| 255507110|https://github.com/opnsense/tools/pull/119| 255980675|https://github.com/opnsense/tools/pull/120| 261906770|https://github.com/opnsense/tools/pull/125| 263800503|https://github.com/opnsense/tools/pull/127| 264038685|https://github.com/opnsense/tools/pull/128| 264696704|https://github.com/opnsense/tools/pull/129| 266660547|https://github.com/opnsense/tools/pull/130| 273120409|https://github.com/opnsense/tools/pull/133| 274370803|https://github.com/opnsense/tools/pull/135| 276600629|https://github.com/opnsense/tools/pull/139| 277303655|https://github.com/opnsense/tools/pull/141| 293033714|https://github.com/opnsense/tools/pull/145| 294827649|https://github.com/opnsense/tools/pull/146| 295140008|https://github.com/opnsense/tools/pull/147| 305690829|https://github.com/opnsense/tools/pull/150| 307077931|https://github.com/opnsense/tools/pull/152| 321782100|https://github.com/opnsense/tools/pull/155| 337265672|https://github.com/opnsense/tools/pull/160| 337267484|https://github.com/opnsense/tools/pull/161| 368251763|https://github.com/opnsense/tools/pull/169| 428262505|https://github.com/opnsense/tools/pull/181| 437557011|https://github.com/opnsense/tools/pull/182| 447079893|https://github.com/opnsense/tools/pull/185| 461822092|https://github.com/opnsense/tools/pull/191| 463290142|https://github.com/opnsense/tools/pull/193| 470112962|https://github.com/opnsense/tools/pull/194| 472644649|https://github.com/opnsense/tools/pull/195| 488696898|https://github.com/opnsense/tools/pull/198| 513289902|https://github.com/opnsense/tools/pull/201| 522530265|https://github.com/opnsense/tools/pull/203| 564443347|https://github.com/opnsense/tools/pull/213| 597579516|https://github.com/opnsense/tools/pull/220| 602860357|https://github.com/opnsense/tools/pull/221| 608744738|https://github.com/opnsense/tools/pull/222| 623279673|https://github.com/opnsense/tools/pull/228| 664656182|https://github.com/opnsense/tools/pull/233| 664781786|https://github.com/opnsense/tools/pull/234| 670683636|https://github.com/opnsense/tools/pull/235| 683150764|https://github.com/opnsense/tools/pull/237| 685016233|https://github.com/opnsense/tools/pull/238| 687099825|https://github.com/opnsense/tools/pull/239| 715705652|https://github.com/opnsense/tools/pull/244| 715721248|https://github.com/opnsense/tools/pull/245|

Individual PRs passed via --pull-request flag behaves as expected (unchanged):

$ github-to-sqlite pull-requests ./github.db opnsense/tools --pull-request 39 --pull-request 237 $ echo "select id, url, merged_by from pull_requests;" | sqlite3 ./github.db 83533612|https://github.com/opnsense/tools/pull/39|1915288 683150764|https://github.com/opnsense/tools/pull/237|

Picking 1 PR that has a merged_by (39) and one that does not (237)

Currently the _through clause can only work if the FK relationship is defined in a specific direction. I don't think there is any reason for this limitation, as an FK allows joining in both directions.

This is an admittedly hacky change to implement bidirectional joins using _through. It does work for our use-case, but I don't know if there are other implications that I haven't thought of. Also if this change is desirable we probably want to make the code a little nicer.

Fixes https://github.com/simonw/datasette/issues/1459

Adds support for --open --get /path to be used in combination.

If --open is provided alone, datasette will open a web page to a default URL. If --get <url> is provided alone, datasette will output the result of doing a GET to that URL and then exit. If --open --get <url> are provided together, datasette will open a web page to that URL.

TODO items: - [ ] update documentation - [ ] print out error message when --root --open --get <url> is used - [ ] adjust code to require that <url> start with a / when -o --get <url> is used - [ ] add test(s)

note, '@CTB' is used in this PR to flag code that needs revisiting.

This makes it so you can use Dogsheep if you're using Datasette with the --base-url /some-path/ setting.

Workflows that have an id in any step result in the following error when running workflows:

e.g.github-to-sqlite workflows github.db nixos/nixpkgs

Traceback (most recent call last): File "/usr/local/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1668, in invokeTraceback (most recent call last): File "/usr/local/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in call return self.main(args, kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 763, in invoke return __callback(args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/github_to_sqlite/cli.py", line 601, in workflows utils.save_workflow(db, repo_id, filename, content) File "/usr/local/lib/python3.8/dist-packages/github_to_sqlite/utils.py", line 865, in save_workflow db["steps"].insert_all( File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 2596, in insert_all self.insert_chunk( File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 2378, in insert_chunk result = self.db.execute(query, params) File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 419, in execute return self.conn.execute(sql, parameters) sqlite3.IntegrityError: datatype mismatch ```

• Information about the ID key in a step for GHA
• An example workflow from a public repo

Changes

I'm proposing that the key for id in step is replaced with step_id so that it no longer interferes with the table id for tracking the record.

Special thanks to @sarcasticadmin @egiffen and @ruebenramirez for helping a bit on this 😄

This solves a weird issue when there is record with metadata key that is only different in letter cases.

See the test for details.

This closes #21.

When there are metadata entries with the same case insensitive string, then there is an error when trying to create a new column for that metadata entry in the database table, because a column with that case insensitive name already exists.

xml <Record type="HKQuantityTypeIdentifierDietaryCholesterol" sourceName="MyFitnessPal" sourceVersion="35120" unit="mg" creationDate="2021-07-04 20:55:27 +0200" startDate="2021-07-04 20:55:00 +0200" endDate="2021-07-04 20:55:00 +0200" value="124"> <MetadataEntry key="meal" value="Dinner"/> <MetadataEntry key="Meal" value="Dinner"/> </Record>

The code added in this PR checks if a key already exists in a record and if so adds a number at its end. The resulting column names look like the example below then. Interestingly, the column names viewed with Datasette are not case insensitive.

text startDate, endDate, value, unit, sourceName, sourceVersion, creationDate, metadata_meal, metadata_Meal_2, metadata_Mahlzeit

Refs #1293, follows #942.

The tool covers only chat imports at the time of opening this PR but I'm planning to import everything else that I feel inquisitive about

ref: https://github.com/gavindsouza/instagram-to-sqlite

WIP

This PR adds the ability to import emails from a Gmail mbox export from Google Takeout.

This is my first PR to a datasette/dogsheep repo. I've tested this on my personal Google Takeout mbox with ~520,000 emails going back to 2004. This took around ~20 minutes to process.

To provide some feedback on the progress of the import I added the "rich" python module. I'm happy to remove that if adding a dependency is discouraged. However, I think it makes a nice addition to give feedback on the progress of a long import.

Do we want to log emails that have errors when trying to import them?

Dealing with encodings with emails is a bit tricky. I'm very open to feedback on how to deal with those better. As well as any other feedback for improvements.

This PR changes the main Dockerfile to use ubuntu:20.10 as base image instead of python:3.9.2-slim-buster (itself based on debian:buster-slim).

The Dockerfile is essentially the one from https://github.com/simonw/datasette/issues/1249#issuecomment-803698983 with some additional cleanups to slim it down.

This fixes a couple of issues: 1. The SQLite version in Debian Buster (2.6.0) doesn't support generated columns 2. Installing SpatiaLite from the Debian sid repositories has the side effect of also installing updates to libc and libstdc++ from sid.

As a bonus, the Docker image becomes smaller:

$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE datasette 0.56-ubuntu f7aca255140a 5 hours ago 212MB datasetteproject/datasette 0.56 efb3b282f390 13 days ago 258MB

Reproduction of the first issue

``` $ curl -O https://latest.datasette.io/fixtures.db % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 260k 0 260k 0 0 489k 0 --:--:-- --:--:-- --:--:-- 489k

$ docker run -v pwd:/mnt datasetteproject/datasette:0.56 datasette /mnt/fixtures.db Traceback (most recent call last): File "/usr/local/bin/datasette", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke return callback(args, kwargs) File "/usr/local/lib/python3.9/site-packages/datasette/cli.py", line 544, in serve asyncio.get_event_loop().run_until_complete(check_databases(ds)) File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete return future.result() File "/usr/local/lib/python3.9/site-packages/datasette/cli.py", line 584, in check_databases await database.execute_fn(check_connection) File "/usr/local/lib/python3.9/site-packages/datasette/database.py", line 155, in execute_fn return await asyncio.get_event_loop().run_in_executor( File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, self.kwargs) File "/usr/local/lib/python3.9/site-packages/datasette/database.py", line 153, in in_thread return fn(conn) File "/usr/local/lib/python3.9/site-packages/datasette/utils/init.py", line 892, in check_connection for r in conn.execute( sqlite3.DatabaseError: malformed database schema (generated_columns) - near "AS": syntax error ```

Here is the SQLite version:

`` $ docker run -vpwd`:/mnt -it datasetteproject/datasette:0.56 /bin/bash root@d9220d3b95dd:/# python3 Python 3.9.2 (default, Mar 27 2021, 02:50:26) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import sqlite3 sqlite3.version '2.6.0' ```

Reproduction of the second issue

$ docker build . -t datasette --build-arg VERSION=0.55 [...snip...] The following packages will be upgraded: libc-bin libc6 libstdc++6 [...snip...] Unpacking libc6:amd64 (2.31-11) over (2.28-10) ... [...snip...] Unpacking libstdc++6:amd64 (10.2.1-6) over (8.3.0-6) ... [...snip...]

Both libc and libstdc++ are backwards compatible, so the image still works, but it will result in a combination of libraries and Python versions that exists only in the Datasette image, so it's likely untested. In addition, since Debian sid is an always-changing rolling-release, the versions of libc, libstdc++, Spatialite, and their dependencies change frequently, so the library versions in the Datasette image will depend on the day when it was built.

Closes #197. I have added support for sorting by multiple parameters as mentioned in the issue above, and together with that, a suggestion on how to implement such sorting in the user interface.

Picking up the work done on #557 with a new PR. Seeing if I can get this working.

Fixes https://github.com/simonw/datasette/issues/1129

NOTE: I don't think this PR is ready, since the arm/v6 and arm/v7 images are failing pytest due to missing dependencies (gcc and friends). But it's pretty close.

Closes https://github.com/simonw/datasette/issues/1344 . Using a build-matrix for the platforms and this test, we test all the platforms in parallel. I also threw in container scanning.

Switch pip install to use either tags or commit shas

Notably! This also changes the Dockerfile so that it accepts tags or commit-shas.

```

It's backwards compatible with tags, but also lets you use shas

root@712071df17af:/# pip install git+git://github.com/simonw/datasette.git@0.56
Collecting git+git://github.com/simonw/datasette.git@0.56
Cloning git://github.com/simonw/datasette.git (to revision 0.56) to /tmp/pip-req-build-u6dhm945
Running command git clone -q git://github.com/simonw/datasette.git /tmp/pip-req-build-u6dhm945
Running command git checkout -q af5a7f1c09f6a902bb2a25e8edf39c7034d2e5de
Collecting Jinja2<2.12.0,>=2.10.3
Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB) ```

This lets you build the containers in CI every push for testing, which maybe resolves this problem?

Workflow run example

You can see the results in my workflow here. The commit history is different because I squashed this branch, also in the testing branch I had to change github.com/simonw to github.com/blairdrummond for the CI to pick up my git_sha.

Why did the builds fail?

NOTE: The results of all the tests fail, but for different reasons! A few fail to install Rust, the amd64 passes the tests (phew!) but has critical CVEs which fail the container scan, the Arm/v6 and Arm/v7 seem to fail to install the test dependencies due to missing programs like gcc. (gcc is not sufficient though, as this run indicates)

Hey .. Awesome work developing this project, that I found very useful to me and saved me some work.. Thanks.. :)

Some background to this PR... I've been searching around for a tool allowing me to transforming my personal collection of Evernote notes to a format easier to search and potentially easier import to future services.

Now I discovered problem processing my large data ~5GB using the existing source using Pythons builtin xml-parser that unfortunately was unable to succeed without exception breaking the process.

My first attempt I tried to adapt to more robust lxml package allowing huge data and with "recover", but even if it worked better it also failed processing the whole data. Even using the memory efficient etree.iterparse() it also unfortunately got into trouble.

And with no luck finding any other libraries successfully parsing this enormous file I instead chose to build a "hugexmlparser" module that allows parsing this huge file using yield (on a byte-to-byte-level) and allows you to set a maximum size for <note> to cater for potential malformed or undesirable large attachments to export, should succeed covering potential exceptions. Some cases found where the parses discover malformed XML within <content> so also in those cases try to save as much as possible by escaping (to be dealt at a later stage, better than nothing), and if a missing end </note> before new (malformed?) it would add this after encounter a new start-tag.

The code for the recovery process is a bit rough and for certain room for refactoring, but at the moment is seem to achieve what I wanted.

Now with the above we pass this a minor changed version of save_note_recovery() assure the existing works. Also adding this as a new recover-enex command to click and kept the original options. A couple of new tests was added as well to check against using this command.

Now this currently works to me, but thought I might share a PR in such as you find use for this yourself or found useful to others finding this repository.

As a second step .. When the time allows it would have been nice to also be able to easily export from SQLite to formatted HTML/MD and attachments saved... but that might perhaps be better a separate project ... or if you or someone else have something that might shared to save some trouble, I would be interested ;-)

Refs #1270, #1268, #1249

Before merging this I need to do some more testing (to make sure that expensive queries really are properly cancelled). I also need to delete a bunch of code relating to the old mechanism of cancelling queries.

[See comment below: this doesn't actually cancel the query due to a thread-local confusion]

Parses the datetime columns so they are subsequently properly recognized as datetime.

Fixes https://github.com/dogsheep/healthkit-to-sqlite/issues/17

Fixes https://github.com/dogsheep/healthkit-to-sqlite/issues/14

Refs #239. Still needs documentation and CLI implementation.

to keep the data write from failing if it came across an error during processing. In particular when trying to convert my HealthKit zip file (and that of my wife's) it would consistently error out with the following:

``` db.py 1709 insert_chunk result = self.db.execute(query, params)

db.py 226 execute return self.conn.execute(sql, parameters)

sqlite3.OperationalError: too many SQL variables

db.py 1709 insert_chunk result = self.db.execute(query, params)

db.py 226 execute return self.conn.execute(sql, parameters)

sqlite3.OperationalError: too many SQL variables

db.py 1709 insert_chunk result = self.db.execute(query, params)

db.py 226 execute return self.conn.execute(sql, parameters)

sqlite3.OperationalError: table rBodyMass has no column named metadata_HKWasUserEntered

healthkit-to-sqlite 8 <module> sys.exit(cli())

core.py 829 call return self.main(args, *kwargs)

core.py 782 main rv = self.invoke(ctx)

core.py 1066 invoke return ctx.invoke(self.callback, **ctx.params)

core.py 610 invoke return callback(args, *kwargs)

cli.py 57 cli convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)

utils.py 42 convert_xml_to_sqlite write_records(records, db)

utils.py 143 write_records db[table].insert_all(

db.py 1899 insert_all self.insert_chunk(

db.py 1720 insert_chunk self.insert_chunk(

db.py 1720 insert_chunk self.insert_chunk(

db.py 1714 insert_chunk result = self.db.execute(query, params)

db.py 226 execute return self.conn.execute(sql, parameters)

sqlite3.OperationalError: table rBodyMass has no column named metadata_HKWasUserEntered ```

Adding the try / except in the write_records seems to fix that issue.

Currently if we run datasette package on a configuration directory we'll get an exception when we try to hard link to the directory. This PR copies the tree and makes the Dockerfile run inspect on all *.db files.

Finally got around to trying dogsheep-photos but I want to use backblaze's b2 service instead of AWS S3. Had to add a way to optionally specify the endpoint_url to connect to. Then with the bucket being empty the initial key retrieval would fail. Probably a better way to see that the bucket is empty than doing a test inside the paginator loop.

Also probably a better way to specify the endpoint_url as we get and test for it twice using the same code in two different places but did not want to spend too much time worrying about it.

Potential fix for https://github.com/simonw/datasette/issues/859.

Disabling table counts for hidden tables speeds up database page quite a bit. In my setup it reduced load time by 2/3 (~300 -> ~90ms)

Refs #417.

Refs #510

See #569

This adds a straightforward set of classes to aid in the construction of SQL queries.

My plan for this was to allow plugins to manipulate the Datasette-generated SQL in a more structured way. I'm not sure that's going to work, but I feel like this is still a step forward - it reduces the number of intermediate variables in TableView.data which aids readability, and also factors out a lot of the boring string concatenation.

There are a fair number of minor structure changes in here too as I've tried to make the ordering of TableView.data a bit more logical. As far as I can tell, I haven't broken anything...

This PR is related to #293, but now all commits have been merged.

The purpose is to support other file formats that aren't SQLite, like files with PyTables format. I've tried to accomplish that using external connectors published with entry points.

The modifications in the original datasette code are minimal and many are in a separated file.

Adds the -A option to include apps from all organizations when searching app names for publish.