Documentation for sqlite-utils extract: https://sqlite-utils.readthedocs.io/en/latest/cli.html#extracting-columns-into-a-separate-table

Trying this demo again: wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv sqlite-utils extract global.db power_plants country country_long --table countries --rename country_long name It worked!

The documentation for the .extract() method is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#extracting-columns-into-a-separate-table

I tried out the prototype of the CLI on the Global Power Plants data: wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv sqlite-utils extract global.db power_plants country country_long This threw an error because rowid columns are not yet supported. I fixed that like so: sqlite-utils transform global.db power_plants --rename rowid id sqlite-utils extract global.db power_plants country country_long That worked! But it didn't play great with Datasette, because the resulting extracted table had columns country and country_long and neither of those are called name or value or title.

Based on this I need to add rowid table support AND I need to implement the proposed rename= argument for renaming columns on their way into the new table.

The command-line version of this needs to accept a table and one or more columns, then a --table and --fk-column option.

Here's how I'm generating the examples for the documentation: ``` In [2]: import sqlite_utils

In [3]: db = sqlite_utils.Database(memory=True)

In [4]: db["Trees"].insert({"id": 1, "TreeAddress": "52 Vine St", "CommonName": ...: "Palm", "LatinName": "foo"}, pk="id") Out[4]: <Table Trees (id, TreeAddress, CommonName, LatinName)>

In [5]: db["Trees"].extract(["CommonName", "LatinName"], table="Species", fk_col ...: umn="species_id")

In [6]: print(db["Trees"].schema) CREATE TABLE "Trees" ( [id] INTEGER PRIMARY KEY, [TreeAddress] TEXT, [species_id] INTEGER, FOREIGN KEY(species_id) REFERENCES Species(id) )

In [7]: print(db["Species"].schema) CREATE TABLE [Species] ( [id] INTEGER PRIMARY KEY, [CommonName] TEXT, [LatinName] TEXT ) ```

Need to make sure this works correctly for rowid tables.

What to do if the table already exists? The .lookup() function already knows how to modify an existing table to create the correct constraints etc, so I'll rely on that mechanism.

So .extract() probably takes a batch_size= argument too, which defaults to maybe 1000.

table.extract() can take an optional progress= argument which is a callback which will be used to report progress - called after each batch with (num_done, total). It will get called with (0, total) once at the start to allow progress bars to be initialized. The command-line progress bar will use this.

And if you want to rename some of the columns in the new table: python db["trees"].extract(["common_name", "latin_name"], table="species", rename={"common_name": "name"})

In Python it looks like this: ```python

Simple case - species column species_id pointing to species table

db["trees"].extract("species")

Setting a custom table

db["trees"].extract("species", table="Species")

Custom foreign key column on trees

db["trees"].extract("species", fk_column="species")

Extracting multiple columns

db["trees"].extract(["common_name", "latin_name"])

(this creates a lookup table called common_name_latin_name ref'd by common_name_latin_name_id)

Or with explicit table (fk_column here defaults to species_id because of the table name)

db["trees"].extract(["common_name", "latin_name"], table="species") ```

I think I understand the shape of this feature now. It lets you specify one or more columns on the source table which will be extracted into another table. It uses the .lookup() mechanism to populate that other table, which means each unique column value / pair / triple will be assigned an integer ID.

That integer ID gets written back into the first of the columns that are being transformed. A .transform() call then converts that column to an integer (and drops the additional columns). Finally we set up the new foreign key relationship.

I think the new foreign key column is called company_name_id by default in this example but can be customized by passing --fk-column=xxx

Thinking more about this one: $ sqlite-utils extract my.db \ dea_sales company_name company_address \ --table companies The goal here is to pull the company name and address pair out into a separate table.

Some questions: - should this first verify that every company_name has just one company_address? I like the idea of a unique constraint on the created table for this. - what should the foreign key column that gets added to the companies table be called?

CLI design idea:

$ sqlite-utils extract my.db \
    dea_sales company_name

Here we just specify the original table and column - the new extracted table will automatically be called "company_name" and will have "id" and "value" columns, by default.

To set a custom extract table:

$ sqlite-utils extract my.db \
    dea_sales company_name \
    --table companies

And for extracting multiple columns and renaming them on the created table, maybe something like this:

$ sqlite-utils extract my.db \
    dea_sales company_name company_address \
    --table companies \
    --column company_name name \
    --column company_address address

.transform() has landed now which should make this a lot easier to solve.

This is going to need #114 (the transform_table() method) in order to convert string columns into integer foreign key columns.

Since these operations could take a long time against large tables, it would be neat if there was a progress bar option for the CLI command.

The operations are full table scans so calculating progress shouldn't be too difficult.

How about the Python version? That should be easier to design.

python db["dea_sales"].extract( columns=["company_name", "company_address"], to_table="companies" ) If we want to transform the extracted data (e.g. rename those columns) maybe support a transform= argument?

python db["dea_sales"].extract( columns=["company_name", "company_address"], to_table="companies", transform = lambda extracted: { "name": extracted["company_name"], "address": extracted["company_address"], } ) This would create a new "companies" table with three columns: id, name and address.

Would also be nice if there was a syntax for saying "... and use the value from this column as the primary key column in the newly created table".

So what could the interface to this look like? Especially for the CLI?

One option:

sqlite-utils extract dea_sales company_name companies name

Tricky thing here is that it's quite a large number of positional arguments:

sqlite-utils extract dea_sales company_name companies name
                     Table     column       New table New column (maybe optional?)

It would be great if this could supported multiple columns - for if a spreadsheet has e.g. a “Company Name”, “Company Address” pair of fields that always match each other and areduplicated many times.

This could be handled by creating the new table with two columns that are indexed as a unique compound key. Then you can easily get-or-create on the pairs (or triples or whatever) from the original table.

Challenge here is what does the CLI syntax look like. Something like this?

$ sqlite-utils extract dea_sales -c company_name -c company_address \
  --to companies --to-col name --to-col address

Perhaps the columns in the new table are FORCED to be the same as the old ones, hence avoiding some options? Bit restrictive… maybe they default to the same but you can customize?

$ sqlite-utils extract dea_sales -c company_name -c company_address -t companies