Via https://stackoverflow.com/a/18622264/454773 I note the ability to register custom handlers for novel datatypes that can map into and out of things like sqlite BLOBs.

From a quick look and a quick play, I didn't spot a way to do this in sqlite_utils?

For example:

```python

Via https://stackoverflow.com/a/18622264/454773

import sqlite3 import numpy as np import io

def adapt_array(arr): """ http://stackoverflow.com/a/31312102/190597 (SoulNibbler) """ out = io.BytesIO() np.save(out, arr) out.seek(0) return sqlite3.Binary(out.read())

def convert_array(text): out = io.BytesIO(text) out.seek(0) return np.load(out)

Converts np.array to TEXT when inserting

sqlite3.register_adapter(np.ndarray, adapt_array)

Converts TEXT to np.array when selecting

sqlite3.register_converter("array", convert_array) ```

```python from sqlite_utils import Database db = Database('test.db')

Reset the database connection to used the parsed datatype

sqlite_utils doesn't seem to support eg:

Database('test.db', detect_types=sqlite3.PARSE_DECLTYPES)

db.conn = sqlite3.connect(db_name, detect_types=sqlite3.PARSE_DECLTYPES)

Create a table the old fashioned way

but using the new custom data type

vector_table_create = """ CREATE TABLE dummy (title TEXT, vector array ); """

cur = db.conn.cursor() cur.execute(vector_table_create)

sqlite_utils doesn't appear to support custom types (yet?!)

The following errors on the "array" datatype

""" db["dummy"].create({ "title": str, "vector": "array", }) """ ```

We can then add / retrieve records from the database where the datatype of the vector field is a custom registered array type (which is to say, a numpy array):

```python import numpy as np

db["dummy"].insert({'title':"test1", 'vector':np.array([1,2,3])})

for row in db.query("SELECT * FROM dummy"): print(row['title'], row['vector'], type(row['vector']))

""" test1 [1 2 3] <class 'numpy.ndarray'> """ ```

It would be handy to be able to do this idiomatically in sqlite_utils.

Hi

I note that you define _hash() to create a hash_id from non-id column values in a table here.

It would be useful to be able to call a complementary function to generate a corresponding _id from a subset of specified columns when adding items to another table, eg to support the creation of foreign keys.

Or is there a better pattern for doing that?

If I try to add a list of dicts to an empty table using upsert_all, I get an error:

```python import sqlite3 from sqlite_utils import Database import pandas as pd

conx = sqlite3.connect(':memory') cx = conx.cursor() cx.executescript('CREATE TABLE "test" ("Col1" TEXT);')

q="SELECT * FROM test;" pd.read_sql(q, conx) #shows empty table

db = Database(conx) db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])

TypeError Traceback (most recent call last) <ipython-input-74-8c26d93d7587> in <module> 1 db = Database(conx) ----> 2 db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts) 1157 alter=alter, 1158 extracts=extracts, -> 1159 upsert=True, 1160 ) 1161

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert) 1040 sql = "INSERT OR IGNORE INTO {table} VALUES({pk_placeholders});".format( 1041 table=self.name, -> 1042 pks=", ".join(["[{}]".format(p) for p in pks]), 1043 pk_placeholders=", ".join(["?" for p in pks]), 1044 )

TypeError: 'NoneType' object is not iterable

```

A hacky workaround in use is:

python try: db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}]) except: db['test'].insert_all([{'Col1':'a'},{'Col1':'b'}])

Irrrespective of whether using column names containing a space or - character is good practice, SQLite does allow it, but sqlite-utils throws an error in the following cases:

```python from sqlite_utils import Database

dbname = 'test.db' DB = Database(sqlite3.connect(dbname))

import pandas as pd df = pd.DataFrame({'col1':range(3), 'col2':range(3)})

Convert pandas dataframe to appropriate list/dict format

DB['test1'].insert_all( df.to_dict(orient='records') )

Works fine

```

However:

python df = pd.DataFrame({'col 1':range(3), 'col2':range(3)}) DB['test1'].insert_all(df.to_dict(orient='records'))

throws:

```

OperationalError Traceback (most recent call last) <ipython-input-27-070b758f4f92> in <module>() 1 import pandas as pd 2 df = pd.DataFrame({'col 1':range(3), 'col2':range(3)}) ----> 3 DB['test1'].insert_all(df.to_dict(orient='records'))

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) 327 jsonify_if_needed(record.get(key, None)) for key in all_columns 328 ) --> 329 result = self.db.conn.execute(sql, values) 330 self.db.conn.commit() 331 self.last_id = result.lastrowid

OperationalError: near "1": syntax error ```

and:

python df = pd.DataFrame({'col-1':range(3), 'col2':range(3)}) DB['test1'].upsert_all(df.to_dict(orient='records'))

results in:

```

OperationalError Traceback (most recent call last) <ipython-input-28-654523549d20> in <module>() 1 import pandas as pd 2 df = pd.DataFrame({'col-1':range(3), 'col2':range(3)}) ----> 3 DB['test1'].insert_all(df.to_dict(orient='records'))

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) 327 jsonify_if_needed(record.get(key, None)) for key in all_columns 328 ) --> 329 result = self.db.conn.execute(sql, values) 330 self.db.conn.commit() 331 self.last_id = result.lastrowid

OperationalError: near "-": syntax error ```

The following works fine: ``` connX = sqlite3.connect('DELME.db', timeout=10)

dfX=pd.DataFrame({'col1':range(3),'col2':range(3)}) DBX = Database(connX) DBX['test'].upsert_all(dfX.to_dict(orient='records')) ```

But if a column is named order: ``` connX = sqlite3.connect('DELME.db', timeout=10)

dfX=pd.DataFrame({'order':range(3),'col2':range(3)}) DBX = Database(connX) DBX['test'].upsert_all(dfX.to_dict(orient='records')) ```

it throws an error:

```

OperationalError Traceback (most recent call last) <ipython-input-130-7dba33cd806c> in <module> 3 dfX=pd.DataFrame({'order':range(3),'col2':range(3)}) 4 DBX = Database(connX) ----> 5 DBX['test'].upsert_all(dfX.to_dict(orient='records'))

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order) 347 foreign_keys=foreign_keys, 348 upsert=True, --> 349 column_order=column_order, 350 ) 351

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) 327 jsonify_if_needed(record.get(key, None)) for key in all_columns 328 ) --> 329 result = self.db.conn.execute(sql, values) 330 self.db.conn.commit() 331 self.last_id = result.lastrowid

OperationalError: near "order": syntax error ```