html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1422681850,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1422681850,IC_kwDOCGYnMM5UzGb6,21095447,2023-02-08T14:25:50Z,2023-02-08T14:29:09Z,NONE,"I live the patch here for others:
_original code_
```shell
$ which sqlite-utils | xargs cat
```
```python
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from sqlite_utils.cli import cli
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(cli())
```
_patched/sqlite-utils.py_
```python
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from sqlite_utils.cli import cli
# New imports
from unittest.mock import patch
from sqlite_utils.cli import VALID_COLUMN_TYPES
if __name__ == '__main__':
# Choices of the option `--type`
cli.commands['transform'].params[2].type.types[1].choices.append('DATETIME')
# The dicts has to be extended with a new type
with patch.dict('sqlite_utils.db.COLUMN_TYPE_MAPPING', {'DATETIME': 'DATETIME'}),\
patch('sqlite_utils.cli.VALID_COLUMN_TYPES', VALID_COLUMN_TYPES + (""DATETIME"", )):
# Command is unchanged
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(cli())
```
And now it's working
```bash
$ sqlite-utils schema events.sqlite cards.chunk.get
CREATE TABLE ""cards.chunk.get"" (
[id] INTEGER PRIMARY KEY NOT NULL,
[timestamp] TEXT,
)
$ python patched/sqlite-utils.py transform events.sqlite cards.chunk.get --type timestamp DATETIME
$ sqlite-utils schema events.sqlite cards.chunk.get
CREATE TABLE ""cards.chunk.get"" (
[id] INTEGER PRIMARY KEY NOT NULL,
[timestamp] DATETIME,
)
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421177666,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421177666,IC_kwDOCGYnMM5UtXNC,21095447,2023-02-07T17:39:00Z,2023-02-07T17:39:00Z,NONE,"> lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed
I've got your concern. Let's see if we will be replied on it and i'll close the issue some later.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421081939,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421081939,IC_kwDOCGYnMM5Us_1T,193185,2023-02-07T16:42:25Z,2023-02-07T16:43:42Z,NONE,"Ha, yes, I might end up making something very niche. That's OK.
I'm building a UI for [Datasette](https://datasette.io/) that lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed. And so on. I had hoped that I could just lean on sqlite-utils, but I think it's a little too surprising.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421055590,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421055590,IC_kwDOCGYnMM5Us5Zm,21095447,2023-02-07T16:25:31Z,2023-02-07T16:25:31Z,NONE,"> Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178
>
> I suspect you could overwrite the datetime entry to achieve what you want
And thank you for pointing me to it. At least, i can make a monkey patch for my need...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421052195,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421052195,IC_kwDOCGYnMM5Us4kj,21095447,2023-02-07T16:23:17Z,2023-02-07T16:23:57Z,NONE,"Isn't your suggestion too fundamental for the utility?
The bigger flexibility, the bigger complexity. Your idea make sense defenitely, but how often do you make schema changes? And how many people could benefit from it, what do you think?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421033725,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421033725,IC_kwDOCGYnMM5Us0D9,193185,2023-02-07T16:12:13Z,2023-02-07T16:12:13Z,NONE,"I think the bigger issue is that `sqlite-utils` mixes mechanism (it implements the [12-step way to alter SQLite tables](https://www.sqlite.org/lang_altertable.html#otheralter)) and policy (it has an opinionated stance on what column types should be used).
That might be a design choice to make it accessible to users by providing a reasonable set of defaults, but it doesn't quite fit my use case.
It might make sense to extract a separate library that provides just the mechanisms, and then `sqlite-utils` would sit on top of that library with its opinionated set of policies.
That would be a very big change, though.
I might take a stab at extracting the library, but just for the table schema migration piece, not all the other features that `sqlite-utils` supports. I wouldn't expect `sqlite-utils` to depend on it.
Part of my motivation is that I want to provide some other abilities, too, like support for CHECK constraints. I see that the issue in this repo (https://github.com/simonw/sqlite-utils/issues/358) proposes a bunch of short-hand constraints, which I wouldn't want to accidentally expose to people -- I want a layer that is a 1:1 mapping to SQLite.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421022917,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421022917,IC_kwDOCGYnMM5UsxbF,21095447,2023-02-07T16:06:03Z,2023-02-07T16:08:58Z,NONE,"> Do you see a way to enable it without affecting existing users or bumping the major version number?
I don't see a clean solution, only extending code with a side variable that tells us we want to apply advanced types instead of basic.
it could be a similiar command like `tranform-v2 --type column DATETIME` or a cli option `transform --adv-type column DATETIME` along with a dict that contains the advanced types. Then with knowledge that we run an advanced command we take that dictionary somehow, we can wrap the current and new dictionaries by a superdict and work with it everywhere according to the knowledge. This way shouldn't affect users who are using the previous lib versions and it have to be merged in the next major one.
But this way looks a bad design, too messy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420992261,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420992261,IC_kwDOCGYnMM5Usp8F,193185,2023-02-07T15:45:58Z,2023-02-07T15:45:58Z,NONE,"I'd support that, but I'm not the author of this library.
One challenge is that would be a breaking change. Do you see a way to enable it without affecting existing users or bumping the major version number?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420966995,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420966995,IC_kwDOCGYnMM5UsjxT,21095447,2023-02-07T15:29:28Z,2023-02-07T15:29:28Z,NONE,"I could, of course.
Doest it worth bringing such the improvement to the library?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420809773,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420809773,IC_kwDOCGYnMM5Ur9Yt,193185,2023-02-07T13:53:01Z,2023-02-07T13:53:01Z,NONE,"Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178
I suspect you could overwrite the datetime entry to achieve what you want","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420496447,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420496447,IC_kwDOCGYnMM5Uqw4_,21095447,2023-02-07T09:57:38Z,2023-02-07T09:57:38Z,NONE,"> That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work.
It works, but a column becomes `TEXT`
```python
In [1]: import sqlite_utils
In [2]: db = sqlite_utils.Database('events.sqlite')
In [3]: table = db['cards.chunk.get']
In [4]: table.columns_dict
Out[4]:
{'id': int,
'timestamp': float,
'data_chunk_number': int,
'user_id': str,
'meta_duplication_source_id': int,
'context_sort_attribute': str,
'context_sort_order': str}
In [5]: from datetime import datetime
In [7]: table.transform(types={'timestamp': datetime})
In [8]: table.columns_dict
Out[8]:
{'id': int,
'timestamp': str,
'data_chunk_number': int,
'user_id': str,
'meta_duplication_source_id': int,
'context_sort_attribute': str,
'context_sort_order': str}
```
```bash
❯ sqlite-utils schema events.sqlite cards.chunk.get
CREATE TABLE ""cards.chunk.get"" (
[id] INTEGER PRIMARY KEY NOT NULL,
[timestamp] TEXT,
...
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419734229,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1419734229,IC_kwDOCGYnMM5Un2zV,193185,2023-02-06T20:53:28Z,2023-02-06T21:16:29Z,NONE,"I think it's not currently possible: sqlite-utils requires that it be one of `integer`, `text`, `float`, `blob` ([see code](https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/cli.py#L2266))
IMO, this is a bit of friction and it would be nice if it was more permissive. SQLite permits developers to use any data type when creating a table. For example, this is a perfectly cromulent sqlite session that creates a table with columns of type `baz` and `bar`:
```
sqlite> create table foo(column1 baz, column2 bar);
sqlite> .schema foo
CREATE TABLE foo(column1 baz, column2 bar);
sqlite> select * from pragma_table_info('foo');
cid name type notnull dflt_value pk
---------- ---------- ---------- ---------- ---------- ----------
0 column1 baz 0 0
1 column2 bar 0 0
```
The idea is that the application developer will know what meaning to ascribe to those types. For example, I'm working on a plugin to Datasette. Dates are tricky to handle. If you have some existing rows, you can look at the values in them to know how a user is serializing the dates -- as an ISO 8601 string? An RFC 3339 string? With millisecond precision? With timezone offset? But if you don't yet have any rows, you have to guess. If the column is of type `TEXT`, you don't even know that it's meant to hold a date! In this case, my plugin will look to see if the column is of type `DATE` or `DATETIME`, and assume a certain representation when writing.
Perhaps there is an argument that sqlite-utils is trying to conform to SQLite's strict mode, and that is why it limits the choices. In strict mode, SQLite requires that the data type be one of `INT`, `INTEGER`, `REAL`, `TEXT`, `BLOB`, `ANY`. But that can't be the case -- sqlite-utils supports `FLOAT`, which is not one of the valid types in strict mode, and it rejects `INT`, `REAL` and `ANY`, which _are_ valid.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419740776,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1419740776,IC_kwDOCGYnMM5Un4Zo,193185,2023-02-06T20:59:01Z,2023-02-06T20:59:01Z,NONE,"That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419390560,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1419390560,IC_kwDOCGYnMM5Umi5g,21095447,2023-02-06T16:43:47Z,2023-02-06T16:43:47Z,NONE,"> SQLite doesn't have a native `DATETIME` type. It stores dates internally as strings and then has [functions](https://www.sqlite.org/lang_datefunc.html) to work with date-like strings. Yes it's weird.
That's correct. But my issue is about the application level libraries that, i suppose, have better data understanding if see a specific type such as `DATETIME`.
I'm writing data with **dataset** i've mentioned. The lib changes its behavior depending on a type. I saw different behavior with types `DATETIME, FLOAT, TEXT`. Dataset, for their part, is built upon Sqlalchemy, you know what it is.
To be honest, i didn't dive into the details of why the behavior changes, but when i altered manually by other util a type of column to `DATETIME` things got back to normal.
On the matter, can i achieve it with Sqlite Utils at the moment?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,