home / github

Menu
  • Search all tables
  • GraphQL API

issues

Table actions
  • GraphQL API for issues

238 rows where comments = 3, state = "closed" and type = "issue" sorted by updated_at descending

✖
✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: milestone, author_association, created_at (date), updated_at (date), closed_at (date)

repo 9

  • datasette 170
  • sqlite-utils 47
  • twitter-to-sqlite 7
  • github-to-sqlite 7
  • pocket-to-sqlite 3
  • dogsheep-beta 1
  • healthkit-to-sqlite 1
  • dogsheep-photos 1
  • apple-notes-to-sqlite 1

type 1

  • issue · 238 ✖

state 1

  • closed · 238 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
410384988 MDU6SXNzdWU0MTAzODQ5ODg= 411 How to pass named parameter into spatialite MakePoint() function dazzag24 1055831 closed 0     3 2019-02-14T16:30:22Z 2023-10-25T13:23:04Z 2019-05-05T12:25:04Z NONE  

Hi, datasette version: "0.26.2" extensions: spatialite: "4.4.0-RC0" sqlite version: "3.22.0"

I have a table of airports with latitude and longitude columns. I've added spatialite (with KNN support). After creating the db using csvs-to-sqlit, I run these commands to setup the spatialite tables:

``` conn.execute('SELECT InitSpatialMetadata(1)')

conn.execute("SELECT AddGeometryColumn('airports', 'point_geom', 4326, 'POINT', 2);")

conn.execute('''UPDATE airports SET point_geom = GeomFromText('POINT('||"longitude"||' '||"latitude"||')',4326);''')

conn.execute("SELECT CreateSpatialIndex('airports', 'point_geom');") ```

I'm attempting to create a canned query and have this in my metadata.json file: "find_airports_nearest_to_point":{ "sql":"SELECT a.pos AS rank, b.id, b.name, b.country, b.latitude AS latitude, b.longitude AS longitude, a.distance / 1000.0 AS dist_km FROM KNN AS a JOIN airports AS b ON (b.rowid = a.fid) WHERE f_table_name = \"airports\" AND ref_geometry = MakePoint( :Long , :Lat ) AND max_items = 10;"} which doesn't seem to perform the templating of the name parameters correctly and I get no results.

Have also tired: MakePoint( || :Long || , || :Lat || ) which returns this error: near "||": syntax error

However I cannot seem to find the correct combination of named parameter syntax (:Lat) or sqlite concatenation operator to make it work. Any ideas if using named parameters inside functions is supported?

Thanks Darren

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/411/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1930008379 I_kwDOBm6k_c5zCZc7 2197 click-default-group-wheel dependency conflict ar-jan 1176293 closed 0     3 2023-10-06T11:49:20Z 2023-10-12T21:53:17Z 2023-10-12T21:53:17Z NONE  

I upgraded my dependencies, then ran into this problem running datasette inspect:

env/lib/python3.9/site-packages/datasette/cli.py", line 6, in <module> from click_default_group import DefaultGroup ModuleNotFoundError: No module named 'click_default_group'

Turns out the released version of datasette still depends on click-default-group-wheel, so click-default-group doesn't get installed/recognized:

$ virtualenv venv $ source venv/bin/activate $ pip install datasette $ pip list | grep click-default-group click-default-group 1.2.4 click-default-group-wheel 1.2.3 $ python -c "from click_default_group import DefaultGroup" Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'click_default_group' $ pip install --force-reinstall click-default-group ... ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. datasette 0.64.4 requires click-default-group-wheel>=1.2.2, which is not installed. Successfully installed click-8.1.7 click-default-group-1.2.4

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2197/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
787098146 MDU6SXNzdWU3ODcwOTgxNDY= 1190 `datasette publish upload` mechanism for uploading databases to an existing Datasette instance tomershvueli 1024355 closed 0     3 2021-01-15T18:18:42Z 2023-08-30T22:16:39Z 2023-08-30T22:16:38Z NONE  

If I have a self-hosted instance of Datasette up and running, I'd like to be able to the use the CLI to publish databases to that instance, not only Google or Heroku. Ideally there'd be a url parameter or something similar to which one could point the publish command to their instance.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1190/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1781022369 I_kwDOBm6k_c5qKD6h 2091 Drop support for Python 3.7 simonw 9599 closed 0     3 2023-06-29T15:06:38Z 2023-08-23T18:18:18Z 2023-08-23T18:18:18Z OWNER  

It's EOL now, as of 2023-06-27 (two days ago): https://devguide.python.org/versions/

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2091/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
324720095 MDU6SXNzdWUzMjQ3MjAwOTU= 275 "config" section in metadata.json (root, database and table level) simonw 9599 closed 0     3 2018-05-20T16:02:28Z 2023-08-23T01:28:37Z 2023-08-23T01:28:37Z OWNER  

Split off from #274

Metadata should an optional "config" section at root, table or database level.

The TableView and RowView and DatabaseView and BaseView classes could all have a .config("key") method which knows how to resolve the hierarchy of configs.

This will allow individual tables (or databases) to set their own config settings for things like sql_time_limit_ms

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/275/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1857851384 I_kwDOCGYnMM5uvI_4 587 New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference simonw 9599 closed 0     3 2023-08-19T20:01:26Z 2023-08-19T20:04:33Z 2023-08-19T20:04:32Z OWNER  

Extremely detailed story of how I got to this point:

  • https://github.com/simonw/llm/issues/162

Steps to reproduce (only if that pragma is on though): bash python -c ' import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute(""" CREATE TABLE "logs" ( [id] INTEGER PRIMARY KEY, [model] TEXT, [prompt] TEXT, [system] TEXT, [prompt_json] TEXT, [options_json] TEXT, [response] TEXT, [response_json] TEXT, [reply_to_id] INTEGER, [chat_id] INTEGER REFERENCES [log]([id]), [duration_ms] INTEGER, [datetime_utc] TEXT ); """) db["logs"].add_foreign_key("reply_to_id", "logs", "id") ' This succeeds in some environments, fails in others.

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/587/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1838266862 I_kwDOBm6k_c5tkbnu 2126 Permissions in metadata.yml / metadata.json ctsrc 36199671 closed 0     3 2023-08-06T16:24:10Z 2023-08-11T05:52:30Z 2023-08-11T05:52:29Z NONE  

https://docs.datasette.io/en/latest/authentication.html#other-permissions-in-metadata says the following:

For all other permissions, you can use one or more "permissions" blocks in your metadata.

To grant access to the permissions debug tool to all signed in users you can grant permissions-debug to any actor with an id matching the wildcard * by adding this a the root of your metadata:

yaml permissions: debug-menu: id: '*'

I tried this.

My metadata.yml file looks like:

yaml permissions: debug-menu: id: '*' permissions-debug: id: '*' plugins: datasette-auth-passwords: myuser_password_hash: $env: "PASSWORD_HASH_MYUSER"

And then I run

zsh datasette -m metadata.yml tiddlywiki.db --root

And I open a session for the "root" user of datasette with the link given.

I open a private browser session and log in as "myuser" from http://127.0.0.1:8001/-/login

Then I check http://127.0.0.1:8001/-/actor which confirms that I am logged in as the "myuser" actor

json { "actor": { "id": "myuser" } }

In the session where I am logged in as "myuser" I then try to go to http://127.0.0.1:8001/-/permissions

But all I get there as the logged in user "myuser" is

Forbidden

Permission denied

And then if I check the http://127.0.0.1:8001/-/permissions as the datasette "root" user from another browser session, I see:

permissions-debug checked at 2023-08-06T16:22:58.997841 ✗ (used default)

Actor: {"id": "myuser"}

It seems that in spite of having tried to give the permissions-debug permission to the "myuser" user in my metadata.yml file, datasette does not agree that "myuser" has permission permissions-debug..

What do I need to do differently so that my "myuser" user is able to access http://127.0.0.1:8001/-/permissions ?

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2126/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1823393475 I_kwDOBm6k_c5srsbD 2119 database color shows only on index page, not other pages simonw 9599 closed 0   Datasette 1.0 3268330 3 2023-07-27T00:19:39Z 2023-08-11T05:25:45Z 2023-08-11T05:16:24Z OWNER  

I think this has been a bug for a long time.

https://latest.datasette.io/ currently shows:

Those colors are based on a hash of the database name. But when you click through to https://latest.datasette.io/fixtures

It's red on all sub-pages too.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2119/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1843600087 I_kwDOBm6k_c5t4xrX 2135 Release notes for 1.0a3 simonw 9599 closed 0   Datasette 1.0a3 9700784 3 2023-08-09T16:09:26Z 2023-08-09T19:17:07Z 2023-08-09T19:17:06Z OWNER  

118 commits! https://github.com/simonw/datasette/compare/1.0a2...26be9f0445b753fb84c802c356b0791a72269f25

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2135/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1822940263 I_kwDOBm6k_c5sp9xn 2114 Implement canned queries against new query JSON work simonw 9599 closed 0   Datasette 1.0a3 9700784 3 2023-07-26T18:24:50Z 2023-08-09T15:26:58Z 2023-08-09T15:26:57Z OWNER  
  • 2109

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2114/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1822949756 I_kwDOBm6k_c5sqAF8 2116 Turn DatabaseDownload into an async view function simonw 9599 closed 0   Datasette 1.0a3 9700784 3 2023-07-26T18:31:59Z 2023-07-26T18:44:00Z 2023-07-26T18:44:00Z OWNER  

A minor refactor, but it is a good starting point for this new branch. Refs: - #2109

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2116/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1816857442 I_kwDOBm6k_c5sSwti 2106 `datasette install -e` option simonw 9599 closed 0     3 2023-07-22T18:33:42Z 2023-07-26T18:28:33Z 2023-07-22T18:42:54Z OWNER  

As seen in LLM and now in sqlite-utils too: - https://github.com/simonw/sqlite-utils/issues/570

Useful for developing plugins, see tutorial at https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2106/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1816918185 I_kwDOCGYnMM5sS_ip 574 `prepare_connection()` plugin hook simonw 9599 closed 0     3 2023-07-22T22:52:47Z 2023-07-22T23:13:14Z 2023-07-22T22:59:10Z OWNER  

Splitting off an issue for prepare_connection() since Alex got the PR in seconds before I shipped 3.34!

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646686424

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/574/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1816852402 I_kwDOCGYnMM5sSvey 569 register_command plugin hook simonw 9599 closed 0     3 2023-07-22T18:17:27Z 2023-07-22T19:19:35Z 2023-07-22T19:19:35Z OWNER  

I'm going to start by adding the register_command hook using the exact same pattern as Datasette and LLM.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643450

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/569/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1718595700 I_kwDOCGYnMM5mb7B0 550 AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7 simonw 9599 closed 0     3 2023-05-21T18:24:39Z 2023-05-21T18:42:25Z 2023-05-21T18:41:58Z OWNER  

https://github.com/simonw/sqlite-utils/actions/runs/5039064797/jobs/9036965488

Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.7.16/x64/bin/flake8", line 8, in <module> sys.exit(main()) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/cli.py", line 22, in main app.run(argv) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py", line 363, in run self._run(argv) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py", line 350, in _run self.initialize(argv) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py", line 330, in initialize self.find_plugins(config_finder) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py", line 153, in find_plugins self.check_plugins = plugin_manager.Checkers(local_plugins.extension) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 357, in __init__ self.namespace, local_plugins=local_plugins File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 238, in __init__ self._load_entrypoint_plugins() File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 254, in _load_entrypoint_plugins eps = importlib_metadata.entry_points().get(self.namespace, ()) AttributeError: 'EntryPoints' object has no attribute 'get' Error: Process completed with exit code 1.

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/550/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1516644980 I_kwDOCGYnMM5aZip0 520 rows_from_file() raises confusing error if file-like object is not in binary mode simonw 9599 closed 0     3 2023-01-02T19:00:14Z 2023-05-08T22:08:07Z 2023-05-08T22:08:07Z OWNER  

I got this error:

File "/Users/simon/Dropbox/Development/openai-to-sqlite/openai_to_sqlite/cli.py", line 27, in embeddings rows, _ = rows_from_file(input) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/simon/.local/share/virtualenvs/openai-to-sqlite-jt4obeb2/lib/python3.11/site-packages/sqlite_utils/utils.py", line 305, in rows_from_file first_bytes = buffered.peek(2048).strip() ^^^^^^^^^^^^^^^^^^^ From this code: ```python

@cli.command() @click.argument( "db_path", type=click.Path(file_okay=True, dir_okay=False, allow_dash=False), ) @click.option( "-i", "--input", type=click.File("r"), default="-", ) def embeddings(db_path, input): "Store embeddings for one or more text documents" click.echo("Here is some output") db = sqlite_utils.Database(db_path) rows, _ = rows_from_file(input) print(list(rows)) `` The error went away when I changed it totype=click.File("rb")`.

This should either be called out in the documentation or rows_from_file() should be fixed to handle text-mode files in addition to binary files.

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/520/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1615891776 I_kwDOBm6k_c5gUI1A 2037 Test failure: FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError simonw 9599 closed 0     3 2023-03-08T20:30:06Z 2023-03-09T22:33:39Z 2023-03-09T22:33:39Z OWNER  

FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError: [Errno 2] No such file or directory

From https://github.com/simonw/datasette/actions/runs/4348548218/jobs/7597208191

``` =================================== FAILURES =================================== ____ test_install_requirements _______

run_module = <MagicMock name='run_module' id='139768358191936'>

@mock.patch("datasette.cli.run_module")
def test_install_requirements(run_module):
    runner = CliRunner()
  with runner.isolated_filesystem():

/home/runner/work/datasette/datasette/tests/test_cli.py:184:


/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/contextlib.py:119: in enter return next(self.gen)


self = <click.testing.CliRunner object at 0x7f1e5bfb9490>, temp_dir = None

@contextlib.contextmanager
def isolated_filesystem(
    self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None
) -> t.Iterator[str]:
    """A context manager that creates a temporary directory and
    changes the current working directory to it. This isolates tests
    that affect the contents of the CWD to prevent them from
    interfering with each other.

    :param temp_dir: Create the temporary directory under this
        directory. If given, the created directory is not removed
        when exiting.

    .. versionchanged:: 8.0
        Added the ``temp_dir`` parameter.
    """
  cwd = os.getcwd()

E FileNotFoundError: [Errno 2] No such file or directory

/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/click/testing.py:466: FileNotFoundError ``` Not sure why it only affected the "Calculate test coverage" one.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2037/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1616347574 I_kwDOJHON9s5gV4G2 1 Initial proof of concept with ChatGPT simonw 9599 closed 0     3 2023-03-09T03:44:39Z 2023-03-09T03:51:55Z 2023-03-09T03:51:55Z MEMBER  

I'm using ChatGPT to figure out enough AppleScript to get at my notes data.

apple-notes-to-sqlite 611552758 issue    
{
    "url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1515185383 I_kwDOBm6k_c5aT-Tn 1971 Upgrade for Sphinx 6.0 (once Furo has support for it) simonw 9599 closed 0     3 2022-12-31T19:04:35Z 2023-01-10T02:02:34Z 2023-01-10T02:02:34Z OWNER  

A deployment of #1967 to ReadTheDocs just failed like this: https://readthedocs.org/projects/datasette/builds/19045460/

``` Running Sphinx v6.0.0 making output directory... done building [mo]: targets for 0 po files that are out of date building [html]: targets for 28 source files that are out of date updating environment: [new config] 28 added, 0 changed, 0 removed reading sources... [ 3%] authentication reading sources... [ 7%] binary_data reading sources... [ 10%] changelog

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 299, in next_line self.line = self.input_lines[self.line_offset] File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 1136, in getitem return self.data[i] IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 226, in run self.next_line() File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 302, in next_line raise EOFError EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/cmd/build.py", line 281, in build_main app.build(args.force_all, args.filenames) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/application.py", line 344, in build self.builder.build_update() File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/builders/init.py", line 310, in build_update self.build(to_build, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/builders/init.py", line 326, in build updated_docnames = set(self.read()) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/builders/init.py", line 433, in read self._read_serial(docnames) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/builders/init.py", line 454, in _read_serial self.read_doc(docname) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/builders/init.py", line 510, in read_doc publisher.publish() File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/core.py", line 224, in publish self.document = self.reader.read(self.source, self.parser, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/io.py", line 103, in read self.parse() File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/readers/init.py", line 76, in parse self.parser.parse(self.input, document) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/parsers.py", line 78, in parse self.statemachine.run(inputlines, document, inliner=self.inliner) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 169, in run results = StateMachineWS.run(self, input_lines, input_offset, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 233, in run context, next_state, result = self.check_line( File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 445, in check_line return method(match, context, next_state) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 3024, in text self.section(title.lstrip(), source, style, lineno + 1, messages) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 325, in section self.new_subsection(title, lineno, messages) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 391, in new_subsection newabsoffset = self.nested_parse( File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 279, in nested_parse state_machine.run(block, input_offset, memo=self.memo, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 195, in run results = StateMachineWS.run(self, input_lines, input_offset) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 233, in run context, next_state, result = self.check_line( File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 445, in check_line return method(match, context, next_state) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2785, in underline self.section(title, source, style, lineno - 1, messages) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 325, in section self.new_subsection(title, lineno, messages) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 391, in new_subsection newabsoffset = self.nested_parse( File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 279, in nested_parse state_machine.run(block, input_offset, memo=self.memo, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 195, in run results = StateMachineWS.run(self, input_lines, input_offset) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 233, in run context, next_state, result = self.check_line( File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 445, in check_line return method(match, context, next_state) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 1273, in bullet i, blank_finish = self.list_item(match.end()) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 1295, in list_item self.nested_parse(indented, input_offset=line_offset, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 279, in nested_parse state_machine.run(block, input_offset, memo=self.memo, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 195, in run results = StateMachineWS.run(self, input_lines, input_offset) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/statemachine.py", line 239, in run result = state.eof(context) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2725, in eof self.blank(None, context, None) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2716, in blank paragraph, literalnext = self.paragraph( File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 416, in paragraph textnodes, messages = self.inline_text(text, lineno) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 425, in inline_text nodes, messages = self.inliner.parse(text, lineno, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 649, in parse before, inlines, remaining, sysmessages = method(self, match, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 792, in interpreted_or_phrase_ref nodelist, messages = self.interpreted(rawsource, escaped, role, File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 889, in interpreted nodes, messages2 = role_fn(role, rawsource, text, lineno, self) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/ext/extlinks.py", line 101, in role title = caption % part TypeError: not all arguments converted during string formatting

Exception occurred: File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/latest/lib/python3.9/site-packages/sphinx/ext/extlinks.py", line 101, in role title = caption % part TypeError: not all arguments converted during string formatting The full traceback has been saved in /tmp/sphinx-err-kq7ylgqo.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks! ```

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1971/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
710650633 MDU6SXNzdWU3MTA2NTA2MzM= 979 Default table view JSON should include CREATE TABLE simonw 9599 closed 0     3 2020-09-28T23:54:58Z 2023-01-09T15:32:39Z 2023-01-09T15:32:22Z OWNER  

https://latest.datasette.io/fixtures/facetable.json doesn't currently include the CREATE TABLE statement for the page, even though it's available on the HTML version at https://latest.datasette.io/fixtures/facetable

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/979/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1214859703 I_kwDOBm6k_c5IaUm3 1719 Refactor `RowView` and remove `RowTableShared` simonw 9599 closed 0     3 2022-04-25T18:06:24Z 2022-12-01T21:15:19Z 2022-04-25T18:33:44Z OWNER  

The RowTableShared class is making this a whole lot more complicated.

I'm going to split the RowView view out into an entirely separate views/row.py module.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1715#issuecomment-1108875068

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1719/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1470320227 I_kwDOBm6k_c5Xo05j 1923 latest.datasette.io Cloud Run deploys failing simonw 9599 closed 0     3 2022-11-30T22:49:34Z 2022-11-30T23:04:56Z 2022-11-30T23:04:56Z OWNER  

https://github.com/simonw/datasette/actions/runs/3587402085/jobs/6038106719v

Warning: "service_account_key" has been deprecated. Please switch to using google-github-actions/auth which supports both Workload Identity Federation and Service Account Key JSON authentication. For more details, see https://github.com/google-github-actions/setup-gcloud#authorization Error: google-github-actions/setup-gcloud failed with: failed to execute command `gcloud --quiet auth activate-service-account *** --key-file -`: /opt/hostedtoolcache/gcloud/275.0.0/x64/lib/googlecloudsdk/core/console/console_io.py:544: SyntaxWarning: "is" with a literal. Did you mean "=="? if answer is None or (answer is '' and default is not None): ERROR: gcloud failed to load: module 'collections' has no attribute 'MutableMapping'

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1923/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1450312343 I_kwDOBm6k_c5WcgKX 1892 Merge 1.0-dev branch back to main simonw 9599 closed 0   Datasette 1.0a0 8658075 3 2022-11-15T20:04:25Z 2022-11-29T19:40:23Z 2022-11-29T19:40:23Z OWNER  

I'm committed enough to the 1.0 work now that I'm ready for the main branch to reflect that instead.

If I need to make any dot-releases against 0.63 I can do those from a branch.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1892/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1434911255 I_kwDOCGYnMM5VhwIX 510 Cannot enable FTS5 despite it being available ar-jan 1176293 closed 0     3 2022-11-03T16:03:49Z 2022-11-18T18:37:52Z 2022-11-17T10:36:28Z NONE  

When I do sqlite-utils enable-fts my.db table_name column_name (with or without --fts5), I get an FTS4 virtual table instead of the expected FTS5.

FTS5 is however available and Python/SQLite versions do not seem to be the issue. I can manually create the FTS5 virtual table, and then Datasette also works with it from this same Python environment.

>>> sqlite3.version 2.6.0 >>> sqlite3.sqlite_version 3.39.4

PRAGMA compile_options; includes ENABLE_FTS5.

sqlite-utils, version 3.30.

Any ideas what's happening and how to fix?

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/510/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1450952393 I_kwDOCGYnMM5We8bJ 512 mypy failures in CI simonw 9599 closed 0     3 2022-11-16T06:22:48Z 2022-11-16T07:49:51Z 2022-11-16T07:49:50Z OWNER  

https://github.com/simonw/sqlite-utils/actions/runs/3472012235 failed on Python 3.11:

Truncated output: sqlite_utils/db.py:2467: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True sqlite_utils/db.py:2467: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase sqlite_utils/db.py:2530: error: Incompatible default for argument "where" (default has type "None", argument has type "str") [assignment] sqlite_utils/db.py:2530: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True sqlite_utils/db.py:2530: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase sqlite_utils/db.py:2658: error: Argument 1 to "count_where" of "Queryable" has incompatible type "Optional[str]"; expected "str" [arg-type] Found 23 errors in 1 file (checked 51 source files) Best look at https://github.com/hauntsaninja/no_implicit_optional

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/512/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1423369494 I_kwDOBm6k_c5U1uUW 1859 datasette create-token CLI command simonw 9599 closed 0   Datasette 1.0a0 8658075 3 2022-10-26T03:12:59Z 2022-11-15T19:59:00Z 2022-10-26T04:31:39Z OWNER  

The CLI equivalent of the /-/create-token page.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1859/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1423336122 I_kwDOBm6k_c5U1mK6 1856 allow_signed_tokens setting for disabling API signed token mechanism simonw 9599 closed 0   Datasette 1.0a0 8658075 3 2022-10-26T02:20:55Z 2022-11-15T19:57:05Z 2022-10-26T02:58:35Z OWNER  

Had some design thoughts here: https://github.com/simonw/datasette/issues/1852#issuecomment-1291272280

I liked this option the most:

--setting allow_create_tokens off
datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1856/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1429029604 I_kwDOCGYnMM5VLULk 506 Make `cursor.rowcount` accessible (wontfix) simonw 9599 closed 0     3 2022-10-30T21:51:55Z 2022-11-01T17:37:47Z 2022-11-01T17:37:13Z OWNER  

In building this Datasette feature on top of sqlite-utils I thought it might be useful to expose the number of rows that had been affected by a bulk insert or update - the cursor.rowcount:

  • https://github.com/simonw/datasette/issues/1866

This isn't currently exposed by sqlite-utils.

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/506/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1428560020 I_kwDOBm6k_c5VJhiU 1872 SITE-BUSTING ERROR: "render_template() called before await ds.invoke_startup()" mroswell 192568 closed 0     3 2022-10-30T02:28:39Z 2022-10-30T06:26:01Z 2022-10-30T06:26:01Z CONTRIBUTOR  
  1. My https://list.saferdisinfectants.org/disinfectants/listN page (linked from https://SaferDisinfectants.org ) has been running beautifully for a year and a half, including a GitHub Actions workflow that's been routinely updating the database.

  2. I received a recent report that the list page is down. I don't know when it went down, but the content is replaced with: "render_template() called before await ds.invoke_startup()"

  3. The local datasette repo runs without incident.

  4. The site is hosted on vercel, linked to my github repo. Perhaps some vercel changes were made, but not by anyone on our side. Here is a screenshot of the current project settings:

Here a screenshot of the latest deployment status:

This is my repository: https://github.com/mroswell/list-N (I notice: datasette==0.59 in my requirements.txt file)

Because it's been long while since I actively worked on this or any other datasette project, I forget a lot of what I knew at one point. Perhaps some configuration file could be missing? Or perhaps I just need to know the right incantation to add to that vercel settings page.

Help is welcome as the nonprofit org is soon hosting its annual conference, and we'd love to have the page working again.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1872/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1426253476 I_kwDOBm6k_c5VAuak 1869 Release 0.63 simonw 9599 closed 0     3 2022-10-27T20:53:01Z 2022-10-27T22:24:38Z 2022-10-27T22:11:33Z OWNER  

Most of the release notes are already written: - https://github.com/simonw/datasette/releases/tag/0.63a0 - https://github.com/simonw/datasette/releases/tag/0.63a1

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1869/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1392690202 I_kwDOCGYnMM5TAsQa 495 Support JSON values returned from .convert() functions mhalle 649467 closed 0     3 2022-09-30T16:33:49Z 2022-10-25T21:23:37Z 2022-10-25T21:23:28Z NONE  

When using the convert function on a JSON column, the result of the conversion function must be a string. If the return value is either a dict (object) or a list (array), the convert call will error out with an unhelpful user defined function exception.

It makes sense that since the original column value was a string and required conversion to data structures, the result should be converted back into a JSON string as well. However, other functions auto-convert to JSON string representation, so the fact that convert doesn't could be surprising.

At least the documentation should note this requirement, because the sqlite error messages won't readily reveal the issue.

Jf only sqlite's JSON column type meant something :)

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/495/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1422973111 I_kwDOBm6k_c5U0Ni3 1854 Flaky test: test_serve_localhost_http simonw 9599 closed 0     3 2022-10-25T19:37:35Z 2022-10-25T19:53:02Z 2022-10-25T19:53:02Z OWNER  

Failing on Python 3.10 at the moment: https://github.com/simonw/datasette/actions/runs/3323629947/jobs/5494340302

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1854/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1420090659 I_kwDOBm6k_c5UpN0j 1848 Private database page should show padlock on every table simonw 9599 closed 0     3 2022-10-24T02:28:38Z 2022-10-24T02:50:29Z 2022-10-24T02:42:34Z OWNER  

Following: - #1829

https://latest.datasette.io/_internal looks like this:

But those queries and tables are private too, and should also show the padlock icon.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1848/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1157182254 I_kwDOBm6k_c5E-TMu 1646 Configuration directory mode does not pick up other file extensions than .db dnsos 15640196 closed 0     3 2022-03-02T13:15:23Z 2022-10-07T23:06:17Z 2022-10-07T23:03:35Z NONE  

Hello, I've been trying to run Datasette with the configuration directory mode with a structure such as this one:

plain some-directory/ example.sqlite3 another-example.db one-more.custom [...]

(In my scenario I can't just change the filename extension without other problems arising)

Now databases with the .sqlite3 or the custom filename extension are ignored by Datasette in this case. I'm aware that the docs state that a .db extension is required, but I was wondering if there is a reason for restricting this or any workaround available? When I run datasette example.sqlite3 or datasette one-more.custom the databases are served by Datasette without a problem.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1646/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1388631785 I_kwDOBm6k_c5SxNbp 1826 render_cell documentation example doesn't match the method signature pjamargh 66709385 closed 0     3 2022-09-28T02:37:59Z 2022-09-28T04:30:28Z 2022-09-28T04:05:16Z NONE  

Open Datasette stable doc at https://docs.datasette.io/en/stable/plugin_hooks.html?highlight=render_cell#render-cell-row-value-column-table-database-datasette

render_cell plugin hook method signature is render_cell(row, value, column, table, database, datasette), the example shown inline uses render_cell(value).

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1826/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1385026210 I_kwDOBm6k_c5SjdKi 1819 Preserve query on timeout danp 2182 closed 0     3 2022-09-25T13:32:31Z 2022-09-26T23:16:15Z 2022-09-26T23:06:06Z CONTRIBUTOR  

If a query hits the timeout it shows a message like:

SQL query took too long. The time limit is controlled by the sql_time_limit_ms configuration option.

But the query is lost. Hitting the browser back button shows the query before the one that errored.

It would be nice if the query that errored was preserved for more tweaking. This would make it similar to how "invalid syntax" works since #1346 / #619.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1819/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1246826792 I_kwDODLZ_YM5KUREo 10 When running `auth` command, don't overwrite an existing auth.json file ashanan 11887 closed 0     3 2022-05-24T16:42:20Z 2022-09-07T15:07:38Z 2022-08-22T16:17:19Z NONE  

Ran the auth command in the same directory I'd previously set up an auth.json file for twitter-to-sqlite and it was completely overwritten. Not the biggest issue, but still unexpected. Ideally, for me, the keys would just be added to the existing file, but getting a warning and a chance to back out would be a good solution as well.

pocket-to-sqlite 213286752 issue    
{
    "url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1362363685 I_kwDOBm6k_c5RNAUl 1800 Remove upper bound dependencies as a default policy simonw 9599 closed 0     3 2022-09-05T18:23:45Z 2022-09-05T18:39:52Z 2022-09-05T18:35:41Z OWNER  

https://iscinumpy.dev/post/bound-version-constraints/ has convinced me not to use upper bound dependencies unless I'm certain they are needed.

Relevant PR: - https://github.com/simonw/datasette/pull/1799

Also:

https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/setup.py#L45-L46

https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/setup.py#L48-L49

https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/setup.py#L51-L55

https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/setup.py#L57-L59

https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/setup.py#L75-L78

https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/setup.py#L81-L82

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1800/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1345452427 I_kwDODLZ_YM5QMfmL 11 -a option is used for "--auth" and for "--all" fernand0 2467 closed 0     3 2022-08-21T10:50:48Z 2022-08-21T21:11:57Z 2022-08-21T21:11:57Z NONE  

I'm not sure which option is best, instead of -a -all.

pocket-to-sqlite 213286752 issue    
{
    "url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1278571700 I_kwDOCGYnMM5MNXS0 447 Incorrect syntax highlighting in docs CLI reference simonw 9599 closed 0     3 2022-06-21T14:53:10Z 2022-06-21T18:48:47Z 2022-06-21T18:48:46Z OWNER  

https://sqlite-utils.datasette.io/en/stable/cli-reference.html#insert

It looks like Python keywords are being incorrectly highlighted here.

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/447/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1277295119 I_kwDOCGYnMM5MIfoP 445 `sqlite_utils.utils.TypeTracker` should be a documented API simonw 9599 closed 0     3 2022-06-20T19:08:28Z 2022-06-20T19:49:02Z 2022-06-20T19:46:58Z OWNER  

I've used it in a couple of external places now:

  • https://github.com/simonw/datasette-socrata/blob/32fb256a461bf0e790eca10bdc7dd9d96c20f7c4/datasette_socrata/init.py#L264-L280
  • https://github.com/simonw/datasette-lite/blob/caa8eade10f0321c64f9f65c4561186f02d57c5b/webworker.js#L55-L64

Refs: - https://github.com/simonw/datasette-lite/issues/32

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/445/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1243151184 I_kwDOCGYnMM5KGPtQ 434 `detect_fts()` identifies the wrong table if tables have names that are subsets of each other ryascott 559711 closed 0     3 2022-05-20T13:28:31Z 2022-06-14T23:24:09Z 2022-06-14T23:24:09Z NONE  

Windows 10 Python 3.9.6

When I was running a full text search through the Python library, I noticed that the query was being run on a different full text search table than the one I was trying to search.

I took a look at the following function

https://github.com/simonw/sqlite-utils/blob/841ad44bacaff05ec79ef78166d12e80c82ba6d7/sqlite_utils/db.py#L2213

and noticed:

python sql LIKE '%VIRTUAL TABLE%USING FTS%content=%{table}%'

My database contains tables with similar names and %{table}% was matching another table that ended differently in its name. I have included a sample test that shows this occurring:

I search for Marsupials in db["books"] and The Clue of the Broken Blade is returned.

This occurs since the search for Marsupials was "successfully" done against db["booksb"] and rowid 1 is returned. "The Clue of the Broken Blade" has a rowid of 1 in db["books"] and this is what is returned from the search.

```python def test_fts_search_with_similar_table_names(fresh_db): db = Database(memory=True) db["books"].insert_all( [ { "title": "The Clue of the Broken Blade", "author": "Franklin W. Dixon", }, { "title": "Habits of Australian Marsupials", "author": "Marlee Hawkins", }, ] ) db["booksb"].insert( { "title": "Habits of Australian Marsupials", "author": "Marlee Hawkins", } )

db["booksb"].enable_fts(["title", "author"])
db["books"].enable_fts(["title", "author"])


query = "Marsupials"

assert [
        {   "rowid": 1,
            "title": "Habits of Australian Marsupials",
            "author": "Marlee Hawkins",
        },
    ] == list(db["books"].search(query))

```

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/434/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1223241647 I_kwDOBm6k_c5I6S-v 1734 Remove python-baseconv dependency simonw 9599 closed 0     3 2022-05-02T19:08:37Z 2022-05-02T23:25:49Z 2022-05-02T19:39:20Z OWNER  

I was going to vendor baseconv.py, but then I reconsidered - what if there are plugins out there that expect import baseconv to work because they have depended on Datasette?

I used https://cs.github.com/ and as far as I can tell there aren't any!

So I'm going to remove that dependency and work out a smarter way to do this - probably by providing a utility function within Datasette itself.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1733#issuecomment-1115258737

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1734/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1216619276 I_kwDOBm6k_c5IhCMM 1724 ?_trace=1 doesn't work on Global Power Plants demo simonw 9599 closed 0     3 2022-04-27T00:15:02Z 2022-04-27T06:15:14Z 2022-04-27T00:18:30Z OWNER  

https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_trace=1 is not showing the trace JSON at the bottom of the page.

Confirmed that trace_debug is true on https://global-power-plants.datasettes.com/-/settings

Possibly related:

  • https://github.com/simonw/datasette-total-page-time/issues/1
datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1724/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1202227104 I_kwDOBm6k_c5HqIeg 1712 Make "<Binary: 2427344 bytes>" easier to read simonw 9599 closed 0     3 2022-04-12T18:17:07Z 2022-04-12T19:12:22Z 2022-04-12T18:44:20Z OWNER  

Binary: 2,427,344 bytes would be nicer - even better, include a tooltip showing that size translated using this function: https://github.com/simonw/datasette/blob/138e4d9a53e3982137294ba383303c3a848cfca4/datasette/utils/init.py#L837-L846

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1712/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1181432624 I_kwDOBm6k_c5Gazsw 1688 [plugins][documentation] Is it possible to serve per-plugin static folders when writing one-off (single file) plugins? hydrosquall 9020979 closed 0     3 2022-03-26T01:17:44Z 2022-03-27T01:01:14Z 2022-03-26T21:34:47Z CONTRIBUTOR  

I'm trying to make a small plugin that depends on static assets, by following the guide here. I made a plugins/ directory with datasette_nteract_data_explorer.py.

I am trying to follow the example of datasette_vega, and serving static assets. I created a statics/ directory within plugins/ to serve my JS and CSS.

https://github.com/simonw/datasette-vega/blob/00de059ab1ef77394ba9f9547abfacf966c479c4/datasette_vega/init.py#L13

Unfortunately, datasette doesn't seem to be able to find my assets.

Input:

bash datasette ~/Library/Safari/History.db --plugins-dir=plugins/

Output:

I suspect this issue might go away if I move away from "one-off" plugin mode, but it's been a while since I created a new python package so I'm not sure how much work there is to go between "one off" and "packaged for PyPI". I'd like to try to avoid needing to repackage a new tar.gz file and or reinstall my library repeatedly when developing new python code.

  1. Is there a way to serve a static assets when using the plugins/ directory method instead of installing plugins as a new python package?
  2. If not, is there a way I can work on developing a plugin without creating and repackaging tar.gz files after every change, or is that the recommended path?

Thanks for your help!

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1688/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1175744654 I_kwDOCGYnMM5GFHCO 417 insert fails on JSONL with whitespace blaine 9954 closed 0     3 2022-03-21T17:58:14Z 2022-03-25T21:19:06Z 2022-03-25T21:17:13Z NONE  

Any JSON that is newline-delimited and has whitespace (newlines) between the start of a JSON object and an attribute fails due to a parse error.

e.g. given the valid JSONL:

{ "attribute": "value" } { "attribute": "value2" }

I would expect that sqlite-utils insert --nl my.db mytable file.jsonl would properly import the data into mytable. However, the following error is thrown instead:

json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)

It makes sense that since the file is intended to be newline separated, the thing being parsed is "{" (which obviously fails), however the default newline-separated output of jq isn't compact. Using jq -c avoids this problem, but the fix is unintuitive and undocumented.

Proposed solutions: 1. Default to a "loose" newline-separated parse; this could be implemented internally as [the equivalent of] a jq -c filter ahead of the insert step. 2. Catch the JSONDecodeError (or pre-empt it in the case of a record === "{\n") and give the user a "it looks like your json isn't actually newline-delimited; try running it through jq -c instead" error message.

It might just have been too early in the morning when I was playing with this, but running pipes of data through sqlite-utils without the 'knack' of it led to some false starts.

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/417/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1174302994 I_kwDOBm6k_c5F_nES 1667 Make route matched pattern groups more consistent simonw 9599 closed 0   Datasette 1.0 3268330 3 2022-03-19T16:32:35Z 2022-03-19T20:37:42Z 2022-03-19T20:37:41Z OWNER  

... highlights how inconsistent the way the capturing works is. Especially as_format which can be None or "" or .json or json or not used at all in the case of TableView.

https://github.com/simonw/datasette/blob/764738dfcb16cd98b0987d443f59d5baa9d3c332/tests/test_routes.py#L12-L36

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1666#issuecomment-1073039670

Part of: - #1660

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1667/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1174162781 I_kwDOBm6k_c5F_E1d 1666 Refactor URL routing to enable testing simonw 9599 closed 0   Datasette 1.0 3268330 3 2022-03-19T03:52:29Z 2022-03-19T16:32:03Z 2022-03-19T16:32:03Z OWNER  

I ran into some bugs earlier with URL routing - having more robust testing around this (especially since they are defined using regular expressions) would be really useful.

  • A utility function that resolves a path against a list of reflexes and returns the match
  • Make the routes and regular expressions available from a private Datasette method
  • Add tests that exercise them

Related: - #1660

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1666/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1082765654 I_kwDOBm6k_c5AibFW 1561 add hash id to "_memory" url if hashed url mode is turned on and crossdb is also turned on fgregg 536941 closed 0     3 2021-12-17T00:45:12Z 2022-03-19T04:45:40Z 2022-03-19T04:45:40Z CONTRIBUTOR  

If hashed_url mode is turned on and crossdb is also turned on, then queries to _memory should have a hash_id.

One way that it could work is to have the _memory hash be a hash of all the individual databases.

Otherwise, crossdb queries can get quit out of data if using aggressive caching.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1561/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1123903919 I_kwDOCGYnMM5C_Wmv 397 Support IF NOT EXISTS for table creation rafguns 738408 closed 0     3 2022-02-04T07:41:15Z 2022-02-06T01:30:46Z 2022-02-06T01:29:01Z NONE  

Currently, I have a bunch of code that looks like this:

python subjects = db["subjects"] if db["subjects"].exists() else db["subjects"].create({ ... }) It would be neat if sqlite-utils could simplify that by supporting CREATE TABLE IF NOT EXISTS, so that I'd be able to write, e.g.

python subjects = db["subjects"].create({...}, if_not_exists=True)

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/397/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
683805434 MDU6SXNzdWU2ODM4MDU0MzQ= 135 Code for finding SpatiaLite in the usual locations simonw 9599 closed 0     3 2020-08-21T20:15:34Z 2022-02-05T00:04:26Z 2020-08-21T20:30:13Z OWNER  

I built this for shapefile-to-sqlite but it would be useful in sqlite-utils too:

https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L16-L19

python SPATIALITE_PATHS = ( "/usr/lib/x86_64-linux-gnu/mod_spatialite.so", "/usr/local/lib/mod_spatialite.dylib", )

https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L105-L109

python def find_spatialite(): for path in SPATIALITE_PATHS: if os.path.exists(path): return path return None

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/135/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
683812642 MDU6SXNzdWU2ODM4MTI2NDI= 136 --load-extension=spatialite shortcut option simonw 9599 closed 0     3 2020-08-21T20:31:25Z 2022-02-05T00:04:26Z 2020-10-16T19:14:32Z OWNER  

In conjunction with #135 - this would do the same thing as --load-extension=path-to-spatialite (see #134)

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/136/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
534507142 MDU6SXNzdWU1MzQ1MDcxNDI= 69 Feature request: enable extensions loading aborruso 30607 closed 0     3 2019-12-08T08:06:25Z 2022-02-05T00:04:25Z 2020-10-16T18:42:49Z NONE  

Hi, it would be great to add a parameter that enables the load of a sqlite extension you need.

Something like "-ext modspatialite".

In this way your great tool would be even more comfortable and powerful.

Thank you very much

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/69/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1075893249 I_kwDOBm6k_c5AINQB 1545 Custom pages don't work on windows ryascott 559711 closed 0     3 2021-12-09T18:53:05Z 2022-02-03T02:08:31Z 2022-02-03T01:58:35Z NONE  

It seems that custom pages don't work when put in templates/pages

To reproduce on datasette version 0.59.4 using PowerShell on WIndows 10 with Python 3.10.0

mkdir -p templates/pages

echo "hello world" >> templates/pages/about.html

Start datasette

datasette --template-dir templates/

Navigate to http://127.0.0.1:8001/about and receive:

Error 404: Database not found: about

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1545/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1121618041 I_kwDOBm6k_c5C2oh5 1620 Link: rel="alternate" to JSON for queries too simonw 9599 closed 0   Datasette 1.0 3268330 3 2022-02-02T08:02:42Z 2022-02-02T21:53:02Z 2022-02-02T21:33:00Z OWNER  

Following: - #1533

I implemented it for tables and rows but I should have done queries as well.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1620/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1108846067 I_kwDOBm6k_c5CF6Xz 1606 Tests failing against Python 3.6 simonw 9599 closed 0     3 2022-01-20T04:22:44Z 2022-01-20T04:36:42Z 2022-01-20T04:36:42Z OWNER  

https://github.com/simonw/datasette/runs/4877484366

E File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/uvicorn/server.py", line 67, in run E return asyncio.run(self.serve(sockets=sockets)) E AttributeError: module 'asyncio' has no attribute 'run' I think this may mean uvicorn has dropped support for Python 3.6.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1606/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1102359726 I_kwDOBm6k_c5BtKyu 1594 Add a CLI reference page to the docs, inspired by sqlite-utils simonw 9599 closed 0   Datasette 0.60 7571612 3 2022-01-13T20:55:08Z 2022-01-13T22:28:22Z 2022-01-13T21:38:48Z OWNER  

Thought of this while posting this comment: https://github.com/simonw/datasette/issues/1591#issuecomment-1012506595

I added https://sqlite-utils.datasette.io/en/stable/cli-reference.html to sqlite-utils in https://github.com/simonw/sqlite-utils/issues/383 and I really like it - it's a page showing the --help output of every CLI command for that tool.

It's maintained using cog. One of the benefits is that I get a free commit history of changes to --help at https://github.com/simonw/sqlite-utils/commits/main/docs/cli-reference.rst

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1594/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1097101917 I_kwDOBm6k_c5BZHJd 1588 `explain query plan select` is too strict about whitespace simonw 9599 closed 0   Datasette 0.60 7571612 3 2022-01-09T04:22:42Z 2022-01-13T22:28:19Z 2022-01-13T20:35:05Z OWNER  

explain query plan select * from facetable is allowed: https://latest.datasette.io/fixtures?sql=explain+query+plan+select+*+from+facetable

But... explain query plan select * from facetable (with two spaces before the select) returns a "Statement must be a SELECT" error: https://latest.datasette.io/fixtures?sql=explain+query+plan++select+*+from+facetable

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1588/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1077893013 I_kwDOBm6k_c5AP1eV 1551 `keep_blank_values=True` when parsing `request.args` simonw 9599 closed 0   Datasette 0.60 7571612 3 2021-12-12T19:53:07Z 2022-01-13T22:26:04Z 2021-12-12T20:02:01Z OWNER  

This code in TableView wouldn't be necessary: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/views/table.py#L396-L399

If that happened here instead: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/utils/asgi.py#L98-L100

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1518#issuecomment-991827468

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1551/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1097251014 I_kwDOCGYnMM5BZrjG 375 `sqlite-utils bulk` command simonw 9599 closed 0   3.21 7558727 3 2022-01-09T17:12:38Z 2022-01-11T02:12:58Z 2022-01-11T02:10:55Z OWNER  

The .executemany() method is a very efficient way to execute the same SQL query against a huge list of parameters.

sqlite-utils insert supports a bunch of ways of loading a list of dictionaries - from CSV, TSV, JSON, newline JSON and more thanks to: - #361

What if you could load a list of dictionaries and provide a SQL query with :named parameters that correspond to keys in those dictionaries instead?

This would need to be a new command - I thought about adding a --sql option to insert but that doesn't make sense as that command already requires a table name.

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/375/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1097135732 I_kwDOCGYnMM5BZPZ0 373 List `--fmt` options in the docs simonw 9599 closed 0   3.21 7558727 3 2022-01-09T08:22:11Z 2022-01-10T19:27:24Z 2022-01-09T17:49:00Z OWNER  

https://sqlite-utils.datasette.io/en/stable/cli.html#table-formatted-output currently cheats and tells the user to run --help - can fix this using cog.

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/373/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1097087280 I_kwDOCGYnMM5BZDkw 368 Offer `python -m sqlite_utils` as an alternative to `sqlite-utils` simonw 9599 closed 0   3.21 7558727 3 2022-01-09T02:29:30Z 2022-01-10T19:27:20Z 2022-01-09T02:40:50Z OWNER  

Add this to sqlite_utils/cli.py:

python if __name__ == "__main__": cli() Now the tool can be run using python -m sqlite_utils.cli --help

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/368/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1059509927 I_kwDOBm6k_c4_Jtan 1525 "Links from other tables" broken for columns starting with underscore simonw 9599 closed 0     3 2021-11-21T22:55:08Z 2021-11-30T06:39:01Z 2021-11-30T06:34:35Z OWNER  

Same bug as #1506, this time it's this link or the row page:

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1525/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1066288689 I_kwDOBm6k_c4_jkYx 1538 Research pattern for re-registering existing Click tools with register_commands simonw 9599 closed 0     3 2021-11-29T17:09:47Z 2021-11-29T17:32:44Z 2021-11-29T17:27:16Z OWNER  

Building a Datasette plugin that imports an existing Click CLI tool and re-registers it is proving hard - Click doesn't really want you to do that. I tried this: ```python from datasette import hookimpl from git_history.cli import file as git_history_file

@hookimpl def register_commands(cli): cli.command(name="git-history")(git_history_file.callback) But when I run this: % datasette git-history --help
Usage: datasette git-history [OPTIONS]

Analyze the history of a specific file and write it to SQLite

Options: --help Show this message and exit. ``` The options are all missing - which means that the command doesn't actually work. Will need to research this pattern separately.

Originally posted by @simonw in https://github.com/simonw/git-history/issues/21#issuecomment-981835305

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1538/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
964400482 MDU6SXNzdWU5NjQ0MDA0ODI= 310 `sqlite-utils insert --flatten` option to flatten nested JSON simonw 9599 closed 0     3 2021-08-09T21:23:08Z 2021-10-16T13:54:56Z 2021-08-09T21:44:06Z OWNER  

I had to do this with a jq recipe today: https://til.simonwillison.net/cloudrun/tailing-cloud-run-request-logs

cat log.json | jq -c '[leaf_paths as $path | { "key": $path | join("_"), "value": getpath($path) }] | from_entries' \ | sqlite-utils insert /tmp/logs.db logs - --nl --alter --batch-size 1 That was to turn something like this: json { "httpRequest": { "latency": "0.112114537s", "requestMethod": "GET", "requestSize": "534", "status": 200, }, "insertId": "6111722f000b5b4c4d4071e2", "labels": { "service": "datasette-io" } } Into this instead: json { "httpRequest_latency": "0.112114537s", "httpRequest_requestMethod": "GET", "httpRequest_requestSize": "534", "httpRequest_status": 200, "insertId": "6111722f000b5b4c4d4071e2", "labels_service": "datasette-io" } I have to do this often enough that I think it should be an option, --flatten - so I can do this instead: cat log.json | sqlite-utils insert /tmp/logs.db logs - --flatten

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/310/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1004613267 I_kwDOCGYnMM474S6T 328 Invalid JSON output when no rows gravis 12752 closed 0     3 2021-09-22T18:37:26Z 2021-09-22T20:21:34Z 2021-09-22T20:20:18Z NONE  

sqlite-utils query generates a JSON output with the result from the query:

json [{...},{...}] If no rows are returned by the query, I'm expecting an empty JSON array:

json []

But actually I'm getting an empty string. To be consistent, the output should be [] when the request succeeds (return code == 0).

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/328/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
989986586 MDU6SXNzdWU5ODk5ODY1ODY= 1461 Try blacken-docs simonw 9599 closed 0     3 2021-09-07T13:28:50Z 2021-09-07T16:13:59Z 2021-09-07T16:13:59Z OWNER  

https://github.com/asottile/blacken-docs

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1461/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
978357984 MDU6SXNzdWU5NzgzNTc5ODQ= 1446 Modify base.html template to support optional sticky footer simonw 9599 closed 0     3 2021-08-24T18:11:12Z 2021-08-31T01:54:59Z 2021-08-24T20:32:47Z OWNER  

The neatest way to have the footer stick to the bottom of the browser window that I've found is to use the flexbox pattern from https://css-tricks.com/couple-takes-sticky-footer/

```html

<body>
content
<footer class="footer"></footer> </body>

css html, body { height: 100%; } body { display: flex; flex-direction: column; } .content { flex: 1 0 auto; } .footer { flex-shrink: 0; } `` I tried this in a custom plugin but it ended up having to duplicate the entirebase.html` template just to get a wrapper around the not-footer content.

I think Datasette's own base.html template should have this wrapper element instead.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1446/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
913135723 MDU6SXNzdWU5MTMxMzU3MjM= 266 Add some types, enforce with mypy simonw 9599 closed 0     3 2021-06-07T06:05:56Z 2021-08-18T22:25:38Z 2021-08-18T22:25:38Z OWNER  

A good starting point would be adding type information to the members of these named tuples and the introspection methods that return them:

https://github.com/simonw/sqlite-utils/blob/9dff7a38831d471b1dff16d40d89eb5c3b4e84d6/sqlite_utils/db.py#L51-L75

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/266/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
957731178 MDU6SXNzdWU5NTc3MzExNzg= 304 `table.convert(..., where=)` and `sqlite-utils convert ... --where=` simonw 9599 closed 0     3 2021-08-02T04:27:23Z 2021-08-02T19:00:00Z 2021-08-02T18:58:10Z OWNER  

For applying the conversion to a subset of rows selected using the where clause.

Should also take optional arguments, as seen in db["dogs"].delete_where("age < ?", [3]).

Follows #302 and #251. This was originally https://github.com/simonw/sqlite-transform/issues/9

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/304/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
955316250 MDU6SXNzdWU5NTUzMTYyNTA= 1405 utils.parse_metadata() should be a documented internal function simonw 9599 closed 0     3 2021-07-28T23:51:39Z 2021-07-29T23:33:30Z 2021-07-29T23:30:24Z OWNER  

Because it's used by this plugin: https://github.com/simonw/datasette-remote-metadata

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1405/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
612673948 MDU6SXNzdWU2MTI2NzM5NDg= 759 fts search on a column doesn't work anymore due to escape_fts Krazybug 133845 closed 0     3 2020-05-05T15:03:44Z 2021-07-16T02:11:54Z 2020-05-06T17:50:57Z NONE  

Hi and first, thank you for this awesome work you make with this projet.

On a db indexed in full text search, I can't query on indexed column anymore.

This request "cauvin language:ita": is running smoothly on a old version of datasette but not on the current version.

Compare the current version query select uuid, title, authors, year, series, language, formats, publisher, tags, identifiers from summary where rowid in (select rowid from summary_fts where summary_fts match escape_fts(:search)) order by uuid limit 101

To an older version:

select title, authors, series, uuid, language, identifiers, tags, publisher, formats, year, links from summary where rowid in (select rowid from summary_fts where summary_fts match :search) order by uuid limit 101

language is a searchable column but now the search string is known as "cauvin language:ita" literally as a search term. columns are not parsed.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/759/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
539590148 MDU6SXNzdWU1Mzk1OTAxNDg= 651 fts5 syntax error when using punctuation clausjuhl 2181410 closed 0     3 2019-12-18T10:25:35Z 2021-07-14T19:26:06Z 2019-12-30T06:42:55Z NONE  

Hi Simon

I get a syntax error when using punctuation or special characters in a fulltext search (using fts5). I created the virtual table using sqlite-utils' "enable-fts"-command.

The same error appears on Niche Museums https://www.niche-museums.com/browse/search?q=park., but works fine in most of your other datasette-examples, e.g. register-of-members-interests https://register-of-members-interests.datasettes.com/regmem-98dc8b7/items?_search=mins.

What am I doing wrong? Many thanks!

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/651/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
920884085 MDU6SXNzdWU5MjA4ODQwODU= 1377 Mechanism for plugins to exclude certain paths from CSRF checks simonw 9599 closed 0     3 2021-06-15T00:48:20Z 2021-06-23T22:51:33Z 2021-06-23T22:51:33Z OWNER  

I need this for a plugin I'm building that offers a POST API.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1377/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
925677191 MDU6SXNzdWU5MjU2NzcxOTE= 289 Mypy fixes for rows_from_file() adamchainz 857609 closed 0     3 2021-06-20T20:34:59Z 2021-06-22T18:44:36Z 2021-06-22T18:13:26Z NONE  

Following https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864328927

You had two mypy errors.

The first:

sqlite_utils/utils.py:157: error: Argument 1 to "BufferedReader" has incompatible type "BinaryIO"; expected "RawIOBase"

Looking at the BufferedReader docs, it seems to expect a RawIOBase, and this has been copied into typeshed. There may be scope to change how BufferedReader is documented and typed upstream, but for now it wouldn't be too bad to use a typing.cast():

```

Detect the format, then call this recursively

buffered = io.BufferedReader( cast(io.RawIOBase, fp), # Undocumented BufferedReader support for BinaryIO buffer_size=4096, ) ```

The second error seems to be flagging a legitimate bug in your code:

sqlite_utils/utils.py:163: error: Argument 1 to "decode" of "bytes" has incompatible type "Optional[str]"; expected "str"

From your type hints, encoding may be None. In the CSV format block, you use encoding or "utf-8-sig" to set a default, maybe that's desirable in this case too?

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/289/reactions",
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
923697888 MDU6SXNzdWU5MjM2OTc4ODg= 278 Support db as first parameter before subcommand, or as environment variable mcint 601708 closed 0     3 2021-06-17T09:26:29Z 2021-06-20T22:39:57Z 2021-06-18T15:43:19Z CONTRIBUTOR  
sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/278/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
732685643 MDU6SXNzdWU3MzI2ODU2NDM= 1063 .csv should link to .blob downloads simonw 9599 closed 0   0.51 6026070 3 2020-10-29T21:45:58Z 2021-06-17T18:12:30Z 2020-10-29T22:47:45Z OWNER  
  • [x] Update .csv output to link to these things (and get that xfail test to pass)
  • ~~Add a .csv?_blob_base64=1 argument that causes them to be output in base64 in the CSV~~

Moving the CSV work to a separate ticket. Originally posted by @simonw in https://github.com/simonw/datasette/pull/1061#issuecomment-719042601

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1063/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
919250621 MDU6SXNzdWU5MTkyNTA2MjE= 269 bool type not supported frafra 4068 closed 0     3 2021-06-11T22:00:36Z 2021-06-15T01:34:10Z 2021-06-15T01:34:10Z NONE  

Hi! Thank you for sharing this very nice tool :) It would be nice to have support for more types, like bool: it is not possible to convert to boolean at the moment. My suggestion would be to handle it as bool(int(value)), like csvkit does.

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/269/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
912418094 MDU6SXNzdWU5MTI0MTgwOTQ= 1358 Release Datasette 0.57 simonw 9599 closed 0     3 2021-06-05T19:56:13Z 2021-06-05T22:20:07Z 2021-06-05T22:20:07Z OWNER  

Need release notes. Changes are here: https://github.com/simonw/datasette/compare/0.56...368aa5f1b16ca35f82d90ff747023b9a2bfa27c1

Partial release notes already exist for the two alphas, https://github.com/simonw/datasette/releases/tag/0.57a0 and https://github.com/simonw/datasette/releases/tag/0.57a1

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1358/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
907642546 MDU6SXNzdWU5MDc2NDI1NDY= 264 Supporting additional output formats, like GeoJSON eyeseast 25778 closed 0     3 2021-05-31T18:03:32Z 2021-06-03T05:12:21Z 2021-06-03T05:12:21Z CONTRIBUTOR  

I have a project going where it would be useful to do some spatial processing in SQLite (instead of large files) and then output GeoJSON. So my workflow would be something like this:

  1. Read Shapefiles, GeoJSON, CSVs into a SQLite database
  2. Join, filter, prune as needed
  3. Export GeoJSON for just the stuff I need at that moment, while still having a database of things that will be useful later

I'm wondering if this is worth adding to SQLite-utils itself (GeoJSON, at least), or if it's better to make a counterpart to the ecosystem of *-to-sqlite tools, say a suite of sqlite-to-* things. Or would it be crazy to have a plugin system?

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/264/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
323671577 MDU6SXNzdWUzMjM2NzE1Nzc= 263 Facets should not execute for ?shape=array|object simonw 9599 closed 0     3 2018-05-16T15:26:13Z 2021-06-02T02:54:34Z 2021-06-02T02:54:34Z OWNER  

Split off from #255 - there's no point executing the facet SQL for the ?_shape=array and ?_shape=object API responses.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/263/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
838148087 MDU6SXNzdWU4MzgxNDgwODc= 250 Handle byte order marks (BOMs) in CSV files simonw 9599 closed 0     3 2021-03-22T22:13:18Z 2021-05-29T05:34:21Z 2021-05-29T05:34:21Z OWNER  

I often find sqlite-utils insert ... --csv creates a first column with a weird character at the start of it - which it turns out is the UTF-8 BOM. Fix that.

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/250/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
866668415 MDU6SXNzdWU4NjY2Njg0MTU= 1308 Columns named "link" display in bold simonw 9599 closed 0     3 2021-04-24T05:58:11Z 2021-04-24T06:07:49Z 2021-04-24T06:07:49Z OWNER  

Reported in office hours today.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1308/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
453131917 MDU6SXNzdWU0NTMxMzE5MTc= 502 Exporting sqlite database(s)? chrismp 7936571 closed 0     3 2019-06-06T16:39:53Z 2021-04-03T05:16:54Z 2019-06-11T18:50:42Z NONE  

I'm working on datasette from one computer. But if I want to work on it from another computer and want to copy the SQLite database(s) already on the Heroku datasette instance, how to I copy the database(s) to the second computer so that I can then update it and push to online via datasette's command line code that pushes code to Heroku?

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/502/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
849543502 MDU6SXNzdWU4NDk1NDM1MDI= 1289 Speed up tests with pytest-xdist simonw 9599 closed 0     3 2021-04-03T00:47:39Z 2021-04-03T03:42:28Z 2021-04-03T03:42:28Z OWNER  

I think I can get this working for almost every test, then use the pattern in https://github.com/pytest-dev/pytest-xdist/issues/385#issuecomment-444545641 to opt specific tests out of being run in parallel.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1289/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
280013907 MDU6SXNzdWUyODAwMTM5MDc= 164 datasette skeleton command for kick-starting database and table metadata simonw 9599 closed 0   Custom templates edition 2949431 3 2017-12-07T06:13:28Z 2021-03-23T02:45:12Z 2017-12-07T06:20:45Z OWNER  

Generates an example metadata.json file populated with all of the databases and tables inspected from the specified databases.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/164/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
832092321 MDU6SXNzdWU4MzIwOTIzMjE= 1261 Some links aren't properly URL encoded. brimstone 812795 closed 0     3 2021-03-15T18:43:59Z 2021-03-21T02:06:44Z 2021-03-20T21:36:06Z NONE  

It seems like a percent sign in the query causes some links to end invalid.

The json and CSV links on this page don't behave like expected: https://honeypot-brimston3.vercel.app/honeypot?sql=select+time%2C+count%28time%29+as+count+from+%28select+strftime%28%22%25Y-%25m-%25d%22%2C+_etime%29+as+time+from+ssh+%29+group+by+time+order+by+time%3B

I can take a swing at trying to fix this, but my python isn't strong and I need a pointer at the right approach and files to change.

Thanks!

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1261/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
836123030 MDU6SXNzdWU4MzYxMjMwMzA= 1265 Support for HTTP Basic Authentication yunzheng 468612 closed 0     3 2021-03-19T15:31:09Z 2021-03-19T22:05:12Z 2021-03-19T21:03:09Z NONE  

It would be nice if datasette could support HTTP Basic Authentication.

For now I could ofcourse leverage Nginx for basic authentication, but it would be nice to have support for this in datasette by default or via a plugin like datasette-auth-github.

My main usecase is to put the whole datasette instance behind a username/password prompt via Basic Auth and not specific urls.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1265/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
817597268 MDU6SXNzdWU4MTc1OTcyNjg= 1246 Suggest for ArrayFacet possibly confused by blank values simonw 9599 closed 0     3 2021-02-26T19:11:52Z 2021-03-01T03:46:11Z 2021-03-01T03:46:11Z OWNER  

I sometimes don't get the suggestion for facet-by-array for columns that contain arrays. I think it may be because they have empty spaces in them - or perhaps it's because the null detection doesn't actually work.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1246/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
718259202 MDU6SXNzdWU3MTgyNTkyMDI= 1005 Remove xfail tests when new httpx is released simonw 9599 closed 0   Datasette 1.0 3268330 3 2020-10-09T16:00:19Z 2021-02-28T22:41:08Z 2021-02-28T22:41:08Z OWNER  

My httpx pull request adding raw_path support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet.

I'm going to mark these tests as xfail so I can land this change - I'll remove that once an httpx release comes out that I can use to get the tests passing.

Originally posted by @simonw in https://github.com/simonw/datasette/pull/1000#issuecomment-706263157

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1005/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
814591962 MDU6SXNzdWU4MTQ1OTE5NjI= 1240 Allow facetting on custom queries Kabouik 7107523 closed 0     3 2021-02-23T15:52:19Z 2021-02-26T18:19:46Z 2021-02-26T18:18:18Z NONE  

Facets are a tremendously useful feature, especially for people peeking at the database for the first time and still having little knowledge about the details of the data. It is of great assistance to discover interesting features to explore futher in advanced queries.

Yet, it seems it's impossible to use facets when running a custom SQL query, be it from the little gear icons in column names, the facet suggestions at the top (hidden when performing a custom query), or by appending a facet code to the URL.

Is there a technical limitation, or is this something that could be unlocked easily?

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1240/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
815554385 MDU6SXNzdWU4MTU1NTQzODU= 237 db["my_table"].drop(ignore=True) parameter, plus sqlite-utils drop-table --ignore and drop-view --ignore mhalle 649467 closed 0     3 2021-02-24T14:55:06Z 2021-02-25T17:11:41Z 2021-02-25T17:11:41Z NONE  

When I'm generating a derived table in python, I often drop the table and create it from scratch. However, the first time I generate the table, it doesn't exist, so the drop raises an exception. That means more boilerplate.

I was going to submit a pull request that adds an "if_exists" option to the drop method of tables and views.

However, for a utility like sqlite_utils, perhaps the "IF EXISTS" SQL semantics is what you want most of the time, and thus should be the default.

What do you think?

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/237/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
783778672 MDU6SXNzdWU3ODM3Nzg2NzI= 220 Better error message for *_fts methods against views mhalle 649467 closed 0     3 2021-01-11T23:24:00Z 2021-02-22T20:44:51Z 2021-02-14T22:34:26Z NONE  

enable_fts and its related methods only work on tables, not views.

Could those methods and possibly others move up to the Queryable superclass?

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/220/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
807817197 MDU6SXNzdWU4MDc4MTcxOTc= 229 Hitting `_csv.Error: field larger than field limit (131072)` frosencrantz 631242 closed 0     3 2021-02-13T19:52:44Z 2021-02-14T21:33:33Z 2021-02-14T21:33:33Z NONE  

I have a csv file where one of the fields is so large it is throwing an exception with this error and stops loading: _csv.Error: field larger than field limit (131072)

The stack trace occurs here: https://github.com/simonw/sqlite-utils/blob/3.1/sqlite_utils/cli.py#L633

There is a way to handle this that helps: https://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072

One issue I had with this problem was sqlite-utils only provides limited context as to where the problem line is. There is the progress bar, but that is by percent rather than by line number. It would have been helpful if it could have provided a line number.

Also, it would have been useful if it had allowed the loading to continue with later lines.

sqlite-utils 140912432 issue    
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/229/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
796736607 MDU6SXNzdWU3OTY3MzY2MDc= 56 Not all quoted statuses get fetched? gsajko 42315895 closed 0     3 2021-01-29T09:48:44Z 2021-02-03T10:36:36Z 2021-02-03T10:36:36Z NONE  

In my database I have 13300 quote tweets, but eta 3600 have quoted_status empty.

I fetched some of them using https://api.twitter.com/1.1/statuses/show.json?id=xx and they did have ids of quoted tweets.

twitter-to-sqlite 206156866 issue    
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
799693777 MDU6SXNzdWU3OTk2OTM3Nzc= 1214 Re-submitting filter form duplicates _x querystring arguments simonw 9599 closed 0     3 2021-02-02T21:13:35Z 2021-02-02T21:28:53Z 2021-02-02T21:21:13Z OWNER  

Really nasty bug, caused by #1194 fix in 07e163561592c743e4117f72102fcd350a600909

Navigate to this page: https://github-to-sqlite.dogsheep.net/github/labels?_search=help&_sort=id

Click "Apply" to submit the form and the resulting URL is https://github-to-sqlite.dogsheep.net/github/labels?_search=help&_sort=id&_search=help&_sort=id

That's because the (truncated) HTML for the form looks like this:

html ... <input id="_search" type="search" name="_search" value="help"> ... <div class="select-wrapper small-screen-only"> <select name="_sort" id="sort_by"> <option value="">Sort...</option> <option value="id" selected>Sort by id</option> <option value="node_id">Sort by node_id</option> ... </select> </div> ... <input type="hidden" name="_search" value="help"> <input type="hidden" name="_sort" value="id"> <input type="submit" value="Apply">

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1214/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
793027837 MDU6SXNzdWU3OTMwMjc4Mzc= 1205 Rename /:memory: to /_memory simonw 9599 closed 0   Datasette 1.0 3268330 3 2021-01-25T05:04:56Z 2021-01-28T22:55:02Z 2021-01-28T22:51:42Z OWNER  

For consistency with /_internal - and because then we don't need to escape the : characters.

This change would need to be in before Datasette 1.0. I could land it earlier and set up redirects from the old URLs though.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1205/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
788447787 MDU6SXNzdWU3ODg0NDc3ODc= 1194 ?_size= argument is not persisted by hidden form fields in the table filters simonw 9599 closed 0   Datasette 0.54 6346396 3 2021-01-18T17:41:52Z 2021-01-25T03:10:23Z 2021-01-25T03:10:23Z OWNER  

Click "Apply" on https://covid-19.datasettes.com/covid/ny_times_us_counties?_size=1000&county__exact=San+Francisco&state__exact=California&_sort_desc=date#g.mark=line&g.x_column=date&g.x_type=temporal&g.y_column=cases&g.y_type=quantitative and the ?_size=1000 parameter from the URL will no longer apply on the reloaded page.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1194/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
777145954 MDU6SXNzdWU3NzcxNDU5NTQ= 1167 Add Prettier to contributing documentation simonw 9599 closed 0   Datasette 0.54 6346396 3 2020-12-31T22:00:55Z 2021-01-25T02:01:19Z 2021-01-25T01:58:28Z OWNER  

Following #1166 - the docs at https://docs.datasette.io/en/stable/contributing.html should include a section about JavaScript, and it should document how to run Prettier.

I run it in VS Code but it can be run on the command-line too:

npx prettier 'datasette/static/*[!.min].js' --write
datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1167/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);
Powered by Datasette · Queries took 1640.456ms · About: github-to-sqlite
  • Sort ascending
  • Sort descending
  • Facet by this
  • Hide this column
  • Show all columns
  • Show not-blank rows