issues

In: - https://github.com/simonw/datasette.io/issues/152#issuecomment-1712323347

This turned out to be the fix:

bash dogsheep-beta index dogsheep-index.db templates/dogsheep-beta.yml sqlite-utils rebuild-fts dogsheep-index.db

When creating a dogsheep index from config.yml file on pocket.db (created using pocket-to-sqlite), I am getting this error

Traceback (most recent call last): File "/Users/khushmeeet/.pyenv/versions/3.11.2/bin/dogsheep-beta", line 8, in <module> sys.exit(cli()) ^^^^^ File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/cli.py", line 36, in index run_indexer( File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py", line 32, in run_indexer ensure_table_and_indexes(db, tokenize) File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py", line 91, in ensure_table_and_indexes table.add_foreign_key(*fk) File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py", line 2155, in add_foreign_key self.db.add_foreign_keys([(self.name, column, other_table, other_column)]) File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py", line 1116, in add_foreign_keys cursor.execute( sqlite3.OperationalError: table sqlite_master may not be modified

Command I ran to get this error dogsheep-beta index pocket.db config.yml

Dogsheep version dogsheep-beta, version 0.10.2

Python version Python 3.11.2

Hi,

First of all, thank you for this wonderful project :smile:

I started to use dogsheep to make my personal data searchable, and by using the project I noticed an issue with the index command.

It always expects you are running it from the root folder from where the data is located, so I got some errors while trying to make it work on my setup.

I separate all databases inside a data folder (I published my setup to be easier to follow: https://github.com/humrochagf/my-dogsheep)

Before, I configured dogsheep.yml to add the data folder to its path like this:

yml data/twitter.db: tweets: sql: |- ...

And running the index command like this:

dogsheep-beta index data/dogsheep.db dogsheep.yml

It worked to the normal search feature with no problem this way, but when I started adding display_sql rules the app started to crash, because at datasette get_database it was looking for data/twitter and it only had a db called twitter there.

So my workaround to that was to cd into the data folder and run the indexer. You can check the way I'm doing it at this line of the makefile: https://github.com/humrochagf/my-dogsheep/blob/main/makefile#L3

It works but it would be nice to have an option to pass the path where the data is located to the index function.

Like on https://til.simonwillison.net/til/search?q=Snippet

Refs https://github.com/simonw/datasette/issues/878

A link to /-/beta?category=1&timestamp__date=2018-08-01&q=swedish should be to /-/beta?category=1&timestamp__date=2018-08-01&q=swedish&sort=newest

Useful for #16 (timeline view) since you can now filter to just the items on a specific day - but if there are more than 50 items you can't see them all.

Ability to browse (and facet) by date.

I can bring these over from my personal Dogsheep.

quantified-self habit-tracking google-fit time-tracking wearables quantifiedself for example

As tables get bigger reindexing everything on a schedule (essentially recreating the entire index from scratch) will start to become a performance bottleneck.

Here's why: https://github.com/dogsheep/dogsheep-beta/blob/24f7898d41a39218058f174c75ba62f7c0fcfff6/dogsheep_beta/utils.py#L44-L53

That should probably do DELETE FROM index1.search_index WHERE [table] = ? first.

It would be cool if the display_sql could return a column populated with GeoJSON which would the automatically be displayed on a map in the results (or maybe default JS would look for a class="geojson" element output by the display template) - ala https://github.com/simonw/datasette-leaflet-geojson

Then I could render workout routes on a map, or Swarm checkin points.

Once I've got the Datasette plugin to a state where it's worth building a demo: #3

I can use data from my public https://github-to-sqlite.dogsheep.net/ demo plus the Pocket data subset I use for the demo in https://github.com/dogsheep/pocket-to-sqlite/issues/5 - I could pull in the https://dogsheep-photos.dogsheep.net/ photos data too.