issues

1. My https://list.saferdisinfectants.org/disinfectants/listN page (linked from https://SaferDisinfectants.org ) has been running beautifully for a year and a half, including a GitHub Actions workflow that's been routinely updating the database.

2. I received a recent report that the list page is down. I don't know when it went down, but the content is replaced with: "render_template() called before await ds.invoke_startup()"

3. The local datasette repo runs without incident.

4. The site is hosted on vercel, linked to my github repo. Perhaps some vercel changes were made, but not by anyone on our side. Here is a screenshot of the current project settings:

Here a screenshot of the latest deployment status:

This is my repository: https://github.com/mroswell/list-N (I notice: datasette==0.59 in my requirements.txt file)

Because it's been long while since I actively worked on this or any other datasette project, I forget a lot of what I knew at one point. Perhaps some configuration file could be missing? Or perhaps I just need to know the right incantation to add to that vercel settings page.

Help is welcome as the nonprofit org is soon hosting its annual conference, and we'd love to have the page working again.

Wide tables aren't a huge problem if you know to click and drag right. But it's not at all obvious to do that. (it also tends to blue-select any content as it's dragging.) Depending on column widths, public users might entirely miss all the columns to the right.

There is a scrollbar at the bottom of the table, but I'm displaying ALL my records because it's the only way for datasette-vega to make accurate charts. So that bottom scrollbar is likely to be missed. I wonder if some sort of javascript-y mouseover to an arrow might help, similar to those seen in image carousels. Ah: here's a perfect example:

1. Visit http://google.com
2. Search for: animals endangered
3. Note the 'g-right-button' (in the code) that looks like a right-facing caret in a circle.
4. Click on that and the carousel scrolls right (and 'g-left-button' appears on the left).

Might be tricky to do that on a table, rather than a one-row carousel, but it's worth experimenting with.

Another option is just to put the scrollbars at the top of the table, too.

Meantime, I'm trying to build a button like the "View/hide all columns on https://salaries.news.baltimoresun.com/salaries-be494cf/2019+Maryland+state+salaries Might be nice to have that available by default, with settings in the metadata showing which are on by default.

(I saw some other closed issues related to horizontal scrolling, and admit I don't entirely understand them. For instance, the animated gif at https://github.com/simonw/datasette/issues/998#issuecomment-714117534 confuses me. )

I'd like to be able to add to vercel.json. But Datasette overwrites whatever I put in that file. I originally reported this here: https://github.com/simonw/datasette-publish-vercel/issues/51

In that case, I wanted to do a rewrite... and now I need to do 301 redirects (because we had to rename our site).

Can this be addressed?

It would be great to have a sample showing how to move a single database that has a single table, to the index page. I'm trying it now, and find there is a real depth of Datasette and Python understanding that's required to be successful.

I've got all the basic jinja concepts down... variables, template control structures, template inheritance, template overrides, css, html, the --template-dir and --static arguments, etc.

But copying the table.html file to index.html doesn't work. There are undocumented functions and filters... I can figure some of them out (yay, url_builder.py and utils/init.py!) but it's a slog better handled by a much stronger Python developer.

One sample would make a world of difference. The ideal form of this documentation would be a diff between the default table.html and how that would look if essentially moved to index.html. The use case is for everyone who wants to create a public-facing website to explore a single table at the root directory. (Maybe a second bit of documentation for people who have a single database with multiple tables.)

(Hmm... might be cool to have a setting for that, where it happens automagically! If only one table, then home page is at the table level. if only one database, then home page is at the database level.... as an option.)

I suppose I could ignore this, and somehow do this in the DNS settings once I hook up Vercel to a domain name, maybe.. and remove the breadcrumbs in table.html... but for now, a documentation request in the form of a diff... for viewing a single table (or a single database) at the root.

(Actually, there's probably room for a whole expanded section on templates. Noticed some nice table metadata in one of the datasette examples, for instance... Hmm... maybe a whole library of solutions in one place... maybe a documentation hackathon! If that's of interest, of course it's a separate issue. )

My data already has an ID that comes from a federal agency. Would love to have documentation on how to modify the template to: - Remove the generated ID from the table - Link the federal ID to the detail page - and to ensure that the JSON file uses that as the ID. I'd be happy to include the database ID in the export, but not as a key.

I don't want to remove the ID from the database, though, because my experience with the federal agency is that data often has anomalies. I don't want all hell to break loose if they end up applying the same ID to multiple rows (which they haven't done yet). I just don't want it to display in the table or the data exports.

Perhaps this isn't a template issue, maybe more of a db manipulation...

Margie

I have two connected sites: http://www.SaferOrToxic.org (a Hugo website) and: http://disinfectants.SaferOrToxic.org/disinfectants/listN (a datasette table page)

The latter is linked as "The List" in the former's menu. (I'd love a prettier URL, but that's what I've got.)

On: http://disinfectants.SaferOrToxic.org/disinfectants/listN ... all the other menu links should point back to: https://www.SaferOrToxic.org And they do!

But the logo, for some reason--though it has an href pointing to: https://www.SaferOrToxic.org Keeps going to this instead: https://disinfectants.saferortoxic.org/disinfectants/undefined

What is causing that? How can I fix it?

In #1284 back in March, I was doing battle with the index.html template, in a still unresolved issue. (I wanted only a single table page at the root.)

But I thought, well, if I can't resolve that, at least I could just point the main website to the datasette page ("The List,") and then have the List point back to the home website.

The menu hrefs to https://www.SaferOrToxic.org work just fine, exactly as they should, from the datasette page. Even the Home link works properly.

But the logo link keeps rewriting to: https://disinfectants.saferortoxic.org/disinfectants/undefined

This is the HTML: <a class="text-3xl font-bold leading-none" href="https://www.saferortoxic.org"><img src="https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp" class="custom-logo" alt="Logo: Safer or Toxic?" width="300px"></a>

Is this somehow related to cloudflare? Or something in the datasette code?

I'm starting to think it's a cloudflare issue.

Can I at least rule out it being a datasette issue?

My repository is here: https://github.com/mroswell/list-N

(BTW, I couldn't figure out how to reference a local image, either, on the datasette side, which is why I'm using the image from the www home page.)

Would be great to have template filters that convert array fields to bullets and/or delimited lists upon table display: |to_bullets |to_comma_delimited |to_semicolon_delimited or maybe: |join_array("bullet") |join_array("bullet","square") |join_array(";") |join_array(",") Keeping in mind that bullets show up in html as \<li> while other delimiting characters appear after the value.

Of course, the fields themselves would remain as facetable arrays.

Is there a way to add a parameter to the URL to modify default_facet_size?

LIkewise, a way to produce a link on the three dots to expand to all items (or match previous number of items, or add x more)?

1. Clone https://github.com/mroswell/list-N
2. Run datasette disinfectants.db -o
3. Select the Safer_or_Toxic facet.
4. Select Toxic.
5. Close out the Safer_or_Toxic facet.
6. Examine Suggested facets list. Safer_or_Toxic is GONE.
7. Try some other facets. When you select an element, and then close the list, in some cases, the facet properly returns to the Suggested facet list... Arrays and dates properly return to the list, but fields with strings don't return to the list.

Since my site is devoted to whether disinfectants are Safer or Toxic, having the suggested facet disappear from the suggested facet list is very confusing* to end-users. This, along with a few other issues, unfortunately proved beyond my own programming ability to address. So I hired a Senior-level developer to address a number of issues, including this disappearing act.

1. Open a new terminal. Run datasette disinfectants.db -m metadata.json --static static:static/ --template-dir templates/ --plugins-dir plugins/ -p 8001 -o
2. Repeat steps 3-6, but this time, the Safer_or_Toxic facet returns to the list (and the related URL parameters are removed).

I'm not sure how to do a pull request for this, because the plugin contains other functionality that goes beyond this bug. I wanted the facets sorted in a certain order (both in the suggested facet list, and the detail lists) (... the detail lists were hopping around all over the place before...) I wanted the duplicate facets removed (leaving only the one where you can facet by individual item in an array.) I wanted the arrays to be presented in a prettier fashion (I did that in the template... That could be moved over to the plugin at some point)

I'm thinking it'll be very helpful if applicable parts of my project's plugin (sort_suggested_facets_plugin.py) will be able to be incorporated back into datasette, but I leave that to you to consider.

(* The disappearing facet bug was especially confusing because I'm removing the filters and sql from the table page, at the request of the organization. The filters and sql detail created a lot of confusion for end users who try to find disinfectants used by Hospitals, for instance, as an '=' won't find them, since they are part of the Use_site array.) My disappearing-facet confusion was documented in my own issue: https://github.com/mroswell/list-N/issues/57 (addressed by the plugin). Other facet-related issues here: https://github.com/mroswell/list-N/issues/54 (addressed by the plugin); https://github.com/mroswell/list-N/issues/15 (addressed by template); https://github.com/mroswell/list-N/issues/53 (not yet addressed).

https://docs.datasette.io/en/stable/facets.html notes that:

If your SQLite installation provides the json1 extension (you can check using /-/versions) Datasette will automatically detect columns that contain JSON arrays...

When I check -/versions I see two sections relevant to json1: "extensions": { "json1": null }, "compile_options": [ ... "ENABLE_JSON1",

The ENABLE_JSON1 makes me think json1 is likely available. But the "json1": null made me think it wasn't available (because of the null). It would help if the documentation provided clarity about how to know if json1 is installed. It would also be helpful if the /-/versions information signalled somehow that that is to be appended to the hostname or domain name (or whatever you want to call it, or simply show it, using example.com/-/versions instead of /-/versions. Likewise on that last point, for https://docs.datasette.io/en/stable/introspection.html#introspection , at least at some point on that page detailing where those introspection endpoints go. (Sometimes documentation can be so abbreviated that it's hard for new users to figure out what's going on.)

(Feel free to rename this one.)

• The column gear lets you "Show not-blank rows." Then it places a parameter in the URL, which a web developer would notice, but a lot of users won't notice, or know to delete it. Would be good to toggle "Show not-blank rows" with "Show all rows." (Also would be quite helpful to have a "Show blank rows | Show all rows" option)
• The column gear lets you "Sort ascending" and "Sort descending" but then you're stuck with some sort of sorted version thereafter, unless you know to sort the ID column, or to remove the full _sort parameter and its value in the URL. Would be good to offer a "Remove sort" option in the gear.
• These requests are in the same camp as: https://github.com/simonw/datasette-vega/issues/36
• I suspect there are other url parameter instances where similar analysis would be helpful, but the three above are the use cases I've run across.

UPDATE: - It would be helpful to have a "Previous page" available for all but the first table page.

It would be great to offer facets for numeric data ranges.

The ranges could pull from typical GIS methods of creating choropleth maps. https://gisgeography.com/choropleth-maps-data-classification/ Of the following, for mapping, I've always preferred a Jenks Natural Breaks, or a cross between Jenks and Pretty breaks.

• Equal Intervals
• Quantile (equal count)
• Standard Deviation
• Natural Breaks (Jenks) Classification
• Pretty Breaks
• Some sort of Aggregate Jenks Classification (this isn't standard, but it would be nice to be able to set classification ranges that work across tables.)

Here are some links for Natural Breaks, in case this method is unfamiliar.

• https://en.wikipedia.org/wiki/Jenks_natural_breaks_optimization
• http://wiki.gis.com/wiki/index.php/Jenks_Natural_Breaks_Classification
• https://medium.com/analytics-vidhya/jenks-natural-breaks-best-range-finder-algorithm-8d1907192051

Per that last link, there is a Jenks Python module... They also describe it as data-intensive for larger datasets. Maybe this is a good plugin idea.

An example of equal Intervals would be 0 – < 10 10 – < 20 20 – < 30 30 – < 40

It's kind of confusing to have that less-than sign in there. it could also be displayed as: 0 – 10 10 – 20 20 – 30 30 – 40

But then it's not completely clear which category 10 is in, for instance.

(Best to right-justify.. and use an "en dash" between numbers.)

1. Visit a datasette table page
2. Click on the "(advanced)" link. This adds a fragment identifier "#export" to the URL, and scrolls down to the "Advanced export" div with the "export" id.
3. Manually scroll back up, and click on a suggested facet. The fragment identifier is still present, and the app scrolls back down to the "Advanced export" div. I think this is unwanted behavior.

The user remedy seems to be to manually remove the "#export" from the URL.

This behavior happens in my project, and in: https://covid-19.datasettes.com/covid/economist_excess_deaths (for instance) but not in this table: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants

I don't know what license to use... Would be nice to be able to add a comment regarding that uncertainty in my metadata.json file

I like laktak's little video comment in favor of Human json (Hjson) https://stackoverflow.com/questions/244777/can-comments-be-used-in-json

Hmmm... one of the commenters there said comments are allowed in yaml... so that's a good argument for yaml.

Anyhow, just came to mind, and thought I'd mention it here. Looks like https://hjson.github.io/ has the details.