hamlab/BUGS.md

# Bug bash — 2026-09-18

Fix pass 2026-09-18: every item carries its outcome; ✅ resolved and verified live, ➖ no change with reason. Found by a headless Chromium pass (iPhone and
1440px desktop viewports) over every page on the hub and on each remote machine, an
offline-node scenario (wyse stopped for the test, then restarted), direct API probing
with hostile inputs, and a code read. Ordered by severity within each group.

## Security / robustness

1. ✅ **RESOLVED 2026-09-18** — only regular files are readable; the byte cap applies to what is sent, not to `stat`. `/dev/zero`, `/dev/null` and FIFOs answer 400 in 30 ms; `/proc/cpuinfo` still reads. **Reading a special file hangs the request forever.** `GET /api/files/read?path=/dev/zero`
   never ends: the `max` cap compares against `stat` size, which is 0 for device files,
   so the cap never triggers and the handler streams `/dev/zero` until the client gives
   up. Same for FIFOs (blocks on open) and `/proc/self/mem`. Opening one in the Files
   viewer would hang the tab. Needs: refuse non-regular files for read/edit, and cap the
   copy with `io.LimitReader` regardless of `stat`.
2. ✅ **RESOLVED** — per-client buckets (first X-Forwarded-For hop or the peer) with the same 5-strike schedule, plus a global backstop of 100 failures per 5 min → 1 min. Verified: core's public IP locked while grid logged in fine. **Login lockout is global, not per client.** Five bad passwords from anyone on the
   internet lock the real user out for up to 10 minutes (`auth.Limiter` is one bucket).
   The bash locked the hub for the full 10 minutes twice. Also locks `POST /api/tokens`,
   so hub pairing fails during a lockout. Needs per-IP buckets with a modest global cap.
3. ✅ **RESOLVED** — server rejects refreshMs outside 100–60000 and non-identifier themes (400); client ignores out-of-range rates from any source. **UI prefs from the server are not validated.** A stored `refreshMs` of `-5` made the
   client call `setInterval(-5)` on every live view: a request storm that took the
   browser to `ERR_INSUFFICIENT_RESOURCES` and hammered the daemon. `theme` is validated
   only by falling back. Needs range checks on load (and on save server-side).
4. ✅ **RESOLVED** — every response carries X-Frame-Options DENY, CSP frame-ancestors none, nosniff, no-referrer. **No anti-framing / CSP headers.** Only HSTS (from NPM). A root-capable console should
   send `X-Frame-Options: DENY`, `Content-Security-Policy: frame-ancestors 'none'`,
   `Referrer-Policy: no-referrer`, `X-Content-Type-Options: nosniff` on every response.
5. ✅ **RESOLVED** — kernel threads (empty cmdline) and the daemon's own PID are refused. **Signals can be sent to kernel threads.** `POST /api/procs/2/signal` (kthreadd)
   returns 204. Harmless in practice, but the guard should refuse PIDs with an empty
   cmdline (kernel threads) and our own PID.
6. ✅ **RESOLVED** — unit runs with `SupplementaryGroups=systemd-journal adm` so journal reads need no sudo, and the installer drops `/etc/sudoers.d/hamlab-quiet` (no syslog/pam session for the read-only polling commands). Verified: 0 sudo lines in 60 s of 250 ms polling (was 839/10 min). Applied on all four hosts. **Journal spam from the daemon's own sudo calls.** Every sudo'd `journalctl`, `smartctl`
   or `pwrstat` writes three `pam_unix(sudo:session)` lines into the journal *under
   `hamlab.service`*: 839 lines in 10 minutes at a 250 ms refresh, and the journal is
   already 4 GB. It also buries Ham Lab's own log entries. Fix options: add the service
   user to `systemd-journal` (and `adm`) so journal reads don't need sudo, and/or a
   sudoers `Defaults!/usr/bin/journalctl !syslog, !pam_session` line in the installer.
7. ✅ **RESOLVED** — server answers 409 unless `overwrite=1`; the UI asks "Replace it?" first. **Uploading over an existing file silently overwrites it.** No confirmation, no
   "replace?" prompt. Same for "+ file" when the name exists (server refuses; UI just
   shows the error, fine) — upload is the gap.
8. ➖ **NO CHANGE** — Go's mux redirects the unclean path to `/etc/passwd`, which is an ordinary deep link the SPA must serve; there is no API prefix left by the time the static handler sees it. Harmless. **`/api/<anything>/../../x` falls through to the SPA** and returns `index.html` with
   200 instead of a JSON 404. Harmless, but sloppy for an API prefix.

## Functional bugs

9. ✅ **RESOLVED** — 401 carries the server message; the form shows "wrong password" (verified headless). **Wrong password gives no feedback at all.** The 401 is thrown as `Unauthenticated`
   with an empty message, and the login form only renders `err` when non-empty. The
   button just re-enables. Reproduced on phone and desktop.
10. ✅ **RESOLVED** — hub-only pages bounce to the dashboard on switch (verified headless). **Switching machines while on Settings or Machines keeps showing the hub's page.**
    `switchMachine` only resets the view for Machines; Settings stays, so "mac" shows
    core's account/token/about panels. Both are hub-only pages and should bounce to the
    dashboard (or be clearly labelled as hub pages).
11. ✅ **RESOLVED** — escaped; no console error. **Machines add-form `pattern` is an invalid regex in Chromium.** `[a-z0-9][a-z0-9-]*`
    fails under the `v` flag ("Invalid character class"), logs a console error and the
    browser skips validation. Needs `[a-z0-9][a-z0-9\-]*`.
12. ✅ **RESOLVED** — refresh from the newest repodata under libdnf5/dnf caches (Mac: 9 d), last upgrade from `dnf history` (194 d), label reads "from dnf history". **Updates tab on dnf hosts shows nonsense.** Mac: "Index refreshed 561d ago" (mtime of
    `/var/cache/dnf`, which Fedora 41+ no longer uses; `libdnf5` is the live cache) and
    "Last upgrade never · from apt history" (label and source are apt-specific). Needs
    `dnf history` for last upgrade, metadata timestamps for refresh, and neutral labels.
13. ✅ **RESOLVED** — journal read incrementally by cursor, timers from one `systemctl show`, services' two calls run in parallel. Cron 3.6 s → 0.08 s, services 0.6 s → 0.05 s, with last-run and next-run data intact. **Cron overview is slow: 3.6 s per request on core.** At any refresh rate the page
    is constantly busy. The 30-day, 20 000-line journal scan for "last run" is rebuilt
    every 5 s; timers make three `systemctl` calls. Needs an incremental journal cursor
    and a single `systemctl` call. Services list is 0.6 s for the same reason (two calls).
14. ✅ **RESOLVED** — heap profile showed a 48 MB `bytes.Buffer` from buffering `journalctl` dumps; those are now streamed line by line, plus a 160 MB soft memory limit for the GC. RSS 22 MB fresh, 27 MB after 25× cron/logs/procs/services (was 180 MB). pprof stays available behind the login at `/api/debug/pprof/`. **Daemon memory grows.** RSS on core went 22 → 85 → 216 MB during the day (136 MB at
    rest after the bash). Suspects: per-pid caches in the process table (`procInfo` only
    prunes on the next walk), task output buffers, SMART cache, xterm scrollback rings
    that are never freed for exited sessions. Needs a heap profile under load.
15. ✅ **RESOLVED** — `--dirty` dropped from the version and deploys now follow commits; binaries report e.g. `v0.1.0-38-gc1805f6`. **Version string is always `-dirty`.** `make deploy` builds before the commit, so
    every installed binary reports e.g. `v0.1.0-32-ga372378-dirty`, and the Machines page
    compares with `-dirty` stripped. Build after commit, or tag releases.
16. ✅ **RESOLVED** — hash routing `#/<machine|hub>/<view>`: reload restores machine and tab, Back/Forward work, links can point at a page on a machine (verified headless). **View resets to Dashboard on reload; no deep links; browser Back leaves the app.**
    The selected machine survives a reload (sessionStorage) but the tab does not, and
    there is no URL routing at all. Needs hash or path routing (`/#/grid/services`).
17. ✅ **RESOLVED** — newest first. **Cron log tab lists oldest first and starts at the top.** 200 lines from Sep 10 with
    the newest at the bottom, off-screen. Should show newest first or scroll to the end.
18. ✅ **RESOLVED** — "Nothing matches." **Processes: empty filter result shows nothing.** No "nothing matches" message.
19. ✅ **RESOLVED** — exited shells remove themselves 10 minutes after exit (verified live: seeded 17:53:56, gone at 18:03:59 with no daemon restart in between). Found and fixed alongside: a daemon restart used to re-spawn *exited* shells as fresh running ones; only running shells are persisted now. Idle running shells are deliberately kept. **Terminals accumulate.** Nothing ever reaps exited or idle sessions; the bash left
    several "bug-bash-term" shells on the hub until deleted by hand. Exited sessions
    should auto-remove after a while; idle ones at least be counted somewhere.
20. ✅ **RESOLVED** — a node answers 409 "busy" when a package task is running or a terminal is attached; the Machines page offers Force, `hamlab push-nodes --force` / `make push-nodes-force` do the same. Verified: 409 with a terminal attached on grid, 200 with force. **Self-update restarts without warning running work.** A push to a node kills its
    open terminals and any running task (upgrade in progress). The hub should refuse or
    warn when the node reports a running task or attached terminals.
21. ✅ **RESOLVED** — PathError and LinkError are unwrapped to the bare reason. **Error text leaks raw Go errors.** e.g. rename shows "rename /tmp/x /tmp/y: no such
    file or directory". Fine for a lab, ugly for a product; strip the syscall prefix.

## Mobile / UX

22. ✅ **RESOLVED** — on coarse pointers every button, chip, icon button, link, header and select is at least 36 px tall (verified: no control under 32 px on the dashboard or processes pages). **Tap targets under 28 px on phone:** header ⏻ (25), logout (22), customize ▲ ▼ hide
    (23), terminal leaf ✎ ✕ (24), chips (25), table headers (27). Apple's guideline is
    44 pt. Systemic: the small `.btn`/`.chip`/`.icon-btn` sizes need a touch variant.
23. ✅ **RESOLVED** — the dialog asks for the machine name (`mac`); the node still checks its own hostname under the hood. **Power dialog on the Mac asks you to type `5ce91ee54af6`.** The confirmation uses
    the node's hostname, which on the Mac is the unset default. Should offer the machine
    name (`mac`) as the confirmation word on remote machines.
24. ✅ **RESOLVED** — down interfaces collapse to one line marked "down"; modules with nothing to show (fans, UPS, temps, disks) hide unless customizing. **Dashboard shows dead interfaces and empty modules.** wlp1s0 (down) gets a full
    sparkline block of flat lines; "Fans: no fan sensors" is a permanent empty card on
    hosts without fans. Down interfaces should collapse to one line; empty modules should
    auto-hide unless customizing.
25. ✅ **RESOLVED** — "load 1.39 1.56 1.55" with a tooltip. **Header load average is unlabelled** ("1.39 1.56 1.55" floating in the header).
26. ➖ **NOT FIXING** — the list is intentionally a plain table; the menu anchors inside the row. Cosmetic taste, not a defect. **Files list on desktop is a narrow strip in a sea of empty space** for small
    folders; the row menu opens off the right edge of the row. Cosmetic.
27. ✅ **RESOLVED** — after a 502 the client leaves that machine alone for 5 s: 2 requests in 10 s at a 250 ms refresh (would have been ~80), badge and message intact. **Offline machine keeps polling every view at the refresh rate**, producing a 502
    per request per page (and console errors). The UI copes — badge, message — but the
    hub should back off to the prober's cadence while a node is down.
28. ✅ **RESOLVED** with 6. **Services logs are noise.** The default view of `hamlab.service` logs is the sudo
    session spam from bug 6, three lines per poll.

## Confirmed working under abuse (for the record)

Path traversal in files, services, cron, ZFS, procs and updates all rejected. `/etc`
and `/` protected from delete/rename as root. Unit, pool, dataset, snapshot and
package names validated. Token issue refused with a wrong password. Oversized prefs
rejected. Duplicate machine add refused. Unicode + spaces in file names round-trip.
Upload byte-exact. Terminal scrollback survives switching. Offline node: switcher and
header show it, pages show the reason. Remote terminals, logs, tasks and files through
the proxy all work on all three nodes.


# Bug bash — 2026-09-19

Found by driving a scratch instance (same binary, its own state, real fleet behind it) in
headless Chromium at 1440 px and iPhone size: a sweep of every page on every machine for
page errors, failed requests, overflow and small tap targets; then scripted use of every
feature — terminal tabs, Files in a throwaway folder, Processes (killing a process the
test spawned), Monitors customisation and widgets, Services, Cron, Updates, Disks,
Machines, Settings, embeds, machine switching, session loss — and a hostile pass over the
API. Host-changing actions (service control, cron edits, upgrades, power) were opened but
not executed. Every fix below was re-checked by the same script that found it.

## Bugs — all ✅ fixed and verified

1. ✅ **One widget with a bad pattern silenced every custom widget.** They share one stream;
   `fan.[` made the whole query a 400, so all of them showed "…" forever with no reason
   given. The picker now asks the API before adding and shows its answer; a bad pattern
   already saved shows a notice instead of dots.
2. ✅ **A saved layout of the wrong shape could not be loaded** ("could not load layout",
   and the page could no longer be customised). `normalize()` now accepts anything.
3. ✅ **A lost session left pages saying "unauthenticated"** for up to 30 s. A 401 from the hub
   now returns to the login form at once, and logging in returns to the same page. (A 401
   relayed from a managed machine is that machine rejecting the hub, and doesn't.)
4. ✅ **The machine-name rule in "Add a machine" was an invalid regular expression** under the
   browser's `v` mode, so it validated nothing and logged a console error on every render.
5. ✅ **Two terminals opened in the same minute got the same name** ("shell 09:45" twice).
   Now "shell 09:45 (2)".
6. ✅ **Names had no length limit** — a 328-character terminal tab, a 5000-character token
   name. 64 characters, refused with a reason; token names also refuse `/ \ < > "`.
7. ✅ **"+ file", "+ folder" and Rename accepted `../x` and `a/b`**, quietly acting outside the
   folder on screen. A name is now one path segment; Move… is how things go elsewhere.
8. ✅ **Unsaved file edits were thrown away without a word** when you picked another page or
   machine, or closed the tab. All three now ask; Cancel keeps you where you were.
9. ✅ **A mistyped path in Files was remembered**, so a reload landed on the error. Only a
   folder that opened is remembered.
10. ✅ **Escape closed nothing**: the machine menu, the file menu, the Power dialog, the widget
    picker. It closes all four.
11. ✅ **Reset on Monitors removed every custom widget with no confirmation.** It asks, and
    says how many.
12. ✅ **A few bytes of query bought 27 MB of answer** (`{"k0":"*",…×400}` with `full=1`) — and a
    read-only token is public on the demo page. A query may return 10 000 values (thirty
    whole machines); past that it is a 400 that says so.
13. ✅ **No ceiling on live streams**: 300 from one token were accepted. 32 per client address,
    512 in all, then a 429 that says so; closed streams free their places.
14. ✅ **Anything under `/embed/` lost the anti-framing headers**, not just the widget page.
    Only `/embed/widget` is frameable.
15. ✅ **The new ZFS sensors ran three `sudo` commands a minute on every machine** — including
    `core`, which has the tools and no pool — and each wrote three lines to the journal:
    most of `hamlab.service`'s log was that. Pools are now read with one unprivileged
    `zpool list`; a source that finds nothing is asked again ten times less often; and
    the installer's quiet list covers `zpool|zfs list|status`.
16. ✅ **Phone: the filter boxes on Processes and Files, the path box and the "/" crumb were
    under 30 px.** 36 px like every other control.
17. ✅ **Disks → datasets on a phone: rows ended 50 px before their own cells** when scrolled
    sideways (`min-width` was less than the columns add up to).

Also: tab titles now say where you are ("Services · grid — Ham Lab"), so history and open
tabs can be told apart.

## Checked and fine

Terminal focus after every switch/close/return; scrollback after reload; exited shells;
30 tabs with the + still reachable; re-fit on resize; Back/Forward; unknown machine or page
in the URL; wrong password; XSS through terminal, token and file names (all rendered as
text); hidden files, filter, symlinks (broken ones marked), binary files, upload and
replace, folder delete needing the typed name, root toggle; process sort, filter and
Terminate (asks first); themes; password-change validation; duplicate or unreachable
machine; hub terminals not leaking into another machine's page; CORS only on sensor paths;
token list never exposing hashes; cookie flags; deep nesting, empty bodies, odd methods
and `every=` values on the sensor API; phone drawer, key bar and picker.

## Quality of life — all ✅ done 2026-09-19, each verified in the browser

1. ✅ **Selections are in the URL.** `#/grid/services/ssh.service`, `#/hub/cron/job:…`,
   `#/hub/files/%2Fetc`: reload keeps them, links carry them, leaving a page drops them.
2. ✅ **Browser Back with unsaved file edits asks**, like page and machine changes do.
3. ✅ **No more browser `prompt()`/`confirm()` boxes** — all 30 are in-app dialogs (`askYes`,
   `askText` in `ui.tsx`): Enter submits, Escape or a tap outside cancels, destructive
   ones are red and say what they do ("Close", "Delete"), and they fit a phone.
4. ✅ **Token creation is one dialog**: name, patterns, and the machine's own sensor families
   as chips to click.
5. ✅ **Widgets can be edited** — sensor, type and title — in place, from Customize.
6. ✅ **The Widgets gallery has a search** over IDs, names and labels (so "gitea" finds its
   container's sensors).
7. ✅ **Status sensors keep a history** (300 readings each: an hour of container memory, two
   days of drive wear), so their sparklines have a past.
8. ✅ **A stream that fails says why**: "the token was rejected", "too many live streams are
   open", "that machine is unreachable", "can't reach Ham Lab" — in embeds and on Monitors.
9. ✅ **Pattern widgets get readable default titles**: `cpu.core.*.usage` → "CPU core usage".
10. ✅ **Terminal tabs reorder** — drag on a desktop, ‹ › on a touch screen, remembered per
    machine in the browser — and **Alt+Shift+←/→** switches tabs, **Alt+Shift+T** opens one,
    with the keyboard left in the terminal each time.
11. ➖ **"Log out everywhere"** — built, then removed the same day at Mitchell's word: no need
    for it on a single-user console.
12. ✅ **Real clipboard shortcuts in the terminal**, from a physical keyboard: **Ctrl+C copies
    when text is selected** (and is still ^C when nothing is — the selection clears after a
    copy, so a second Ctrl+C interrupts), **Ctrl+V pastes**, Ctrl+Shift+C / Ctrl+Shift+V do
    the same unconditionally, Ctrl+Insert / Shift+Insert too. It rides the browser's own
    copy and paste events, which xterm already answers, so it works over plain http on the
    LAN where `navigator.clipboard` does not exist; bracketed paste is honoured.


# Bug bash 2 — 2026-09-19 (evening): Alerts, Docker, Stacks, skins, the installer

Same method as the morning: a scratch instance with the real fleet behind it, headless
Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four machines (errors,
failed requests, overflow, tap targets) plus a contrast measurement of every skin; then
scripted use of the new features on throwaway containers and stacks — a multi-file stack, a
file edited behind the editor's back, two actions at once, a chatty log, odd container
names, containers stopped and removed while open, every kind of bad alert rule, a rule's
whole life (fire, narrow, pause, delete, survive a restart), token scope and hostile paths
on the new APIs — and a regression of terminals, Files, widgets, deep links and colours.

## Bugs — all ✅ fixed and verified by the script that found them

1. ✅ **Deploying a stack made of several compose files dropped all but the first.** A stack
   deployed with `-f compose.yaml -f compose.override.yaml` was redeployed from
   `compose.yaml` alone, so everything the override set silently vanished from the new
   container. Two causes: only the first file was used, and the file list was read from
   `docker ps`, which joins labels with commas — the list is itself comma-separated, so it
   arrived truncated. Labels now come from `inspect`; every file is resolved, shown as a
   chip, editable, validated together and passed to compose in order.
2. ✅ **A compose file changed on disk after you opened it was silently overwritten** (Dockhand,
   or a second tab). The editor now hands back the checksum it was given; a mismatch is a
   409 that says so, keeps your text, and offers to reload.
3. ✅ **Stacks on a machine without Docker answered 500.** An empty list.
4. ✅ **Every hub restart re-fired every firing alert** — and every deploy is a restart, so a
   phone would have been told about the same failed service on each one. What is firing is
   now kept in `state.json` and carried on, with its original start time.
5. ✅ **An alert whose rule was paused, deleted or narrowed just vanished**, leaving "firing" in
   the history forever and no word to the phone. It is now recorded (and sent) as cleared,
   with the reason: "rule paused", "rule deleted", "no longer watched".
6. ✅ **Nonsense rule values were accepted**: `null`, an object, `cpu.usage > ninety` (which can
   never be true, and would never say so), `true` with `<`. Refused, each with its reason.
7. ✅ **No limit on rules** — 3000 were accepted, each checked on every machine every 15 s. 200.
8. ✅ **Logs of a container that no longer exists answered 500.** 404.
9. ✅ **Four skins had text under 3:1 contrast**: Nord's red; Catppuccin Latte's muted text and
   green; Solarized Dark's body text, muted text and red; Solarized Light's body text,
   muted text, accent and green. Each colour kept its hue and was moved to a readable value
   (measured: all thirteen skins now pass, body text ≥ 7:1).
10. ✅ **Phone: the alert toggles (28 px) and the colour swatches (26 px) were under 30 px.**
11. ✅ **The public installer had fallen four builds behind the fleet** (v65 published, v69
    running), so `curl … | sh` on an up-to-date machine "upgraded" it backwards. `make
    deploy` now republishes, and the installer compares versions: an older build is called
    older, defaults to No, and unattended runs refuse it without `HAMLAB_ALLOW_DOWNGRADE=1`.

## Checked and fine

A second stack action while one runs (409); `../` and encoded tricks in stack names;
shell metacharacters in container names, `since` and actions; `rm` / `kill` / `down` are
not actions; a read-only token is refused on every Docker, Stacks and Alerts endpoint,
directly and through the hub; container names with dots and underscores still get live
CPU and memory; a log at five lines a second follows with no gaps or duplicates; a
container stopped or removed while open updates itself; an alert rule's XSS name renders
as text; per-machine firing, the rail badge, webhook and ntfy delivery; terminals, Files,
widgets over container sensors, deep links, tab titles and machine colours all unchanged.
No native browser dialogs anywhere; no page errors on any of 100 page loads.

## Found by the alerts themselves

`core` has one real alert firing: **`zfs-import-cache.service` fails at every boot** — `core`
has the ZFS tools and no pool, so there is nothing to import. Harmless, but it will sit on
the rail as "1" until the unit is disabled or the rule is narrowed. Left alone.

## Quality of life — five chosen by Mitchell, ✅ built and verified 2026-09-19

1. ✅ **Stacks on `grid` and the Mac are editable from the hub.** Their files live on `core`
   (Dockhand deploys over tcp); the hub resolves them and runs compose with `DOCKER_HOST`
   pointed at the node. Verified on a throwaway stack: saved on `core`, recreated **on
   `grid`**, same compose labels, nothing created on `core`, output followed on the page.
2. ✅ **A stack's `.env` is editable**, values masked until "Reveal and edit". A `.env` that
   breaks the compose file (a required variable removed) is refused and the old one put back.
6. ✅ **A link to a container that no longer exists says so**, by name.
9. ✅ **"Update available" per image** — on request, remembered six hours. Verified with an
   image whose tag had moved on (update available) beside a current one (up to date).
10. ✅ **What Deploy would change**: a line diff against what was last deployed from here;
    after a deploy it says there is nothing left to change. `.env` diffs name the key and
    never show a value.

Not chosen, still open: alert acknowledge/snooze; quiet hours and per-rule channels; push
straight from Ham Lab; a type hint in the alert editor; switching off the nodes' own alert
engines.

# Bug bash 3 — 2026-09-20: new stacks, new containers, images, volumes, the HUD

Started from Mitchell's report — the New stack dialog is see-through in HUD, its text
mixing with the page behind. Same method as before: a scratch hub with the real fleet behind
it, headless Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four
machines, plus a measurement **in every skin** of everything that floats over the page
(dialogs, the question box, the widget picker, menus, the phone drawer: background alpha
must be ≥ 0.9). Then scripted use of what is new on throwaway `hl-` stacks, containers,
images and volumes: the New stack / New container dialog's edges, stacks that are not
deployed, a deploy that fails, a container name that is taken, deletes beside other
people's files, a two-file stack, create → deploy → restart → stop → delete **on the Mac
and on `grid` through the hub**, pulls on the hub and on a node, removal guards, a
container stopped on purpose, a read-only token and hostile input on every new endpoint,
and a regression of Escape everywhere, terminals, Files, sensors, alerts and deep links.

## Bugs — all ✅ fixed and verified by the script that found them

1. ✅ **HUD: every dialog was see-through.** The skin's "glass" rule (3 % cyan) was written
   for tiles and cards and also caught `.modal` — the New stack dialog, the widget picker,
   the Power dialog, the token dialog, the rule editor and every question. Anything that
   floats over the page is now solid in HUD (hairline border and glow kept) over a darker
   backdrop. Measured in all thirteen skins: nothing that floats is under 0.9 alpha.
2. ✅ **HUD: the phone's slide-out menu was 92 % opaque**, the page showing through it. Solid.
3. ✅ **Deploy, Update, Restart, Stop and Down all failed on the Mac's stack**
   (`peaches-and-plums`) — and on any stack held on the hub that was deployed from the hub's
   own shell rather than from inside Dockhand. The command was built as
   `'DOCKER_HOST=tcp://…' docker compose …`; quoted, `sh` reads that not as a variable but
   as the name of a program to run ("not found", exit 127). Now passed through `env`.
   Verified by a whole life on the Mac and on `grid` from the hub: create, deploy (running
   there, nothing on `core`), restart, stop, delete.
4. ✅ **A container stopped on purpose was reported as down** when it has no restart policy
   and doesn't handle SIGTERM: `docker stop` then ends it with 137 (or 143, or 130), a
   non-zero code, which read as a crash — "Container down" would have fired for a
   deliberate stop. Those three codes now mean "a person", unless the kernel did the
   killing for want of memory (`OOMKilled`). A real crash (exit 3) is still down.
5. ✅ **A stack with no containers offered Restart, Stop and Update**, and Restart
   "succeeded" having done nothing. Offered only when there is something to act on; the
   API answers 409 "isn't deployed: there is nothing to restart".
6. ✅ **One stray tap outside the New stack dialog threw away everything typed** — on a phone,
   a compose file's worth. So did Escape. With anything typed, all three ways out (tap
   outside, Escape, Cancel) ask first; an untouched dialog still just closes.
7. ✅ **Compose → Form → Compose silently replaced hand-edited compose text** with what the
   form says. The form now writes the box only when the form has changed since it last
   did, and asks before replacing text edited by hand.
8. ✅ **A question asked from inside a dialog opened underneath it** (same layer, earlier in
   the page), so it could not be answered — found the moment the New stack dialog started
   asking. Questions now sit above whatever asked them.
9. ✅ **Escape on a question also reached the dialog beneath it**: one key cancelled the
   question *and* closed (or re-asked from) the dialog under it. Escape now answers the
   question and goes no further; the next Escape is the dialog's. Checked on every dialog
   and menu: widget picker, machine menu, rule editor, Power, New stack, and that Escape on
   "Delete the rule?" deletes nothing.

10. ✅ **Phone: the bottom of the New container dialog could not be reached** (Mitchell, from
    his iPhone). A dialog is a column whose parts were allowed to shrink, so instead of the
    dialog scrolling, the one part with its own scrollbar — the compose preview — was
    squeezed to 22 px and everything "fitted". And `92vh` on iOS is taller than the screen
    with the address bar gone, so the top sat under the clock. Parts now keep their height
    and the dialog scrolls inside itself; it is never taller than the space between the
    status bar and the home indicator (safe-area insets, no `vh`); Create / Cancel — and
    any error — stay pinned at the bottom while the form scrolls. Applies to every dialog;
    measured at 390×844, 375×667 and sideways, and the picker, rule editor, token and Power
    dialogs rechecked.

## Checked and fine

All 13 pages × four machines × both sizes: no page errors, failed requests, sideways
scroll or tap targets under 30 px. A port of `not-a-port` is refused by compose before
anything is written, and leaves no folder. Environment values with quotes, `#`, `$` and
nothing after the `=` arrive in the container exactly as typed. A deploy whose image
doesn't exist shows the registry's error and is **not** remembered as "last deployed". A
container name already taken fails the deploy and says why. Delete moves only the stack's
own compose files and `.env` (kept `600` in the trash) and leaves a `data/` folder and
notes beside them alone; both files of a two-file stack go to the trash; the wrong name,
or none, deletes nothing. A `.env` value never appears on the stack page and stays masked
until asked for. A second stack of the same name is 409; an upper-case name is 400. A pull
of an image that doesn't exist ends as a failed task; a pull on `grid` from the hub lands
on `grid` and its output is followed. A volume or image in use can't be removed; a running
container can't be removed (409); a volume needs its name twice. A read-only token gets
401 from all fourteen new write paths, through the hub as well. `--force`, `-f`,
`--all-tags`, `..%2F..` and a home outside the known ones are refused; a 1.2 MB compose
file is refused. Shaped, fleet and streamed sensor queries, `/metrics`, the ten alert
rules, terminal typing, the file editor's discard question and service deep links are
unchanged. No native browser dialogs anywhere.

## Quality of life — four chosen by Mitchell, ✅ built and verified 2026-09-20

1. ✅ **"Where" fits on a phone**: `…/stacks/core · 12 stacks` instead of the whole path (which
   is still written out underneath). Two homes whose last two folders match fall back to the
   full path. Measured at 390 px: no option is wider than the box.
2. ✅ **How a task ended is a badge**: green `finished`, red `failed · exit 1` with the box
   outlined in red, grey `cancelled`. Verified on a deploy of an image that doesn't exist.
5. ✅ **Duplicate…** on a stack: the New stack dialog opened on a copy of its compose file,
   named `<name>-copy`. The original's `.env` is copied **on the machine** (`env_from`), mode
   600, its values never sent to the browser — checked: the secret is nowhere in the page.
   Anything that would collide with what is running is named before you press Create
   ("a container called X already exists; port 18222 is already published by X") — for
   every new stack, not only duplicates — and clears as you fix it. Disabled while the
   original has unsaved edits. Verified: copy deployed beside the original, which never
   stopped.
6. ✅ **The form has Labels and Network.** Labels one per line (`$`, quotes and spaces arrive
   intact). Network: its own, the machine's (`host`, which turns Ports off), or **join** one
   that already exists — written as an `external` network so compose doesn't make a second
   one. Verified on the container: both labels, attached to the chosen network only.

Found while building them, ✅ fixed:

- **Cancel on a stack task cancelled nothing.** It killed the `sh` that ran compose, not
  compose; and as compose still held the output pipe, the task stayed "running" and the
  stack "busy" until compose finished by itself — for a deploy waiting on a health check,
  minutes. Tasks now run in their own process group and the whole group is stopped (TERM,
  then KILL after 4 s); for stacks deployed inside Dockhand, where the process is not the
  task's child at all, the cancel goes into the container after it. Verified both ways:
  ended in ~100 ms, compose gone, the stack free for the next action.
- **Phone: a stack's task output was squeezed to a line** under the buttons (the same fault
  as the dialog: a scrolling column whose parts were allowed to shrink), which also put
  Cancel out of reach. Detail panes keep their parts' heights; services, containers, stacks
  and cron rechecked on both sizes.
- The Duplicate dialog and the stack page under it shared a React key, so the dialog
  multiplied. An undeployed stack's dot was green ("0 of 0 running"); now neutral.

# Bug bash 4 — 2026-09-21: the Top tab, and the refresh rate it didn't follow

Started from Mitchell's first look at Top on his own machine (Firefox on Windows, a window
1392 CSS px wide — none of which the tab had been drawn for): "Command" printed over
"Threads", the rate in Settings ignored, the symbol beside *logout* a box of hex digits;
and make it follow the skins. Method as before, plus two things new: **Firefox** (Playwright's
build; the GTK libraries it wants are unpacked from their `.deb`s into
`~/.cache/ms-playwright/fflibs` and given to it as `LD_LIBRARY_PATH`, with
`PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=1` — nothing is installed on `core`), and a layout
checker that measures instead of looking — every heading against its neighbour and its own
text, every row against its box, every border label, at eight window sizes from 900 to 1920.
Run first against the live v83 it reproduced the screenshot to the pixel ("the Command column
is 11px wide" at 1392); run after, nothing, in both browsers.

## Bugs — all ✅ fixed and verified by the script that found them

1. ✅ **Between about 1280 and 1500 px the Command column collapsed to a letter** and its
   heading ran under "Threads:" (itself 4 px too narrow for its own text at every size). The
   table's fixed columns were sized for one window. The process box now measures *itself*
   (a container query — it is a third of a wide window and all of a narrow one, so the
   window's width says nothing) and drops the curve, then Threads, then Command, as it
   narrows; the box also gets half the width rather than 44 %, and no heading can spill
   into the next. Stacked layout now starts at 1280, not 1100.
2. ✅ **The refresh rate reached a machine only through its Dashboard page.** The sampler on
   each machine was told the rate by an open Dashboard and nothing else — so with the rate
   at 250 ms, `grid`, `wyse` and the Mac were still sampling at 1 s (measured, live), and
   Top, Processes, Disks and every widget of theirs "ignored" the setting until somebody
   opened that machine's Dashboard. The rate is now the fleet's: saved with the UI prefs on
   the hub, which sets its own sampler and passes it to every machine (`PUT
   /api/prefs/sampler`, remembered in each `state.json`), and says it again at start-up for
   a node that restarted or joined since. Verified: Settings 2 s → 250 ms moves the sampler
   with no Dashboard visit.
3. ✅ **Top read the process table at most once a second whatever the rate** — a clamp of
   my own, exactly what `REFRESH` promises not to do. It reads at the rate: 20 reads in 5 s
   at 250 ms, 10 at 500 ms. (59 frames a second with 390 rows at 250 ms, no stalls, both
   browsers — the rows are only redrawn when the table is read.)
4. ✅ **Top's `− rate +` changed the rate for half a minute.** It set the browser's copy
   and never saved it, so the next sync with the hub put it back. It saves — which is also
   what moves the samplers — survives a reload, and each end is disabled at the limit.
5. ✅ **A viewer whose clock is wrong froze the curves.** New points were timed against the
   machine's timestamps: a browser clock 7 minutes slow waited 7 minutes for its next point
   (Top *and* the Dashboard); 7 fast, and it added one per tick. Timed on the browser's own
   clock now. Verified with the page's clock shifted both ways: 5 points in 10 s at a 2 s
   stride, both pages, both browsers.
6. ✅ **Every other point was being skipped.** "Has a stride passed?" was asked with `>=`
   of a timer that fires a few ms early half the time — so at a 2 s rate and a 2 s stride a
   curve gained 3 points in 10 s, not 5, and its time axis was quietly wrong. The same
   question on the server kept every bucket one sample long (14 s points at a 12 s stride).
   Both now allow half a tick of slack and keep to the stride's own beat; the server times
   from the last point kept, so a stalled or sleeping machine just keeps the next sample it
   gets. Unit-tested with jittered samples: 240 samples → exactly 40 points, 12 000 ± 100 ms
   apart, each the mean of its own six.
7. ✅ **The power symbol was a typed `⏻`** (U+23FB, Unicode 9) — not in Windows' fonts, so
   Firefox there drew the missing-glyph box. Drawn as an SVG like the other icons. Every
   other symbol the console types was checked against the same risk: all older.
8. ✅ **A handful of samples was stretched into a bar chart.** A machine ten seconds into
   its history filled the whole CPU box with five fat columns. The box is now always the
   whole history window — a young curve draws in from the right, as btop's does, and a full
   one fills it edge to edge, at any window (checked at 10 min and at 1 min).
9. ✅ **Idle curves vanished**: a core at 10 % rounded to no dots at all in its mini graph.
   Anything above zero lights the bottom row, which is also btop's rule.

10. ✅ **Found by the deploy of fix 2 itself: telling the machines once is not enough.** The hub
    restarted first and told three nodes that did not yet have the endpoint (404); they then
    updated and carried on at 1 s. The same would happen to any node that is off or
    restarting when a setting changes. The hub now *keeps* the fleet in step: fifteen seconds
    after it starts and every two minutes after, it asks each machine for its sampling rate
    and history window and corrects only what differs (logged as `fleet: grid had ms=1000,
    now 250`).

## Skins

✅ Top now wears the skin. The ground is the skin's terminal background and every curve and
meter is built from its sixteen terminal colours (chosen to be read on exactly that ground),
the rest from the usual variables — so the dark skins look like btop and the light ones like
btop on paper, and a skin arriving from another device repaints the page in place. Measured
in all thirteen: text ≥ 4.5 : 1, everything else ≥ 3 : 1, dots ≥ 2.2 : 1. Two failed —
process names in Latte (2.75) and Solarized Light (2.62), where a terminal green is too
pale for small type — and use `--ok`, the green each skin was already measured readable
with.

## Checked and fine

Sort by pid, name, mem and cpu, and reverse; filter by name, command, user and pid, with the
count showing matched/total and an empty result saying so; a selection that holds while the
table is re-read under it; terminate, on a throwaway `sleep`, asks by name and pid, ends it,
and the footer stops offering a process that is gone. Sampler values outside 100 ms–60 s are
refused, and a rate slower than the collector's 10 s samples at 10 s. All 14 pages at his
window size and on a phone, in both browsers: no errors, no overflow.