hamlab/BUGS.md

Bug bash — 2026-09-18

Fix pass 2026-09-18: every item carries its outcome; ✅ resolved and verified live, ➖ no change with reason. Found by a headless Chromium pass (iPhone and 1440px desktop viewports) over every page on the hub and on each remote machine, an offline-node scenario (wyse stopped for the test, then restarted), direct API probing with hostile inputs, and a code read. Ordered by severity within each group.

Security / robustness

1. ✅ RESOLVED 2026-09-18 — only regular files are readable; the byte cap applies to what is sent, not to stat. /dev/zero, /dev/null and FIFOs answer 400 in 30 ms; /proc/cpuinfo still reads. Reading a special file hangs the request forever. GET /api/files/read?path=/dev/zero never ends: the max cap compares against stat size, which is 0 for device files, so the cap never triggers and the handler streams /dev/zero until the client gives up. Same for FIFOs (blocks on open) and /proc/self/mem. Opening one in the Files viewer would hang the tab. Needs: refuse non-regular files for read/edit, and cap the copy with io.LimitReader regardless of stat.
2. ✅ RESOLVED — per-client buckets (first X-Forwarded-For hop or the peer) with the same 5-strike schedule, plus a global backstop of 100 failures per 5 min → 1 min. Verified: core's public IP locked while grid logged in fine. Login lockout is global, not per client. Five bad passwords from anyone on the internet lock the real user out for up to 10 minutes (auth.Limiter is one bucket). The bash locked the hub for the full 10 minutes twice. Also locks POST /api/tokens, so hub pairing fails during a lockout. Needs per-IP buckets with a modest global cap.
3. ✅ RESOLVED — server rejects refreshMs outside 100–60000 and non-identifier themes (400); client ignores out-of-range rates from any source. UI prefs from the server are not validated. A stored refreshMs of -5 made the client call setInterval(-5) on every live view: a request storm that took the browser to ERR_INSUFFICIENT_RESOURCES and hammered the daemon. theme is validated only by falling back. Needs range checks on load (and on save server-side).
4. ✅ RESOLVED — every response carries X-Frame-Options DENY, CSP frame-ancestors none, nosniff, no-referrer. No anti-framing / CSP headers. Only HSTS (from NPM). A root-capable console should send X-Frame-Options: DENY, Content-Security-Policy: frame-ancestors 'none', Referrer-Policy: no-referrer, X-Content-Type-Options: nosniff on every response.
5. ✅ RESOLVED — kernel threads (empty cmdline) and the daemon's own PID are refused. Signals can be sent to kernel threads. POST /api/procs/2/signal (kthreadd) returns 204. Harmless in practice, but the guard should refuse PIDs with an empty cmdline (kernel threads) and our own PID.
6. ✅ RESOLVED — unit runs with SupplementaryGroups=systemd-journal adm so journal reads need no sudo, and the installer drops /etc/sudoers.d/hamlab-quiet (no syslog/pam session for the read-only polling commands). Verified: 0 sudo lines in 60 s of 250 ms polling (was 839/10 min). Applied on all four hosts. Journal spam from the daemon's own sudo calls. Every sudo'd journalctl, smartctl or pwrstat writes three pam_unix(sudo:session) lines into the journal under hamlab.service: 839 lines in 10 minutes at a 250 ms refresh, and the journal is already 4 GB. It also buries Ham Lab's own log entries. Fix options: add the service user to systemd-journal (and adm) so journal reads don't need sudo, and/or a sudoers Defaults!/usr/bin/journalctl !syslog, !pam_session line in the installer.
7. ✅ RESOLVED — server answers 409 unless overwrite=1; the UI asks "Replace it?" first. Uploading over an existing file silently overwrites it. No confirmation, no "replace?" prompt. Same for "+ file" when the name exists (server refuses; UI just shows the error, fine) — upload is the gap.
8. ➖ NO CHANGE — Go's mux redirects the unclean path to /etc/passwd, which is an ordinary deep link the SPA must serve; there is no API prefix left by the time the static handler sees it. Harmless. /api/<anything>/../../x falls through to the SPA and returns index.html with 200 instead of a JSON 404. Harmless, but sloppy for an API prefix.

Functional bugs

9. ✅ RESOLVED — 401 carries the server message; the form shows "wrong password" (verified headless). Wrong password gives no feedback at all. The 401 is thrown as Unauthenticated with an empty message, and the login form only renders err when non-empty. The button just re-enables. Reproduced on phone and desktop.
10. ✅ RESOLVED — hub-only pages bounce to the dashboard on switch (verified headless). Switching machines while on Settings or Machines keeps showing the hub's page. switchMachine only resets the view for Machines; Settings stays, so "mac" shows core's account/token/about panels. Both are hub-only pages and should bounce to the dashboard (or be clearly labelled as hub pages).
11. ✅ RESOLVED — escaped; no console error. Machines add-form pattern is an invalid regex in Chromium. [a-z0-9][a-z0-9-]* fails under the v flag ("Invalid character class"), logs a console error and the browser skips validation. Needs [a-z0-9][a-z0-9\-]*.
12. ✅ RESOLVED — refresh from the newest repodata under libdnf5/dnf caches (Mac: 9 d), last upgrade from dnf history (194 d), label reads "from dnf history". Updates tab on dnf hosts shows nonsense. Mac: "Index refreshed 561d ago" (mtime of /var/cache/dnf, which Fedora 41+ no longer uses; libdnf5 is the live cache) and "Last upgrade never · from apt history" (label and source are apt-specific). Needs dnf history for last upgrade, metadata timestamps for refresh, and neutral labels.
13. ✅ RESOLVED — journal read incrementally by cursor, timers from one systemctl show, services' two calls run in parallel. Cron 3.6 s → 0.08 s, services 0.6 s → 0.05 s, with last-run and next-run data intact. Cron overview is slow: 3.6 s per request on core. At any refresh rate the page is constantly busy. The 30-day, 20 000-line journal scan for "last run" is rebuilt every 5 s; timers make three systemctl calls. Needs an incremental journal cursor and a single systemctl call. Services list is 0.6 s for the same reason (two calls).
14. ✅ RESOLVED — heap profile showed a 48 MB bytes.Buffer from buffering journalctl dumps; those are now streamed line by line, plus a 160 MB soft memory limit for the GC. RSS 22 MB fresh, 27 MB after 25× cron/logs/procs/services (was 180 MB). pprof stays available behind the login at /api/debug/pprof/. Daemon memory grows. RSS on core went 22 → 85 → 216 MB during the day (136 MB at rest after the bash). Suspects: per-pid caches in the process table (procInfo only prunes on the next walk), task output buffers, SMART cache, xterm scrollback rings that are never freed for exited sessions. Needs a heap profile under load.
15. ✅ RESOLVED — --dirty dropped from the version and deploys now follow commits; binaries report e.g. v0.1.0-38-gc1805f6. Version string is always -dirty. make deploy builds before the commit, so every installed binary reports e.g. v0.1.0-32-ga372378-dirty, and the Machines page compares with -dirty stripped. Build after commit, or tag releases.
16. ✅ RESOLVED — hash routing #/<machine|hub>/<view>: reload restores machine and tab, Back/Forward work, links can point at a page on a machine (verified headless). View resets to Dashboard on reload; no deep links; browser Back leaves the app. The selected machine survives a reload (sessionStorage) but the tab does not, and there is no URL routing at all. Needs hash or path routing (/#/grid/services).
17. ✅ RESOLVED — newest first. Cron log tab lists oldest first and starts at the top. 200 lines from Sep 10 with the newest at the bottom, off-screen. Should show newest first or scroll to the end.
18. ✅ RESOLVED — "Nothing matches." Processes: empty filter result shows nothing. No "nothing matches" message.
19. ✅ RESOLVED — exited shells remove themselves 10 minutes after exit (verified live: seeded 17:53:56, gone at 18:03:59 with no daemon restart in between). Found and fixed alongside: a daemon restart used to re-spawn exited shells as fresh running ones; only running shells are persisted now. Idle running shells are deliberately kept. Terminals accumulate. Nothing ever reaps exited or idle sessions; the bash left several "bug-bash-term" shells on the hub until deleted by hand. Exited sessions should auto-remove after a while; idle ones at least be counted somewhere.
20. ✅ RESOLVED — a node answers 409 "busy" when a package task is running or a terminal is attached; the Machines page offers Force, hamlab push-nodes --force / make push-nodes-force do the same. Verified: 409 with a terminal attached on grid, 200 with force. Self-update restarts without warning running work. A push to a node kills its open terminals and any running task (upgrade in progress). The hub should refuse or warn when the node reports a running task or attached terminals.
21. ✅ RESOLVED — PathError and LinkError are unwrapped to the bare reason. Error text leaks raw Go errors. e.g. rename shows "rename /tmp/x /tmp/y: no such file or directory". Fine for a lab, ugly for a product; strip the syscall prefix.

Mobile / UX

22. ✅ RESOLVED — on coarse pointers every button, chip, icon button, link, header and select is at least 36 px tall (verified: no control under 32 px on the dashboard or processes pages). Tap targets under 28 px on phone: header ⏻ (25), logout (22), customize ▲ ▼ hide (23), terminal leaf ✎ ✕ (24), chips (25), table headers (27). Apple's guideline is 44 pt. Systemic: the small .btn/.chip/.icon-btn sizes need a touch variant.
23. ✅ RESOLVED — the dialog asks for the machine name (mac); the node still checks its own hostname under the hood. Power dialog on the Mac asks you to type 5ce91ee54af6. The confirmation uses the node's hostname, which on the Mac is the unset default. Should offer the machine name (mac) as the confirmation word on remote machines.
24. ✅ RESOLVED — down interfaces collapse to one line marked "down"; modules with nothing to show (fans, UPS, temps, disks) hide unless customizing. Dashboard shows dead interfaces and empty modules. wlp1s0 (down) gets a full sparkline block of flat lines; "Fans: no fan sensors" is a permanent empty card on hosts without fans. Down interfaces should collapse to one line; empty modules should auto-hide unless customizing.
25. ✅ RESOLVED — "load 1.39 1.56 1.55" with a tooltip. Header load average is unlabelled ("1.39 1.56 1.55" floating in the header).
26. ➖ NOT FIXING — the list is intentionally a plain table; the menu anchors inside the row. Cosmetic taste, not a defect. Files list on desktop is a narrow strip in a sea of empty space for small folders; the row menu opens off the right edge of the row. Cosmetic.
27. ✅ RESOLVED — after a 502 the client leaves that machine alone for 5 s: 2 requests in 10 s at a 250 ms refresh (would have been ~80), badge and message intact. Offline machine keeps polling every view at the refresh rate, producing a 502 per request per page (and console errors). The UI copes — badge, message — but the hub should back off to the prober's cadence while a node is down.
28. ✅ RESOLVED with 6. Services logs are noise. The default view of hamlab.service logs is the sudo session spam from bug 6, three lines per poll.

Confirmed working under abuse (for the record)

Path traversal in files, services, cron, ZFS, procs and updates all rejected. /etc and / protected from delete/rename as root. Unit, pool, dataset, snapshot and package names validated. Token issue refused with a wrong password. Oversized prefs rejected. Duplicate machine add refused. Unicode + spaces in file names round-trip. Upload byte-exact. Terminal scrollback survives switching. Offline node: switcher and header show it, pages show the reason. Remote terminals, logs, tasks and files through the proxy all work on all three nodes.

Bug bash — 2026-09-19

Found by driving a scratch instance (same binary, its own state, real fleet behind it) in headless Chromium at 1440 px and iPhone size: a sweep of every page on every machine for page errors, failed requests, overflow and small tap targets; then scripted use of every feature — terminal tabs, Files in a throwaway folder, Processes (killing a process the test spawned), Monitors customisation and widgets, Services, Cron, Updates, Disks, Machines, Settings, embeds, machine switching, session loss — and a hostile pass over the API. Host-changing actions (service control, cron edits, upgrades, power) were opened but not executed. Every fix below was re-checked by the same script that found it.

Bugs — all ✅ fixed and verified

1. ✅ One widget with a bad pattern silenced every custom widget. They share one stream; fan.[ made the whole query a 400, so all of them showed "…" forever with no reason given. The picker now asks the API before adding and shows its answer; a bad pattern already saved shows a notice instead of dots.
2. ✅ A saved layout of the wrong shape could not be loaded ("could not load layout", and the page could no longer be customised). normalize() now accepts anything.
3. ✅ A lost session left pages saying "unauthenticated" for up to 30 s. A 401 from the hub now returns to the login form at once, and logging in returns to the same page. (A 401 relayed from a managed machine is that machine rejecting the hub, and doesn't.)
4. ✅ The machine-name rule in "Add a machine" was an invalid regular expression under the browser's v mode, so it validated nothing and logged a console error on every render.
5. ✅ Two terminals opened in the same minute got the same name ("shell 09:45" twice). Now "shell 09:45 (2)".
6. ✅ Names had no length limit — a 328-character terminal tab, a 5000-character token name. 64 characters, refused with a reason; token names also refuse / \ < > ".
7. ✅ "+ file", "+ folder" and Rename accepted ../x and a/b, quietly acting outside the folder on screen. A name is now one path segment; Move… is how things go elsewhere.
8. ✅ Unsaved file edits were thrown away without a word when you picked another page or machine, or closed the tab. All three now ask; Cancel keeps you where you were.
9. ✅ A mistyped path in Files was remembered, so a reload landed on the error. Only a folder that opened is remembered.
10. ✅ Escape closed nothing: the machine menu, the file menu, the Power dialog, the widget picker. It closes all four.
11. ✅ Reset on Monitors removed every custom widget with no confirmation. It asks, and says how many.
12. ✅ A few bytes of query bought 27 MB of answer ({"k0":"*",…×400} with full=1) — and a read-only token is public on the demo page. A query may return 10 000 values (thirty whole machines); past that it is a 400 that says so.
13. ✅ No ceiling on live streams: 300 from one token were accepted. 32 per client address, 512 in all, then a 429 that says so; closed streams free their places.
14. ✅ Anything under /embed/ lost the anti-framing headers, not just the widget page. Only /embed/widget is frameable.
15. ✅ The new ZFS sensors ran three sudo commands a minute on every machine — including core, which has the tools and no pool — and each wrote three lines to the journal: most of hamlab.service's log was that. Pools are now read with one unprivileged zpool list; a source that finds nothing is asked again ten times less often; and the installer's quiet list covers zpool|zfs list|status.
16. ✅ Phone: the filter boxes on Processes and Files, the path box and the "/" crumb were under 30 px. 36 px like every other control.
17. ✅ Disks → datasets on a phone: rows ended 50 px before their own cells when scrolled sideways (min-width was less than the columns add up to).

Also: tab titles now say where you are ("Services · grid — Ham Lab"), so history and open tabs can be told apart.

Checked and fine

Terminal focus after every switch/close/return; scrollback after reload; exited shells; 30 tabs with the + still reachable; re-fit on resize; Back/Forward; unknown machine or page in the URL; wrong password; XSS through terminal, token and file names (all rendered as text); hidden files, filter, symlinks (broken ones marked), binary files, upload and replace, folder delete needing the typed name, root toggle; process sort, filter and Terminate (asks first); themes; password-change validation; duplicate or unreachable machine; hub terminals not leaking into another machine's page; CORS only on sensor paths; token list never exposing hashes; cookie flags; deep nesting, empty bodies, odd methods and every= values on the sensor API; phone drawer, key bar and picker.

Quality of life — all ✅ done 2026-09-19, each verified in the browser

1. ✅ Selections are in the URL. #/grid/services/ssh.service, #/hub/cron/job:…, #/hub/files/%2Fetc: reload keeps them, links carry them, leaving a page drops them.
2. ✅ Browser Back with unsaved file edits asks, like page and machine changes do.
3. ✅ No more browser prompt()/confirm() boxes — all 30 are in-app dialogs (askYes, askText in ui.tsx): Enter submits, Escape or a tap outside cancels, destructive ones are red and say what they do ("Close", "Delete"), and they fit a phone.
4. ✅ Token creation is one dialog: name, patterns, and the machine's own sensor families as chips to click.
5. ✅ Widgets can be edited — sensor, type and title — in place, from Customize.
6. ✅ The Widgets gallery has a search over IDs, names and labels (so "gitea" finds its container's sensors).
7. ✅ Status sensors keep a history (300 readings each: an hour of container memory, two days of drive wear), so their sparklines have a past.
8. ✅ A stream that fails says why: "the token was rejected", "too many live streams are open", "that machine is unreachable", "can't reach Ham Lab" — in embeds and on Monitors.
9. ✅ Pattern widgets get readable default titles: cpu.core.*.usage → "CPU core usage".
10. ✅ Terminal tabs reorder — drag on a desktop, ‹ › on a touch screen, remembered per machine in the browser — and Alt+Shift+←/→ switches tabs, Alt+Shift+T opens one, with the keyboard left in the terminal each time.
11. ➖ "Log out everywhere" — built, then removed the same day at Mitchell's word: no need for it on a single-user console.
12. ✅ Real clipboard shortcuts in the terminal, from a physical keyboard: Ctrl+C copies when text is selected (and is still ^C when nothing is — the selection clears after a copy, so a second Ctrl+C interrupts), Ctrl+V pastes, Ctrl+Shift+C / Ctrl+Shift+V do the same unconditionally, Ctrl+Insert / Shift+Insert too. It rides the browser's own copy and paste events, which xterm already answers, so it works over plain http on the LAN where navigator.clipboard does not exist; bracketed paste is honoured.

Bug bash 2 — 2026-09-19 (evening): Alerts, Docker, Stacks, skins, the installer

Same method as the morning: a scratch instance with the real fleet behind it, headless Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four machines (errors, failed requests, overflow, tap targets) plus a contrast measurement of every skin; then scripted use of the new features on throwaway containers and stacks — a multi-file stack, a file edited behind the editor's back, two actions at once, a chatty log, odd container names, containers stopped and removed while open, every kind of bad alert rule, a rule's whole life (fire, narrow, pause, delete, survive a restart), token scope and hostile paths on the new APIs — and a regression of terminals, Files, widgets, deep links and colours.

Bugs — all ✅ fixed and verified by the script that found them

1. ✅ Deploying a stack made of several compose files dropped all but the first. A stack deployed with -f compose.yaml -f compose.override.yaml was redeployed from compose.yaml alone, so everything the override set silently vanished from the new container. Two causes: only the first file was used, and the file list was read from docker ps, which joins labels with commas — the list is itself comma-separated, so it arrived truncated. Labels now come from inspect; every file is resolved, shown as a chip, editable, validated together and passed to compose in order.
2. ✅ A compose file changed on disk after you opened it was silently overwritten (Dockhand, or a second tab). The editor now hands back the checksum it was given; a mismatch is a 409 that says so, keeps your text, and offers to reload.
3. ✅ Stacks on a machine without Docker answered 500. An empty list.
4. ✅ Every hub restart re-fired every firing alert — and every deploy is a restart, so a phone would have been told about the same failed service on each one. What is firing is now kept in state.json and carried on, with its original start time.
5. ✅ An alert whose rule was paused, deleted or narrowed just vanished, leaving "firing" in the history forever and no word to the phone. It is now recorded (and sent) as cleared, with the reason: "rule paused", "rule deleted", "no longer watched".
6. ✅ Nonsense rule values were accepted: null, an object, cpu.usage > ninety (which can never be true, and would never say so), true with <. Refused, each with its reason.
7. ✅ No limit on rules — 3000 were accepted, each checked on every machine every 15 s. 200.
8. ✅ Logs of a container that no longer exists answered 500. 404.
9. ✅ Four skins had text under 3:1 contrast: Nord's red; Catppuccin Latte's muted text and green; Solarized Dark's body text, muted text and red; Solarized Light's body text, muted text, accent and green. Each colour kept its hue and was moved to a readable value (measured: all thirteen skins now pass, body text ≥ 7:1).
10. ✅ Phone: the alert toggles (28 px) and the colour swatches (26 px) were under 30 px.
11. ✅ The public installer had fallen four builds behind the fleet (v65 published, v69 running), so curl … | sh on an up-to-date machine "upgraded" it backwards. make deploy now republishes, and the installer compares versions: an older build is called older, defaults to No, and unattended runs refuse it without HAMLAB_ALLOW_DOWNGRADE=1.

Checked and fine

A second stack action while one runs (409); ../ and encoded tricks in stack names; shell metacharacters in container names, since and actions; rm / kill / down are not actions; a read-only token is refused on every Docker, Stacks and Alerts endpoint, directly and through the hub; container names with dots and underscores still get live CPU and memory; a log at five lines a second follows with no gaps or duplicates; a container stopped or removed while open updates itself; an alert rule's XSS name renders as text; per-machine firing, the rail badge, webhook and ntfy delivery; terminals, Files, widgets over container sensors, deep links, tab titles and machine colours all unchanged. No native browser dialogs anywhere; no page errors on any of 100 page loads.

Found by the alerts themselves

core has one real alert firing: zfs-import-cache.service fails at every boot — core has the ZFS tools and no pool, so there is nothing to import. Harmless, but it will sit on the rail as "1" until the unit is disabled or the rule is narrowed. Left alone.

Quality of life — five chosen by Mitchell, ✅ built and verified 2026-09-19

1. ✅ Stacks on grid and the Mac are editable from the hub. Their files live on core (Dockhand deploys over tcp); the hub resolves them and runs compose with DOCKER_HOST pointed at the node. Verified on a throwaway stack: saved on core, recreated on grid, same compose labels, nothing created on core, output followed on the page.
2. ✅ A stack's .env is editable, values masked until "Reveal and edit". A .env that breaks the compose file (a required variable removed) is refused and the old one put back.
3. ✅ A link to a container that no longer exists says so, by name.
4. ✅ "Update available" per image — on request, remembered six hours. Verified with an image whose tag had moved on (update available) beside a current one (up to date).
5. ✅ What Deploy would change: a line diff against what was last deployed from here; after a deploy it says there is nothing left to change. .env diffs name the key and never show a value.

Not chosen, still open: alert acknowledge/snooze; quiet hours and per-rule channels; push straight from Ham Lab; a type hint in the alert editor; switching off the nodes' own alert engines.

Bug bash 3 — 2026-09-20: new stacks, new containers, images, volumes, the HUD

Started from Mitchell's report — the New stack dialog is see-through in HUD, its text mixing with the page behind. Same method as before: a scratch hub with the real fleet behind it, headless Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four machines, plus a measurement in every skin of everything that floats over the page (dialogs, the question box, the widget picker, menus, the phone drawer: background alpha must be ≥ 0.9). Then scripted use of what is new on throwaway hl- stacks, containers, images and volumes: the New stack / New container dialog's edges, stacks that are not deployed, a deploy that fails, a container name that is taken, deletes beside other people's files, a two-file stack, create → deploy → restart → stop → delete on the Mac and on grid through the hub, pulls on the hub and on a node, removal guards, a container stopped on purpose, a read-only token and hostile input on every new endpoint, and a regression of Escape everywhere, terminals, Files, sensors, alerts and deep links.

Bugs — all ✅ fixed and verified by the script that found them

1. ✅ HUD: every dialog was see-through. The skin's "glass" rule (3 % cyan) was written for tiles and cards and also caught .modal — the New stack dialog, the widget picker, the Power dialog, the token dialog, the rule editor and every question. Anything that floats over the page is now solid in HUD (hairline border and glow kept) over a darker backdrop. Measured in all thirteen skins: nothing that floats is under 0.9 alpha.

2. ✅ HUD: the phone's slide-out menu was 92 % opaque, the page showing through it. Solid.

3. ✅ Deploy, Update, Restart, Stop and Down all failed on the Mac's stack (peaches-and-plums) — and on any stack held on the hub that was deployed from the hub's own shell rather than from inside Dockhand. The command was built as 'DOCKER_HOST=tcp://…' docker compose …; quoted, sh reads that not as a variable but as the name of a program to run ("not found", exit 127). Now passed through env. Verified by a whole life on the Mac and on grid from the hub: create, deploy (running there, nothing on core), restart, stop, delete.

4. ✅ A container stopped on purpose was reported as down when it has no restart policy and doesn't handle SIGTERM: docker stop then ends it with 137 (or 143, or 130), a non-zero code, which read as a crash — "Container down" would have fired for a deliberate stop. Those three codes now mean "a person", unless the kernel did the killing for want of memory (OOMKilled). A real crash (exit 3) is still down.

5. ✅ A stack with no containers offered Restart, Stop and Update, and Restart "succeeded" having done nothing. Offered only when there is something to act on; the API answers 409 "isn't deployed: there is nothing to restart".

6. ✅ One stray tap outside the New stack dialog threw away everything typed — on a phone, a compose file's worth. So did Escape. With anything typed, all three ways out (tap outside, Escape, Cancel) ask first; an untouched dialog still just closes.

7. ✅ Compose → Form → Compose silently replaced hand-edited compose text with what the form says. The form now writes the box only when the form has changed since it last did, and asks before replacing text edited by hand.

8. ✅ A question asked from inside a dialog opened underneath it (same layer, earlier in the page), so it could not be answered — found the moment the New stack dialog started asking. Questions now sit above whatever asked them.

9. ✅ Escape on a question also reached the dialog beneath it: one key cancelled the question and closed (or re-asked from) the dialog under it. Escape now answers the question and goes no further; the next Escape is the dialog's. Checked on every dialog and menu: widget picker, machine menu, rule editor, Power, New stack, and that Escape on "Delete the rule?" deletes nothing.

10. ✅ Phone: the bottom of the New container dialog could not be reached (Mitchell, from his iPhone). A dialog is a column whose parts were allowed to shrink, so instead of the dialog scrolling, the one part with its own scrollbar — the compose preview — was squeezed to 22 px and everything "fitted". And 92vh on iOS is taller than the screen with the address bar gone, so the top sat under the clock. Parts now keep their height and the dialog scrolls inside itself; it is never taller than the space between the status bar and the home indicator (safe-area insets, no vh); Create / Cancel — and any error — stay pinned at the bottom while the form scrolls. Applies to every dialog; measured at 390×844, 375×667 and sideways, and the picker, rule editor, token and Power dialogs rechecked.

Checked and fine

All 13 pages × four machines × both sizes: no page errors, failed requests, sideways scroll or tap targets under 30 px. A port of not-a-port is refused by compose before anything is written, and leaves no folder. Environment values with quotes, #, $ and nothing after the = arrive in the container exactly as typed. A deploy whose image doesn't exist shows the registry's error and is not remembered as "last deployed". A container name already taken fails the deploy and says why. Delete moves only the stack's own compose files and .env (kept 600 in the trash) and leaves a data/ folder and notes beside them alone; both files of a two-file stack go to the trash; the wrong name, or none, deletes nothing. A .env value never appears on the stack page and stays masked until asked for. A second stack of the same name is 409; an upper-case name is 400. A pull of an image that doesn't exist ends as a failed task; a pull on grid from the hub lands on grid and its output is followed. A volume or image in use can't be removed; a running container can't be removed (409); a volume needs its name twice. A read-only token gets 401 from all fourteen new write paths, through the hub as well. --force, -f, --all-tags, ..%2F.. and a home outside the known ones are refused; a 1.2 MB compose file is refused. Shaped, fleet and streamed sensor queries, /metrics, the ten alert rules, terminal typing, the file editor's discard question and service deep links are unchanged. No native browser dialogs anywhere.

Quality of life — four chosen by Mitchell, ✅ built and verified 2026-09-20

1. ✅ "Where" fits on a phone: …/stacks/core · 12 stacks instead of the whole path (which is still written out underneath). Two homes whose last two folders match fall back to the full path. Measured at 390 px: no option is wider than the box.
2. ✅ How a task ended is a badge: green finished, red failed · exit 1 with the box outlined in red, grey cancelled. Verified on a deploy of an image that doesn't exist.
3. ✅ Duplicate… on a stack: the New stack dialog opened on a copy of its compose file, named <name>-copy. The original's .env is copied on the machine (env_from), mode 600, its values never sent to the browser — checked: the secret is nowhere in the page. Anything that would collide with what is running is named before you press Create ("a container called X already exists; port 18222 is already published by X") — for every new stack, not only duplicates — and clears as you fix it. Disabled while the original has unsaved edits. Verified: copy deployed beside the original, which never stopped.
4. ✅ The form has Labels and Network. Labels one per line ($, quotes and spaces arrive intact). Network: its own, the machine's (host, which turns Ports off), or join one that already exists — written as an external network so compose doesn't make a second one. Verified on the container: both labels, attached to the chosen network only.

Found while building them, ✅ fixed:

• Cancel on a stack task cancelled nothing. It killed the sh that ran compose, not compose; and as compose still held the output pipe, the task stayed "running" and the stack "busy" until compose finished by itself — for a deploy waiting on a health check, minutes. Tasks now run in their own process group and the whole group is stopped (TERM, then KILL after 4 s); for stacks deployed inside Dockhand, where the process is not the task's child at all, the cancel goes into the container after it. Verified both ways: ended in ~100 ms, compose gone, the stack free for the next action.
• Phone: a stack's task output was squeezed to a line under the buttons (the same fault as the dialog: a scrolling column whose parts were allowed to shrink), which also put Cancel out of reach. Detail panes keep their parts' heights; services, containers, stacks and cron rechecked on both sizes.
• The Duplicate dialog and the stack page under it shared a React key, so the dialog multiplied. An undeployed stack's dot was green ("0 of 0 running"); now neutral.

Bug bash 4 — 2026-09-21: the Top tab, and the refresh rate it didn't follow

Started from Mitchell's first look at Top on his own machine (Firefox on Windows, a window 1392 CSS px wide — none of which the tab had been drawn for): "Command" printed over "Threads", the rate in Settings ignored, the symbol beside logout a box of hex digits; and make it follow the skins. Method as before, plus two things new: Firefox (Playwright's build; the GTK libraries it wants are unpacked from their .debs into ~/.cache/ms-playwright/fflibs and given to it as LD_LIBRARY_PATH, with PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=1 — nothing is installed on core), and a layout checker that measures instead of looking — every heading against its neighbour and its own text, every row against its box, every border label, at eight window sizes from 900 to 1920. Run first against the live v83 it reproduced the screenshot to the pixel ("the Command column is 11px wide" at 1392); run after, nothing, in both browsers.

Bugs — all ✅ fixed and verified by the script that found them

1. ✅ Between about 1280 and 1500 px the Command column collapsed to a letter and its heading ran under "Threads:" (itself 4 px too narrow for its own text at every size). The table's fixed columns were sized for one window. The process box now measures itself (a container query — it is a third of a wide window and all of a narrow one, so the window's width says nothing) and drops the curve, then Threads, then Command, as it narrows; the box also gets half the width rather than 44 %, and no heading can spill into the next. Stacked layout now starts at 1280, not 1100.

2. ✅ The refresh rate reached a machine only through its Dashboard page. The sampler on each machine was told the rate by an open Dashboard and nothing else — so with the rate at 250 ms, grid, wyse and the Mac were still sampling at 1 s (measured, live), and Top, Processes, Disks and every widget of theirs "ignored" the setting until somebody opened that machine's Dashboard. The rate is now the fleet's: saved with the UI prefs on the hub, which sets its own sampler and passes it to every machine (PUT /api/prefs/sampler, remembered in each state.json), and says it again at start-up for a node that restarted or joined since. Verified: Settings 2 s → 250 ms moves the sampler with no Dashboard visit.

3. ✅ Top read the process table at most once a second whatever the rate — a clamp of my own, exactly what REFRESH promises not to do. It reads at the rate: 20 reads in 5 s at 250 ms, 10 at 500 ms. (59 frames a second with 390 rows at 250 ms, no stalls, both browsers — the rows are only redrawn when the table is read.)

4. ✅ Top's − rate + changed the rate for half a minute. It set the browser's copy and never saved it, so the next sync with the hub put it back. It saves — which is also what moves the samplers — survives a reload, and each end is disabled at the limit.

5. ✅ A viewer whose clock is wrong froze the curves. New points were timed against the machine's timestamps: a browser clock 7 minutes slow waited 7 minutes for its next point (Top and the Dashboard); 7 fast, and it added one per tick. Timed on the browser's own clock now. Verified with the page's clock shifted both ways: 5 points in 10 s at a 2 s stride, both pages, both browsers.

6. ✅ Every other point was being skipped. "Has a stride passed?" was asked with >= of a timer that fires a few ms early half the time — so at a 2 s rate and a 2 s stride a curve gained 3 points in 10 s, not 5, and its time axis was quietly wrong. The same question on the server kept every bucket one sample long (14 s points at a 12 s stride). Both now allow half a tick of slack and keep to the stride's own beat; the server times from the last point kept, so a stalled or sleeping machine just keeps the next sample it gets. Unit-tested with jittered samples: 240 samples → exactly 40 points, 12 000 ± 100 ms apart, each the mean of its own six.

7. ✅ The power symbol was a typed ⏻ (U+23FB, Unicode 9) — not in Windows' fonts, so Firefox there drew the missing-glyph box. Drawn as an SVG like the other icons. Every other symbol the console types was checked against the same risk: all older.

8. ✅ A handful of samples was stretched into a bar chart. A machine ten seconds into its history filled the whole CPU box with five fat columns. The box is now always the whole history window — a young curve draws in from the right, as btop's does, and a full one fills it edge to edge, at any window (checked at 10 min and at 1 min).

9. ✅ Idle curves vanished: a core at 10 % rounded to no dots at all in its mini graph. Anything above zero lights the bottom row, which is also btop's rule.

10. ✅ Found by the deploy of fix 2 itself: telling the machines once is not enough. The hub restarted first and told three nodes that did not yet have the endpoint (404); they then updated and carried on at 1 s. The same would happen to any node that is off or restarting when a setting changes. The hub now keeps the fleet in step: fifteen seconds after it starts and every two minutes after, it asks each machine for its sampling rate and history window and corrects only what differs (logged as fleet: grid had ms=1000, now 250).

11. ✅ The net graph redrew its own past every second (Mitchell, two screenshots a second apart: one tall spike in the first, a comb of medium ones in the second). That box is about 75 dot-columns wide and holds 300 points, and each column drew the nearest point — one in four, and a different one in four each time the curve advanced, so spikes came and went and the whole shape changed while the scale stayed put. The CPU graph did the same on a phone. Now a column draws the peak of all the points in it, and columns are cut from the data, not from the box (web/src/graphcols.ts): a finished column never changes, the graph only slides left by whole columns. Measured by reading every column's height off the canvas once a second under bursty traffic: on the live build the past changed shape in 18 of 47 frame pairs; after, 0 of 46 — and 0 of 45 on a young history. npm test (now part of make test) scrolls a spiky curve through both rules: nearest- point shows its three spikes as "900,500", then "700", then none; the new rule shows all three, at their heights, on every frame. The scale label now covers what is drawn (the window), not everything kept.

Skins

✅ Top now wears the skin. The ground is the skin's terminal background and every curve and meter is built from its sixteen terminal colours (chosen to be read on exactly that ground), the rest from the usual variables — so the dark skins look like btop and the light ones like btop on paper, and a skin arriving from another device repaints the page in place. Measured in all thirteen: text ≥ 4.5 : 1, everything else ≥ 3 : 1, dots ≥ 2.2 : 1. Two failed — process names in Latte (2.75) and Solarized Light (2.62), where a terminal green is too pale for small type — and use --ok, the green each skin was already measured readable with.

Checked and fine

Sort by pid, name, mem and cpu, and reverse; filter by name, command, user and pid, with the count showing matched/total and an empty result saying so; a selection that holds while the table is re-read under it; terminate, on a throwaway sleep, asks by name and pid, ends it, and the footer stops offering a process that is gone. Sampler values outside 100 ms–60 s are refused, and a rate slower than the collector's 10 s samples at 10 s. All 14 pages at his window size and on a phone, in both browsers: no errors, no overflow.

Bug bash 4, later — 2026-09-21: the SMART table outside its card

Mitchell, from the Disks page of hamtv with SMART open: the attribute table's rows ran out through the right-hand side of the card they belong to, the rules and the Thresh column drawn over the page behind.

12. ✅ A table cannot be narrower than its widest unbreakable cell, and one of hamtv's was 130px wide. .num sets white-space: nowrap, and a SMART raw value is not always a number — the temperature attribute reads 34 (Min/Max 28/39). That one cell set the table's minimum width above the card's, so it spilled. Three changes, in this order, because the first two each looked worse than the bug in their own way:
    • the raw column wraps — but overflow-wrap: anywhere broke 4295163906 across two lines as "429516 / 3906", which reads as two numbers. It wraps at spaces only now;
    • a disk card is given room for six columns of SMART (minmax(420px, 1fr) rather than the shared 320px), and the cells' side padding goes 6px → 4px, which together is the ~40px it was short by;
    • if some drive ever reports a raw value long enough to still not fit, the table scrolls inside its card rather than out of it. Verified on all four machines at six widths from 1721px down to a phone, in Chromium and Firefox: nothing leaves its card, no number is broken across lines, and the fallback scrollbar is never reached. The same measurement was run over every other page's cards and boxes — Dashboard, Top, Docker, Alerts, Machines and the rest — and found nothing.

Bug bash 5 — 2026-09-21: navigation, and the fleet setting nobody had set

A sweep of the whole console after a week of changes, not of any one page: every page on every machine at four widths in Chromium and Firefox, then the everyday flows, then the URL. Measured rather than looked at — a heading against its neighbour and its own text, a box against what it holds, a request against the page that should have made it.

Bugs — all ✅ fixed and verified by the script that found them

1. ✅ A link to a page of the hub reloaded onto whichever machine you last had open. The hub is null internally, so parseHash()?.machine ?? sessionStorage… read straight past it to the remembered machine: open #/hub/docker/stack:gitea after looking at grid, reload, and you were on #/grid/docker. The URL wins when it has an opinion, and "hub" is an opinion; the remembered machine is for a bare address only. Every link shape now survives a reload, and a bare address still lands where you left off.

2. ✅ From Settings, Machines or Alerts, a link to a machine's page arrived at its Dashboard. Those three pages only exist on the hub, so moving to a machine bounces off them — but the bounce was decided from inside the machine change, which read the page being left, and landed after the URL's own page had been set. It is judged now once the machine and the page have both settled. #/hub/settings → #/grid/disks arrives at Disks; #/grid/settings still falls back to that machine's Dashboard.

3. ✅ A page being navigated away from briefly adopted the next page's selection, and asked about it: leaving #/grid/services/ssh.service for #/hub/docker/stack:gitea, Services — still mounted for a moment — opened a log socket for a service called stack:gitea, on grid. Firefox says so in the console; Chromium swallows it. A page now reads the URL's selection only while the URL still points at that page. Checked over every hop between the five pages that keep a selection in the URL, in both browsers.

4. ✅ The hub only held its machines to a sampling rate if somebody had been into Settings. The re-assert loop read the rate out of the saved UI prefs, which are written on change and not before — so on a hub where the rate had never been touched, a node left on a different rate was never pulled back. It compares against what the hub itself is running, which always exists. Verified by setting a node behind the hub's back: corrected within seconds.

5. ✅ "Everything is up to date." sat in a 20px sliver of a card, flush to its border, where the package list it shares the card with is meant to be.

6. ✅ A terminal of the hub's was asked for on a node. Found in Firefox against the live console, which says out loud what Chromium swallows: wss://…/api/nodes/grid/terminals/ adfdeecb/attach, where adfdeecb was one of the hub's — grid had no terminals at all. A pane builds its address when it connects, from whichever machine the console is pointed at now; switch machine while one is reconnecting and the reconnect goes to the new one. The address is fixed when the pane opens. The same was true of a service's log socket, and is fixed with it. Verified by walking the console around the fleet with a terminal open and checking every socket against the machine that actually owns what it asked for.

Checked and fine

Machine titles on every page and against an offline or never-reached machine; the colour a machine wears stays with the name it was added under, not its hostname. Files: create, open, edit, save, rename, delete. Services with its unit and log, cron jobs, containers and stacks, alert rules edited and put back, the terminal, the widget gallery. Login, a wrong password, a session cleared under an open page, logging back in onto the page you were on, and logout leaving nothing behind. The sampler, window and UI prefs endpoints under nonsense and hostile input, and a read-only token refused by all of them. The public demo page and installer. The Top tab on all four machines — including grid's sixteen cores and thirteen filesystems, which scroll inside their boxes — its rate control, its filter, and three skins repainting it. hamtv's eMMC really is called hB8aP>; that is what the device reports.

Firefox earns its place in this: three of the six above were invisible in Chromium, which swallows a failed WebSocket where Firefox writes it to the console. One thing it reports is not a fault — a page torn down by navigation while a terminal's socket is still shaking hands logs "can't establish a connection". Sitting still on a page produces none, the address is right, and the terminal belongs to the machine asked; sockets.cjs checks that ownership rather than the noise.