Mitchell-JohnsonandClaude Fable 5.1
668e716601
The process table's columns fit the box they are in, the rate control saves, curves are timed on the viewer's clock and keep to the stride, the power icon is drawn, and a young history draws in from the right instead of being stretched. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
500 lines
40 KiB
Markdown
500 lines
40 KiB
Markdown
# Bug bash — 2026-09-18
|
||
|
||
Fix pass 2026-09-18: every item carries its outcome; ✅ resolved and verified live, ➖ no change with reason. Found by a headless Chromium pass (iPhone and
|
||
1440px desktop viewports) over every page on the hub and on each remote machine, an
|
||
offline-node scenario (wyse stopped for the test, then restarted), direct API probing
|
||
with hostile inputs, and a code read. Ordered by severity within each group.
|
||
|
||
## Security / robustness
|
||
|
||
1. ✅ **RESOLVED 2026-09-18** — only regular files are readable; the byte cap applies to what is sent, not to `stat`. `/dev/zero`, `/dev/null` and FIFOs answer 400 in 30 ms; `/proc/cpuinfo` still reads. **Reading a special file hangs the request forever.** `GET /api/files/read?path=/dev/zero`
|
||
never ends: the `max` cap compares against `stat` size, which is 0 for device files,
|
||
so the cap never triggers and the handler streams `/dev/zero` until the client gives
|
||
up. Same for FIFOs (blocks on open) and `/proc/self/mem`. Opening one in the Files
|
||
viewer would hang the tab. Needs: refuse non-regular files for read/edit, and cap the
|
||
copy with `io.LimitReader` regardless of `stat`.
|
||
2. ✅ **RESOLVED** — per-client buckets (first X-Forwarded-For hop or the peer) with the same 5-strike schedule, plus a global backstop of 100 failures per 5 min → 1 min. Verified: core's public IP locked while grid logged in fine. **Login lockout is global, not per client.** Five bad passwords from anyone on the
|
||
internet lock the real user out for up to 10 minutes (`auth.Limiter` is one bucket).
|
||
The bash locked the hub for the full 10 minutes twice. Also locks `POST /api/tokens`,
|
||
so hub pairing fails during a lockout. Needs per-IP buckets with a modest global cap.
|
||
3. ✅ **RESOLVED** — server rejects refreshMs outside 100–60000 and non-identifier themes (400); client ignores out-of-range rates from any source. **UI prefs from the server are not validated.** A stored `refreshMs` of `-5` made the
|
||
client call `setInterval(-5)` on every live view: a request storm that took the
|
||
browser to `ERR_INSUFFICIENT_RESOURCES` and hammered the daemon. `theme` is validated
|
||
only by falling back. Needs range checks on load (and on save server-side).
|
||
4. ✅ **RESOLVED** — every response carries X-Frame-Options DENY, CSP frame-ancestors none, nosniff, no-referrer. **No anti-framing / CSP headers.** Only HSTS (from NPM). A root-capable console should
|
||
send `X-Frame-Options: DENY`, `Content-Security-Policy: frame-ancestors 'none'`,
|
||
`Referrer-Policy: no-referrer`, `X-Content-Type-Options: nosniff` on every response.
|
||
5. ✅ **RESOLVED** — kernel threads (empty cmdline) and the daemon's own PID are refused. **Signals can be sent to kernel threads.** `POST /api/procs/2/signal` (kthreadd)
|
||
returns 204. Harmless in practice, but the guard should refuse PIDs with an empty
|
||
cmdline (kernel threads) and our own PID.
|
||
6. ✅ **RESOLVED** — unit runs with `SupplementaryGroups=systemd-journal adm` so journal reads need no sudo, and the installer drops `/etc/sudoers.d/hamlab-quiet` (no syslog/pam session for the read-only polling commands). Verified: 0 sudo lines in 60 s of 250 ms polling (was 839/10 min). Applied on all four hosts. **Journal spam from the daemon's own sudo calls.** Every sudo'd `journalctl`, `smartctl`
|
||
or `pwrstat` writes three `pam_unix(sudo:session)` lines into the journal *under
|
||
`hamlab.service`*: 839 lines in 10 minutes at a 250 ms refresh, and the journal is
|
||
already 4 GB. It also buries Ham Lab's own log entries. Fix options: add the service
|
||
user to `systemd-journal` (and `adm`) so journal reads don't need sudo, and/or a
|
||
sudoers `Defaults!/usr/bin/journalctl !syslog, !pam_session` line in the installer.
|
||
7. ✅ **RESOLVED** — server answers 409 unless `overwrite=1`; the UI asks "Replace it?" first. **Uploading over an existing file silently overwrites it.** No confirmation, no
|
||
"replace?" prompt. Same for "+ file" when the name exists (server refuses; UI just
|
||
shows the error, fine) — upload is the gap.
|
||
8. ➖ **NO CHANGE** — Go's mux redirects the unclean path to `/etc/passwd`, which is an ordinary deep link the SPA must serve; there is no API prefix left by the time the static handler sees it. Harmless. **`/api/<anything>/../../x` falls through to the SPA** and returns `index.html` with
|
||
200 instead of a JSON 404. Harmless, but sloppy for an API prefix.
|
||
|
||
## Functional bugs
|
||
|
||
9. ✅ **RESOLVED** — 401 carries the server message; the form shows "wrong password" (verified headless). **Wrong password gives no feedback at all.** The 401 is thrown as `Unauthenticated`
|
||
with an empty message, and the login form only renders `err` when non-empty. The
|
||
button just re-enables. Reproduced on phone and desktop.
|
||
10. ✅ **RESOLVED** — hub-only pages bounce to the dashboard on switch (verified headless). **Switching machines while on Settings or Machines keeps showing the hub's page.**
|
||
`switchMachine` only resets the view for Machines; Settings stays, so "mac" shows
|
||
core's account/token/about panels. Both are hub-only pages and should bounce to the
|
||
dashboard (or be clearly labelled as hub pages).
|
||
11. ✅ **RESOLVED** — escaped; no console error. **Machines add-form `pattern` is an invalid regex in Chromium.** `[a-z0-9][a-z0-9-]*`
|
||
fails under the `v` flag ("Invalid character class"), logs a console error and the
|
||
browser skips validation. Needs `[a-z0-9][a-z0-9\-]*`.
|
||
12. ✅ **RESOLVED** — refresh from the newest repodata under libdnf5/dnf caches (Mac: 9 d), last upgrade from `dnf history` (194 d), label reads "from dnf history". **Updates tab on dnf hosts shows nonsense.** Mac: "Index refreshed 561d ago" (mtime of
|
||
`/var/cache/dnf`, which Fedora 41+ no longer uses; `libdnf5` is the live cache) and
|
||
"Last upgrade never · from apt history" (label and source are apt-specific). Needs
|
||
`dnf history` for last upgrade, metadata timestamps for refresh, and neutral labels.
|
||
13. ✅ **RESOLVED** — journal read incrementally by cursor, timers from one `systemctl show`, services' two calls run in parallel. Cron 3.6 s → 0.08 s, services 0.6 s → 0.05 s, with last-run and next-run data intact. **Cron overview is slow: 3.6 s per request on core.** At any refresh rate the page
|
||
is constantly busy. The 30-day, 20 000-line journal scan for "last run" is rebuilt
|
||
every 5 s; timers make three `systemctl` calls. Needs an incremental journal cursor
|
||
and a single `systemctl` call. Services list is 0.6 s for the same reason (two calls).
|
||
14. ✅ **RESOLVED** — heap profile showed a 48 MB `bytes.Buffer` from buffering `journalctl` dumps; those are now streamed line by line, plus a 160 MB soft memory limit for the GC. RSS 22 MB fresh, 27 MB after 25× cron/logs/procs/services (was 180 MB). pprof stays available behind the login at `/api/debug/pprof/`. **Daemon memory grows.** RSS on core went 22 → 85 → 216 MB during the day (136 MB at
|
||
rest after the bash). Suspects: per-pid caches in the process table (`procInfo` only
|
||
prunes on the next walk), task output buffers, SMART cache, xterm scrollback rings
|
||
that are never freed for exited sessions. Needs a heap profile under load.
|
||
15. ✅ **RESOLVED** — `--dirty` dropped from the version and deploys now follow commits; binaries report e.g. `v0.1.0-38-gc1805f6`. **Version string is always `-dirty`.** `make deploy` builds before the commit, so
|
||
every installed binary reports e.g. `v0.1.0-32-ga372378-dirty`, and the Machines page
|
||
compares with `-dirty` stripped. Build after commit, or tag releases.
|
||
16. ✅ **RESOLVED** — hash routing `#/<machine|hub>/<view>`: reload restores machine and tab, Back/Forward work, links can point at a page on a machine (verified headless). **View resets to Dashboard on reload; no deep links; browser Back leaves the app.**
|
||
The selected machine survives a reload (sessionStorage) but the tab does not, and
|
||
there is no URL routing at all. Needs hash or path routing (`/#/grid/services`).
|
||
17. ✅ **RESOLVED** — newest first. **Cron log tab lists oldest first and starts at the top.** 200 lines from Sep 10 with
|
||
the newest at the bottom, off-screen. Should show newest first or scroll to the end.
|
||
18. ✅ **RESOLVED** — "Nothing matches." **Processes: empty filter result shows nothing.** No "nothing matches" message.
|
||
19. ✅ **RESOLVED** — exited shells remove themselves 10 minutes after exit (verified live: seeded 17:53:56, gone at 18:03:59 with no daemon restart in between). Found and fixed alongside: a daemon restart used to re-spawn *exited* shells as fresh running ones; only running shells are persisted now. Idle running shells are deliberately kept. **Terminals accumulate.** Nothing ever reaps exited or idle sessions; the bash left
|
||
several "bug-bash-term" shells on the hub until deleted by hand. Exited sessions
|
||
should auto-remove after a while; idle ones at least be counted somewhere.
|
||
20. ✅ **RESOLVED** — a node answers 409 "busy" when a package task is running or a terminal is attached; the Machines page offers Force, `hamlab push-nodes --force` / `make push-nodes-force` do the same. Verified: 409 with a terminal attached on grid, 200 with force. **Self-update restarts without warning running work.** A push to a node kills its
|
||
open terminals and any running task (upgrade in progress). The hub should refuse or
|
||
warn when the node reports a running task or attached terminals.
|
||
21. ✅ **RESOLVED** — PathError and LinkError are unwrapped to the bare reason. **Error text leaks raw Go errors.** e.g. rename shows "rename /tmp/x /tmp/y: no such
|
||
file or directory". Fine for a lab, ugly for a product; strip the syscall prefix.
|
||
|
||
## Mobile / UX
|
||
|
||
22. ✅ **RESOLVED** — on coarse pointers every button, chip, icon button, link, header and select is at least 36 px tall (verified: no control under 32 px on the dashboard or processes pages). **Tap targets under 28 px on phone:** header ⏻ (25), logout (22), customize ▲ ▼ hide
|
||
(23), terminal leaf ✎ ✕ (24), chips (25), table headers (27). Apple's guideline is
|
||
44 pt. Systemic: the small `.btn`/`.chip`/`.icon-btn` sizes need a touch variant.
|
||
23. ✅ **RESOLVED** — the dialog asks for the machine name (`mac`); the node still checks its own hostname under the hood. **Power dialog on the Mac asks you to type `5ce91ee54af6`.** The confirmation uses
|
||
the node's hostname, which on the Mac is the unset default. Should offer the machine
|
||
name (`mac`) as the confirmation word on remote machines.
|
||
24. ✅ **RESOLVED** — down interfaces collapse to one line marked "down"; modules with nothing to show (fans, UPS, temps, disks) hide unless customizing. **Dashboard shows dead interfaces and empty modules.** wlp1s0 (down) gets a full
|
||
sparkline block of flat lines; "Fans: no fan sensors" is a permanent empty card on
|
||
hosts without fans. Down interfaces should collapse to one line; empty modules should
|
||
auto-hide unless customizing.
|
||
25. ✅ **RESOLVED** — "load 1.39 1.56 1.55" with a tooltip. **Header load average is unlabelled** ("1.39 1.56 1.55" floating in the header).
|
||
26. ➖ **NOT FIXING** — the list is intentionally a plain table; the menu anchors inside the row. Cosmetic taste, not a defect. **Files list on desktop is a narrow strip in a sea of empty space** for small
|
||
folders; the row menu opens off the right edge of the row. Cosmetic.
|
||
27. ✅ **RESOLVED** — after a 502 the client leaves that machine alone for 5 s: 2 requests in 10 s at a 250 ms refresh (would have been ~80), badge and message intact. **Offline machine keeps polling every view at the refresh rate**, producing a 502
|
||
per request per page (and console errors). The UI copes — badge, message — but the
|
||
hub should back off to the prober's cadence while a node is down.
|
||
28. ✅ **RESOLVED** with 6. **Services logs are noise.** The default view of `hamlab.service` logs is the sudo
|
||
session spam from bug 6, three lines per poll.
|
||
|
||
## Confirmed working under abuse (for the record)
|
||
|
||
Path traversal in files, services, cron, ZFS, procs and updates all rejected. `/etc`
|
||
and `/` protected from delete/rename as root. Unit, pool, dataset, snapshot and
|
||
package names validated. Token issue refused with a wrong password. Oversized prefs
|
||
rejected. Duplicate machine add refused. Unicode + spaces in file names round-trip.
|
||
Upload byte-exact. Terminal scrollback survives switching. Offline node: switcher and
|
||
header show it, pages show the reason. Remote terminals, logs, tasks and files through
|
||
the proxy all work on all three nodes.
|
||
|
||
|
||
# Bug bash — 2026-09-19
|
||
|
||
Found by driving a scratch instance (same binary, its own state, real fleet behind it) in
|
||
headless Chromium at 1440 px and iPhone size: a sweep of every page on every machine for
|
||
page errors, failed requests, overflow and small tap targets; then scripted use of every
|
||
feature — terminal tabs, Files in a throwaway folder, Processes (killing a process the
|
||
test spawned), Monitors customisation and widgets, Services, Cron, Updates, Disks,
|
||
Machines, Settings, embeds, machine switching, session loss — and a hostile pass over the
|
||
API. Host-changing actions (service control, cron edits, upgrades, power) were opened but
|
||
not executed. Every fix below was re-checked by the same script that found it.
|
||
|
||
## Bugs — all ✅ fixed and verified
|
||
|
||
1. ✅ **One widget with a bad pattern silenced every custom widget.** They share one stream;
|
||
`fan.[` made the whole query a 400, so all of them showed "…" forever with no reason
|
||
given. The picker now asks the API before adding and shows its answer; a bad pattern
|
||
already saved shows a notice instead of dots.
|
||
2. ✅ **A saved layout of the wrong shape could not be loaded** ("could not load layout",
|
||
and the page could no longer be customised). `normalize()` now accepts anything.
|
||
3. ✅ **A lost session left pages saying "unauthenticated"** for up to 30 s. A 401 from the hub
|
||
now returns to the login form at once, and logging in returns to the same page. (A 401
|
||
relayed from a managed machine is that machine rejecting the hub, and doesn't.)
|
||
4. ✅ **The machine-name rule in "Add a machine" was an invalid regular expression** under the
|
||
browser's `v` mode, so it validated nothing and logged a console error on every render.
|
||
5. ✅ **Two terminals opened in the same minute got the same name** ("shell 09:45" twice).
|
||
Now "shell 09:45 (2)".
|
||
6. ✅ **Names had no length limit** — a 328-character terminal tab, a 5000-character token
|
||
name. 64 characters, refused with a reason; token names also refuse `/ \ < > "`.
|
||
7. ✅ **"+ file", "+ folder" and Rename accepted `../x` and `a/b`**, quietly acting outside the
|
||
folder on screen. A name is now one path segment; Move… is how things go elsewhere.
|
||
8. ✅ **Unsaved file edits were thrown away without a word** when you picked another page or
|
||
machine, or closed the tab. All three now ask; Cancel keeps you where you were.
|
||
9. ✅ **A mistyped path in Files was remembered**, so a reload landed on the error. Only a
|
||
folder that opened is remembered.
|
||
10. ✅ **Escape closed nothing**: the machine menu, the file menu, the Power dialog, the widget
|
||
picker. It closes all four.
|
||
11. ✅ **Reset on Monitors removed every custom widget with no confirmation.** It asks, and
|
||
says how many.
|
||
12. ✅ **A few bytes of query bought 27 MB of answer** (`{"k0":"*",…×400}` with `full=1`) — and a
|
||
read-only token is public on the demo page. A query may return 10 000 values (thirty
|
||
whole machines); past that it is a 400 that says so.
|
||
13. ✅ **No ceiling on live streams**: 300 from one token were accepted. 32 per client address,
|
||
512 in all, then a 429 that says so; closed streams free their places.
|
||
14. ✅ **Anything under `/embed/` lost the anti-framing headers**, not just the widget page.
|
||
Only `/embed/widget` is frameable.
|
||
15. ✅ **The new ZFS sensors ran three `sudo` commands a minute on every machine** — including
|
||
`core`, which has the tools and no pool — and each wrote three lines to the journal:
|
||
most of `hamlab.service`'s log was that. Pools are now read with one unprivileged
|
||
`zpool list`; a source that finds nothing is asked again ten times less often; and
|
||
the installer's quiet list covers `zpool|zfs list|status`.
|
||
16. ✅ **Phone: the filter boxes on Processes and Files, the path box and the "/" crumb were
|
||
under 30 px.** 36 px like every other control.
|
||
17. ✅ **Disks → datasets on a phone: rows ended 50 px before their own cells** when scrolled
|
||
sideways (`min-width` was less than the columns add up to).
|
||
|
||
Also: tab titles now say where you are ("Services · grid — Ham Lab"), so history and open
|
||
tabs can be told apart.
|
||
|
||
## Checked and fine
|
||
|
||
Terminal focus after every switch/close/return; scrollback after reload; exited shells;
|
||
30 tabs with the + still reachable; re-fit on resize; Back/Forward; unknown machine or page
|
||
in the URL; wrong password; XSS through terminal, token and file names (all rendered as
|
||
text); hidden files, filter, symlinks (broken ones marked), binary files, upload and
|
||
replace, folder delete needing the typed name, root toggle; process sort, filter and
|
||
Terminate (asks first); themes; password-change validation; duplicate or unreachable
|
||
machine; hub terminals not leaking into another machine's page; CORS only on sensor paths;
|
||
token list never exposing hashes; cookie flags; deep nesting, empty bodies, odd methods
|
||
and `every=` values on the sensor API; phone drawer, key bar and picker.
|
||
|
||
## Quality of life — all ✅ done 2026-09-19, each verified in the browser
|
||
|
||
1. ✅ **Selections are in the URL.** `#/grid/services/ssh.service`, `#/hub/cron/job:…`,
|
||
`#/hub/files/%2Fetc`: reload keeps them, links carry them, leaving a page drops them.
|
||
2. ✅ **Browser Back with unsaved file edits asks**, like page and machine changes do.
|
||
3. ✅ **No more browser `prompt()`/`confirm()` boxes** — all 30 are in-app dialogs (`askYes`,
|
||
`askText` in `ui.tsx`): Enter submits, Escape or a tap outside cancels, destructive
|
||
ones are red and say what they do ("Close", "Delete"), and they fit a phone.
|
||
4. ✅ **Token creation is one dialog**: name, patterns, and the machine's own sensor families
|
||
as chips to click.
|
||
5. ✅ **Widgets can be edited** — sensor, type and title — in place, from Customize.
|
||
6. ✅ **The Widgets gallery has a search** over IDs, names and labels (so "gitea" finds its
|
||
container's sensors).
|
||
7. ✅ **Status sensors keep a history** (300 readings each: an hour of container memory, two
|
||
days of drive wear), so their sparklines have a past.
|
||
8. ✅ **A stream that fails says why**: "the token was rejected", "too many live streams are
|
||
open", "that machine is unreachable", "can't reach Ham Lab" — in embeds and on Monitors.
|
||
9. ✅ **Pattern widgets get readable default titles**: `cpu.core.*.usage` → "CPU core usage".
|
||
10. ✅ **Terminal tabs reorder** — drag on a desktop, ‹ › on a touch screen, remembered per
|
||
machine in the browser — and **Alt+Shift+←/→** switches tabs, **Alt+Shift+T** opens one,
|
||
with the keyboard left in the terminal each time.
|
||
11. ➖ **"Log out everywhere"** — built, then removed the same day at Mitchell's word: no need
|
||
for it on a single-user console.
|
||
12. ✅ **Real clipboard shortcuts in the terminal**, from a physical keyboard: **Ctrl+C copies
|
||
when text is selected** (and is still ^C when nothing is — the selection clears after a
|
||
copy, so a second Ctrl+C interrupts), **Ctrl+V pastes**, Ctrl+Shift+C / Ctrl+Shift+V do
|
||
the same unconditionally, Ctrl+Insert / Shift+Insert too. It rides the browser's own
|
||
copy and paste events, which xterm already answers, so it works over plain http on the
|
||
LAN where `navigator.clipboard` does not exist; bracketed paste is honoured.
|
||
|
||
|
||
# Bug bash 2 — 2026-09-19 (evening): Alerts, Docker, Stacks, skins, the installer
|
||
|
||
Same method as the morning: a scratch instance with the real fleet behind it, headless
|
||
Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four machines (errors,
|
||
failed requests, overflow, tap targets) plus a contrast measurement of every skin; then
|
||
scripted use of the new features on throwaway containers and stacks — a multi-file stack, a
|
||
file edited behind the editor's back, two actions at once, a chatty log, odd container
|
||
names, containers stopped and removed while open, every kind of bad alert rule, a rule's
|
||
whole life (fire, narrow, pause, delete, survive a restart), token scope and hostile paths
|
||
on the new APIs — and a regression of terminals, Files, widgets, deep links and colours.
|
||
|
||
## Bugs — all ✅ fixed and verified by the script that found them
|
||
|
||
1. ✅ **Deploying a stack made of several compose files dropped all but the first.** A stack
|
||
deployed with `-f compose.yaml -f compose.override.yaml` was redeployed from
|
||
`compose.yaml` alone, so everything the override set silently vanished from the new
|
||
container. Two causes: only the first file was used, and the file list was read from
|
||
`docker ps`, which joins labels with commas — the list is itself comma-separated, so it
|
||
arrived truncated. Labels now come from `inspect`; every file is resolved, shown as a
|
||
chip, editable, validated together and passed to compose in order.
|
||
2. ✅ **A compose file changed on disk after you opened it was silently overwritten** (Dockhand,
|
||
or a second tab). The editor now hands back the checksum it was given; a mismatch is a
|
||
409 that says so, keeps your text, and offers to reload.
|
||
3. ✅ **Stacks on a machine without Docker answered 500.** An empty list.
|
||
4. ✅ **Every hub restart re-fired every firing alert** — and every deploy is a restart, so a
|
||
phone would have been told about the same failed service on each one. What is firing is
|
||
now kept in `state.json` and carried on, with its original start time.
|
||
5. ✅ **An alert whose rule was paused, deleted or narrowed just vanished**, leaving "firing" in
|
||
the history forever and no word to the phone. It is now recorded (and sent) as cleared,
|
||
with the reason: "rule paused", "rule deleted", "no longer watched".
|
||
6. ✅ **Nonsense rule values were accepted**: `null`, an object, `cpu.usage > ninety` (which can
|
||
never be true, and would never say so), `true` with `<`. Refused, each with its reason.
|
||
7. ✅ **No limit on rules** — 3000 were accepted, each checked on every machine every 15 s. 200.
|
||
8. ✅ **Logs of a container that no longer exists answered 500.** 404.
|
||
9. ✅ **Four skins had text under 3:1 contrast**: Nord's red; Catppuccin Latte's muted text and
|
||
green; Solarized Dark's body text, muted text and red; Solarized Light's body text,
|
||
muted text, accent and green. Each colour kept its hue and was moved to a readable value
|
||
(measured: all thirteen skins now pass, body text ≥ 7:1).
|
||
10. ✅ **Phone: the alert toggles (28 px) and the colour swatches (26 px) were under 30 px.**
|
||
11. ✅ **The public installer had fallen four builds behind the fleet** (v65 published, v69
|
||
running), so `curl … | sh` on an up-to-date machine "upgraded" it backwards. `make
|
||
deploy` now republishes, and the installer compares versions: an older build is called
|
||
older, defaults to No, and unattended runs refuse it without `HAMLAB_ALLOW_DOWNGRADE=1`.
|
||
|
||
## Checked and fine
|
||
|
||
A second stack action while one runs (409); `../` and encoded tricks in stack names;
|
||
shell metacharacters in container names, `since` and actions; `rm` / `kill` / `down` are
|
||
not actions; a read-only token is refused on every Docker, Stacks and Alerts endpoint,
|
||
directly and through the hub; container names with dots and underscores still get live
|
||
CPU and memory; a log at five lines a second follows with no gaps or duplicates; a
|
||
container stopped or removed while open updates itself; an alert rule's XSS name renders
|
||
as text; per-machine firing, the rail badge, webhook and ntfy delivery; terminals, Files,
|
||
widgets over container sensors, deep links, tab titles and machine colours all unchanged.
|
||
No native browser dialogs anywhere; no page errors on any of 100 page loads.
|
||
|
||
## Found by the alerts themselves
|
||
|
||
`core` has one real alert firing: **`zfs-import-cache.service` fails at every boot** — `core`
|
||
has the ZFS tools and no pool, so there is nothing to import. Harmless, but it will sit on
|
||
the rail as "1" until the unit is disabled or the rule is narrowed. Left alone.
|
||
|
||
## Quality of life — five chosen by Mitchell, ✅ built and verified 2026-09-19
|
||
|
||
1. ✅ **Stacks on `grid` and the Mac are editable from the hub.** Their files live on `core`
|
||
(Dockhand deploys over tcp); the hub resolves them and runs compose with `DOCKER_HOST`
|
||
pointed at the node. Verified on a throwaway stack: saved on `core`, recreated **on
|
||
`grid`**, same compose labels, nothing created on `core`, output followed on the page.
|
||
2. ✅ **A stack's `.env` is editable**, values masked until "Reveal and edit". A `.env` that
|
||
breaks the compose file (a required variable removed) is refused and the old one put back.
|
||
6. ✅ **A link to a container that no longer exists says so**, by name.
|
||
9. ✅ **"Update available" per image** — on request, remembered six hours. Verified with an
|
||
image whose tag had moved on (update available) beside a current one (up to date).
|
||
10. ✅ **What Deploy would change**: a line diff against what was last deployed from here;
|
||
after a deploy it says there is nothing left to change. `.env` diffs name the key and
|
||
never show a value.
|
||
|
||
Not chosen, still open: alert acknowledge/snooze; quiet hours and per-rule channels; push
|
||
straight from Ham Lab; a type hint in the alert editor; switching off the nodes' own alert
|
||
engines.
|
||
|
||
# Bug bash 3 — 2026-09-20: new stacks, new containers, images, volumes, the HUD
|
||
|
||
Started from Mitchell's report — the New stack dialog is see-through in HUD, its text
|
||
mixing with the page behind. Same method as before: a scratch hub with the real fleet behind
|
||
it, headless Chromium at 1440 px and iPhone size. A sweep of all 13 pages on all four
|
||
machines, plus a measurement **in every skin** of everything that floats over the page
|
||
(dialogs, the question box, the widget picker, menus, the phone drawer: background alpha
|
||
must be ≥ 0.9). Then scripted use of what is new on throwaway `hl-` stacks, containers,
|
||
images and volumes: the New stack / New container dialog's edges, stacks that are not
|
||
deployed, a deploy that fails, a container name that is taken, deletes beside other
|
||
people's files, a two-file stack, create → deploy → restart → stop → delete **on the Mac
|
||
and on `grid` through the hub**, pulls on the hub and on a node, removal guards, a
|
||
container stopped on purpose, a read-only token and hostile input on every new endpoint,
|
||
and a regression of Escape everywhere, terminals, Files, sensors, alerts and deep links.
|
||
|
||
## Bugs — all ✅ fixed and verified by the script that found them
|
||
|
||
1. ✅ **HUD: every dialog was see-through.** The skin's "glass" rule (3 % cyan) was written
|
||
for tiles and cards and also caught `.modal` — the New stack dialog, the widget picker,
|
||
the Power dialog, the token dialog, the rule editor and every question. Anything that
|
||
floats over the page is now solid in HUD (hairline border and glow kept) over a darker
|
||
backdrop. Measured in all thirteen skins: nothing that floats is under 0.9 alpha.
|
||
2. ✅ **HUD: the phone's slide-out menu was 92 % opaque**, the page showing through it. Solid.
|
||
3. ✅ **Deploy, Update, Restart, Stop and Down all failed on the Mac's stack**
|
||
(`peaches-and-plums`) — and on any stack held on the hub that was deployed from the hub's
|
||
own shell rather than from inside Dockhand. The command was built as
|
||
`'DOCKER_HOST=tcp://…' docker compose …`; quoted, `sh` reads that not as a variable but
|
||
as the name of a program to run ("not found", exit 127). Now passed through `env`.
|
||
Verified by a whole life on the Mac and on `grid` from the hub: create, deploy (running
|
||
there, nothing on `core`), restart, stop, delete.
|
||
4. ✅ **A container stopped on purpose was reported as down** when it has no restart policy
|
||
and doesn't handle SIGTERM: `docker stop` then ends it with 137 (or 143, or 130), a
|
||
non-zero code, which read as a crash — "Container down" would have fired for a
|
||
deliberate stop. Those three codes now mean "a person", unless the kernel did the
|
||
killing for want of memory (`OOMKilled`). A real crash (exit 3) is still down.
|
||
5. ✅ **A stack with no containers offered Restart, Stop and Update**, and Restart
|
||
"succeeded" having done nothing. Offered only when there is something to act on; the
|
||
API answers 409 "isn't deployed: there is nothing to restart".
|
||
6. ✅ **One stray tap outside the New stack dialog threw away everything typed** — on a phone,
|
||
a compose file's worth. So did Escape. With anything typed, all three ways out (tap
|
||
outside, Escape, Cancel) ask first; an untouched dialog still just closes.
|
||
7. ✅ **Compose → Form → Compose silently replaced hand-edited compose text** with what the
|
||
form says. The form now writes the box only when the form has changed since it last
|
||
did, and asks before replacing text edited by hand.
|
||
8. ✅ **A question asked from inside a dialog opened underneath it** (same layer, earlier in
|
||
the page), so it could not be answered — found the moment the New stack dialog started
|
||
asking. Questions now sit above whatever asked them.
|
||
9. ✅ **Escape on a question also reached the dialog beneath it**: one key cancelled the
|
||
question *and* closed (or re-asked from) the dialog under it. Escape now answers the
|
||
question and goes no further; the next Escape is the dialog's. Checked on every dialog
|
||
and menu: widget picker, machine menu, rule editor, Power, New stack, and that Escape on
|
||
"Delete the rule?" deletes nothing.
|
||
|
||
10. ✅ **Phone: the bottom of the New container dialog could not be reached** (Mitchell, from
|
||
his iPhone). A dialog is a column whose parts were allowed to shrink, so instead of the
|
||
dialog scrolling, the one part with its own scrollbar — the compose preview — was
|
||
squeezed to 22 px and everything "fitted". And `92vh` on iOS is taller than the screen
|
||
with the address bar gone, so the top sat under the clock. Parts now keep their height
|
||
and the dialog scrolls inside itself; it is never taller than the space between the
|
||
status bar and the home indicator (safe-area insets, no `vh`); Create / Cancel — and
|
||
any error — stay pinned at the bottom while the form scrolls. Applies to every dialog;
|
||
measured at 390×844, 375×667 and sideways, and the picker, rule editor, token and Power
|
||
dialogs rechecked.
|
||
|
||
## Checked and fine
|
||
|
||
All 13 pages × four machines × both sizes: no page errors, failed requests, sideways
|
||
scroll or tap targets under 30 px. A port of `not-a-port` is refused by compose before
|
||
anything is written, and leaves no folder. Environment values with quotes, `#`, `$` and
|
||
nothing after the `=` arrive in the container exactly as typed. A deploy whose image
|
||
doesn't exist shows the registry's error and is **not** remembered as "last deployed". A
|
||
container name already taken fails the deploy and says why. Delete moves only the stack's
|
||
own compose files and `.env` (kept `600` in the trash) and leaves a `data/` folder and
|
||
notes beside them alone; both files of a two-file stack go to the trash; the wrong name,
|
||
or none, deletes nothing. A `.env` value never appears on the stack page and stays masked
|
||
until asked for. A second stack of the same name is 409; an upper-case name is 400. A pull
|
||
of an image that doesn't exist ends as a failed task; a pull on `grid` from the hub lands
|
||
on `grid` and its output is followed. A volume or image in use can't be removed; a running
|
||
container can't be removed (409); a volume needs its name twice. A read-only token gets
|
||
401 from all fourteen new write paths, through the hub as well. `--force`, `-f`,
|
||
`--all-tags`, `..%2F..` and a home outside the known ones are refused; a 1.2 MB compose
|
||
file is refused. Shaped, fleet and streamed sensor queries, `/metrics`, the ten alert
|
||
rules, terminal typing, the file editor's discard question and service deep links are
|
||
unchanged. No native browser dialogs anywhere.
|
||
|
||
## Quality of life — four chosen by Mitchell, ✅ built and verified 2026-09-20
|
||
|
||
1. ✅ **"Where" fits on a phone**: `…/stacks/core · 12 stacks` instead of the whole path (which
|
||
is still written out underneath). Two homes whose last two folders match fall back to the
|
||
full path. Measured at 390 px: no option is wider than the box.
|
||
2. ✅ **How a task ended is a badge**: green `finished`, red `failed · exit 1` with the box
|
||
outlined in red, grey `cancelled`. Verified on a deploy of an image that doesn't exist.
|
||
5. ✅ **Duplicate…** on a stack: the New stack dialog opened on a copy of its compose file,
|
||
named `<name>-copy`. The original's `.env` is copied **on the machine** (`env_from`), mode
|
||
600, its values never sent to the browser — checked: the secret is nowhere in the page.
|
||
Anything that would collide with what is running is named before you press Create
|
||
("a container called X already exists; port 18222 is already published by X") — for
|
||
every new stack, not only duplicates — and clears as you fix it. Disabled while the
|
||
original has unsaved edits. Verified: copy deployed beside the original, which never
|
||
stopped.
|
||
6. ✅ **The form has Labels and Network.** Labels one per line (`$`, quotes and spaces arrive
|
||
intact). Network: its own, the machine's (`host`, which turns Ports off), or **join** one
|
||
that already exists — written as an `external` network so compose doesn't make a second
|
||
one. Verified on the container: both labels, attached to the chosen network only.
|
||
|
||
Found while building them, ✅ fixed:
|
||
|
||
- **Cancel on a stack task cancelled nothing.** It killed the `sh` that ran compose, not
|
||
compose; and as compose still held the output pipe, the task stayed "running" and the
|
||
stack "busy" until compose finished by itself — for a deploy waiting on a health check,
|
||
minutes. Tasks now run in their own process group and the whole group is stopped (TERM,
|
||
then KILL after 4 s); for stacks deployed inside Dockhand, where the process is not the
|
||
task's child at all, the cancel goes into the container after it. Verified both ways:
|
||
ended in ~100 ms, compose gone, the stack free for the next action.
|
||
- **Phone: a stack's task output was squeezed to a line** under the buttons (the same fault
|
||
as the dialog: a scrolling column whose parts were allowed to shrink), which also put
|
||
Cancel out of reach. Detail panes keep their parts' heights; services, containers, stacks
|
||
and cron rechecked on both sizes.
|
||
- The Duplicate dialog and the stack page under it shared a React key, so the dialog
|
||
multiplied. An undeployed stack's dot was green ("0 of 0 running"); now neutral.
|
||
|
||
# Bug bash 4 — 2026-09-21: the Top tab, and the refresh rate it didn't follow
|
||
|
||
Started from Mitchell's first look at Top on his own machine (Firefox on Windows, a window
|
||
1392 CSS px wide — none of which the tab had been drawn for): "Command" printed over
|
||
"Threads", the rate in Settings ignored, the symbol beside *logout* a box of hex digits;
|
||
and make it follow the skins. Method as before, plus two things new: **Firefox** (Playwright's
|
||
build; the GTK libraries it wants are unpacked from their `.deb`s into
|
||
`~/.cache/ms-playwright/fflibs` and given to it as `LD_LIBRARY_PATH`, with
|
||
`PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=1` — nothing is installed on `core`), and a layout
|
||
checker that measures instead of looking — every heading against its neighbour and its own
|
||
text, every row against its box, every border label, at eight window sizes from 900 to 1920.
|
||
Run first against the live v83 it reproduced the screenshot to the pixel ("the Command column
|
||
is 11px wide" at 1392); run after, nothing, in both browsers.
|
||
|
||
## Bugs — all ✅ fixed and verified by the script that found them
|
||
|
||
1. ✅ **Between about 1280 and 1500 px the Command column collapsed to a letter** and its
|
||
heading ran under "Threads:" (itself 4 px too narrow for its own text at every size). The
|
||
table's fixed columns were sized for one window. The process box now measures *itself*
|
||
(a container query — it is a third of a wide window and all of a narrow one, so the
|
||
window's width says nothing) and drops the curve, then Threads, then Command, as it
|
||
narrows; the box also gets half the width rather than 44 %, and no heading can spill
|
||
into the next. Stacked layout now starts at 1280, not 1100.
|
||
2. ✅ **The refresh rate reached a machine only through its Dashboard page.** The sampler on
|
||
each machine was told the rate by an open Dashboard and nothing else — so with the rate
|
||
at 250 ms, `grid`, `wyse` and the Mac were still sampling at 1 s (measured, live), and
|
||
Top, Processes, Disks and every widget of theirs "ignored" the setting until somebody
|
||
opened that machine's Dashboard. The rate is now the fleet's: saved with the UI prefs on
|
||
the hub, which sets its own sampler and passes it to every machine (`PUT
|
||
/api/prefs/sampler`, remembered in each `state.json`), and says it again at start-up for
|
||
a node that restarted or joined since. Verified: Settings 2 s → 250 ms moves the sampler
|
||
with no Dashboard visit.
|
||
3. ✅ **Top read the process table at most once a second whatever the rate** — a clamp of
|
||
my own, exactly what `REFRESH` promises not to do. It reads at the rate: 20 reads in 5 s
|
||
at 250 ms, 10 at 500 ms. (59 frames a second with 390 rows at 250 ms, no stalls, both
|
||
browsers — the rows are only redrawn when the table is read.)
|
||
4. ✅ **Top's `− rate +` changed the rate for half a minute.** It set the browser's copy
|
||
and never saved it, so the next sync with the hub put it back. It saves — which is also
|
||
what moves the samplers — survives a reload, and each end is disabled at the limit.
|
||
5. ✅ **A viewer whose clock is wrong froze the curves.** New points were timed against the
|
||
machine's timestamps: a browser clock 7 minutes slow waited 7 minutes for its next point
|
||
(Top *and* the Dashboard); 7 fast, and it added one per tick. Timed on the browser's own
|
||
clock now. Verified with the page's clock shifted both ways: 5 points in 10 s at a 2 s
|
||
stride, both pages, both browsers.
|
||
6. ✅ **Every other point was being skipped.** "Has a stride passed?" was asked with `>=`
|
||
of a timer that fires a few ms early half the time — so at a 2 s rate and a 2 s stride a
|
||
curve gained 3 points in 10 s, not 5, and its time axis was quietly wrong. The same
|
||
question on the server kept every bucket one sample long (14 s points at a 12 s stride).
|
||
Both now allow half a tick of slack and keep to the stride's own beat; the server times
|
||
from the last point kept, so a stalled or sleeping machine just keeps the next sample it
|
||
gets. Unit-tested with jittered samples: 240 samples → exactly 40 points, 12 000 ± 100 ms
|
||
apart, each the mean of its own six.
|
||
7. ✅ **The power symbol was a typed `⏻`** (U+23FB, Unicode 9) — not in Windows' fonts, so
|
||
Firefox there drew the missing-glyph box. Drawn as an SVG like the other icons. Every
|
||
other symbol the console types was checked against the same risk: all older.
|
||
8. ✅ **A handful of samples was stretched into a bar chart.** A machine ten seconds into
|
||
its history filled the whole CPU box with five fat columns. The box is now always the
|
||
whole history window — a young curve draws in from the right, as btop's does, and a full
|
||
one fills it edge to edge, at any window (checked at 10 min and at 1 min).
|
||
9. ✅ **Idle curves vanished**: a core at 10 % rounded to no dots at all in its mini graph.
|
||
Anything above zero lights the bottom row, which is also btop's rule.
|
||
|
||
## Skins
|
||
|
||
✅ Top now wears the skin. The ground is the skin's terminal background and every curve and
|
||
meter is built from its sixteen terminal colours (chosen to be read on exactly that ground),
|
||
the rest from the usual variables — so the dark skins look like btop and the light ones like
|
||
btop on paper, and a skin arriving from another device repaints the page in place. Measured
|
||
in all thirteen: text ≥ 4.5 : 1, everything else ≥ 3 : 1, dots ≥ 2.2 : 1. Two failed —
|
||
process names in Latte (2.75) and Solarized Light (2.62), where a terminal green is too
|
||
pale for small type — and use `--ok`, the green each skin was already measured readable
|
||
with.
|
||
|
||
## Checked and fine
|
||
|
||
Sort by pid, name, mem and cpu, and reverse; filter by name, command, user and pid, with the
|
||
count showing matched/total and an empty result saying so; a selection that holds while the
|
||
table is re-read under it; terminate, on a throwaway `sleep`, asks by name and pid, ends it,
|
||
and the footer stops offering a process that is gone. Sampler values outside 100 ms–60 s are
|
||
refused, and a rate slower than the collector's 10 s samples at 10 s. All 14 pages at his
|
||
window size and on a phone, in both browsers: no errors, no overflow.
|