hamlab/BUGS.md

Bug bash — 2026-09-18

Fix pass 2026-09-18: every item carries its outcome; ✅ resolved and verified live, ➖ no change with reason. Found by a headless Chromium pass (iPhone and 1440px desktop viewports) over every page on the hub and on each remote machine, an offline-node scenario (wyse stopped for the test, then restarted), direct API probing with hostile inputs, and a code read. Ordered by severity within each group.

Security / robustness

1. ✅ RESOLVED 2026-09-18 — only regular files are readable; the byte cap applies to what is sent, not to stat. /dev/zero, /dev/null and FIFOs answer 400 in 30 ms; /proc/cpuinfo still reads. Reading a special file hangs the request forever. GET /api/files/read?path=/dev/zero never ends: the max cap compares against stat size, which is 0 for device files, so the cap never triggers and the handler streams /dev/zero until the client gives up. Same for FIFOs (blocks on open) and /proc/self/mem. Opening one in the Files viewer would hang the tab. Needs: refuse non-regular files for read/edit, and cap the copy with io.LimitReader regardless of stat.
2. ✅ RESOLVED — per-client buckets (first X-Forwarded-For hop or the peer) with the same 5-strike schedule, plus a global backstop of 100 failures per 5 min → 1 min. Verified: core's public IP locked while grid logged in fine. Login lockout is global, not per client. Five bad passwords from anyone on the internet lock the real user out for up to 10 minutes (auth.Limiter is one bucket). The bash locked the hub for the full 10 minutes twice. Also locks POST /api/tokens, so hub pairing fails during a lockout. Needs per-IP buckets with a modest global cap.
3. ✅ RESOLVED — server rejects refreshMs outside 100–60000 and non-identifier themes (400); client ignores out-of-range rates from any source. UI prefs from the server are not validated. A stored refreshMs of -5 made the client call setInterval(-5) on every live view: a request storm that took the browser to ERR_INSUFFICIENT_RESOURCES and hammered the daemon. theme is validated only by falling back. Needs range checks on load (and on save server-side).
4. ✅ RESOLVED — every response carries X-Frame-Options DENY, CSP frame-ancestors none, nosniff, no-referrer. No anti-framing / CSP headers. Only HSTS (from NPM). A root-capable console should send X-Frame-Options: DENY, Content-Security-Policy: frame-ancestors 'none', Referrer-Policy: no-referrer, X-Content-Type-Options: nosniff on every response.
5. ✅ RESOLVED — kernel threads (empty cmdline) and the daemon's own PID are refused. Signals can be sent to kernel threads. POST /api/procs/2/signal (kthreadd) returns 204. Harmless in practice, but the guard should refuse PIDs with an empty cmdline (kernel threads) and our own PID.
6. ✅ RESOLVED — unit runs with SupplementaryGroups=systemd-journal adm so journal reads need no sudo, and the installer drops /etc/sudoers.d/hamlab-quiet (no syslog/pam session for the read-only polling commands). Verified: 0 sudo lines in 60 s of 250 ms polling (was 839/10 min). Applied on all four hosts. Journal spam from the daemon's own sudo calls. Every sudo'd journalctl, smartctl or pwrstat writes three pam_unix(sudo:session) lines into the journal under hamlab.service: 839 lines in 10 minutes at a 250 ms refresh, and the journal is already 4 GB. It also buries Ham Lab's own log entries. Fix options: add the service user to systemd-journal (and adm) so journal reads don't need sudo, and/or a sudoers Defaults!/usr/bin/journalctl !syslog, !pam_session line in the installer.
7. ✅ RESOLVED — server answers 409 unless overwrite=1; the UI asks "Replace it?" first. Uploading over an existing file silently overwrites it. No confirmation, no "replace?" prompt. Same for "+ file" when the name exists (server refuses; UI just shows the error, fine) — upload is the gap.
8. ➖ NO CHANGE — Go's mux redirects the unclean path to /etc/passwd, which is an ordinary deep link the SPA must serve; there is no API prefix left by the time the static handler sees it. Harmless. /api/<anything>/../../x falls through to the SPA and returns index.html with 200 instead of a JSON 404. Harmless, but sloppy for an API prefix.

Functional bugs

9. ✅ RESOLVED — 401 carries the server message; the form shows "wrong password" (verified headless). Wrong password gives no feedback at all. The 401 is thrown as Unauthenticated with an empty message, and the login form only renders err when non-empty. The button just re-enables. Reproduced on phone and desktop.
10. ✅ RESOLVED — hub-only pages bounce to the dashboard on switch (verified headless). Switching machines while on Settings or Machines keeps showing the hub's page. switchMachine only resets the view for Machines; Settings stays, so "mac" shows core's account/token/about panels. Both are hub-only pages and should bounce to the dashboard (or be clearly labelled as hub pages).
11. ✅ RESOLVED — escaped; no console error. Machines add-form pattern is an invalid regex in Chromium. [a-z0-9][a-z0-9-]* fails under the v flag ("Invalid character class"), logs a console error and the browser skips validation. Needs [a-z0-9][a-z0-9\-]*.
12. ✅ RESOLVED — refresh from the newest repodata under libdnf5/dnf caches (Mac: 9 d), last upgrade from dnf history (194 d), label reads "from dnf history". Updates tab on dnf hosts shows nonsense. Mac: "Index refreshed 561d ago" (mtime of /var/cache/dnf, which Fedora 41+ no longer uses; libdnf5 is the live cache) and "Last upgrade never · from apt history" (label and source are apt-specific). Needs dnf history for last upgrade, metadata timestamps for refresh, and neutral labels.
13. ✅ RESOLVED — journal read incrementally by cursor, timers from one systemctl show, services' two calls run in parallel. Cron 3.6 s → 0.08 s, services 0.6 s → 0.05 s, with last-run and next-run data intact. Cron overview is slow: 3.6 s per request on core. At any refresh rate the page is constantly busy. The 30-day, 20 000-line journal scan for "last run" is rebuilt every 5 s; timers make three systemctl calls. Needs an incremental journal cursor and a single systemctl call. Services list is 0.6 s for the same reason (two calls).
14. ✅ RESOLVED — heap profile showed a 48 MB bytes.Buffer from buffering journalctl dumps; those are now streamed line by line, plus a 160 MB soft memory limit for the GC. RSS 22 MB fresh, 27 MB after 25× cron/logs/procs/services (was 180 MB). pprof stays available behind the login at /api/debug/pprof/. Daemon memory grows. RSS on core went 22 → 85 → 216 MB during the day (136 MB at rest after the bash). Suspects: per-pid caches in the process table (procInfo only prunes on the next walk), task output buffers, SMART cache, xterm scrollback rings that are never freed for exited sessions. Needs a heap profile under load.
15. ✅ RESOLVED — --dirty dropped from the version and deploys now follow commits; binaries report e.g. v0.1.0-38-gc1805f6. Version string is always -dirty. make deploy builds before the commit, so every installed binary reports e.g. v0.1.0-32-ga372378-dirty, and the Machines page compares with -dirty stripped. Build after commit, or tag releases.
16. ✅ RESOLVED — hash routing #/<machine|hub>/<view>: reload restores machine and tab, Back/Forward work, links can point at a page on a machine (verified headless). View resets to Dashboard on reload; no deep links; browser Back leaves the app. The selected machine survives a reload (sessionStorage) but the tab does not, and there is no URL routing at all. Needs hash or path routing (/#/grid/services).
17. ✅ RESOLVED — newest first. Cron log tab lists oldest first and starts at the top. 200 lines from Sep 10 with the newest at the bottom, off-screen. Should show newest first or scroll to the end.
18. ✅ RESOLVED — "Nothing matches." Processes: empty filter result shows nothing. No "nothing matches" message.
19. ✅ RESOLVED — exited shells remove themselves 10 minutes after exit (verified live: seeded 17:53:56, gone at 18:03:59 with no daemon restart in between). Found and fixed alongside: a daemon restart used to re-spawn exited shells as fresh running ones; only running shells are persisted now. Idle running shells are deliberately kept. Terminals accumulate. Nothing ever reaps exited or idle sessions; the bash left several "bug-bash-term" shells on the hub until deleted by hand. Exited sessions should auto-remove after a while; idle ones at least be counted somewhere.
20. ✅ RESOLVED — a node answers 409 "busy" when a package task is running or a terminal is attached; the Machines page offers Force, hamlab push-nodes --force / make push-nodes-force do the same. Verified: 409 with a terminal attached on grid, 200 with force. Self-update restarts without warning running work. A push to a node kills its open terminals and any running task (upgrade in progress). The hub should refuse or warn when the node reports a running task or attached terminals.
21. ✅ RESOLVED — PathError and LinkError are unwrapped to the bare reason. Error text leaks raw Go errors. e.g. rename shows "rename /tmp/x /tmp/y: no such file or directory". Fine for a lab, ugly for a product; strip the syscall prefix.

Mobile / UX

22. ✅ RESOLVED — on coarse pointers every button, chip, icon button, link, header and select is at least 36 px tall (verified: no control under 32 px on the dashboard or processes pages). Tap targets under 28 px on phone: header ⏻ (25), logout (22), customize ▲ ▼ hide (23), terminal leaf ✎ ✕ (24), chips (25), table headers (27). Apple's guideline is 44 pt. Systemic: the small .btn/.chip/.icon-btn sizes need a touch variant.
23. ✅ RESOLVED — the dialog asks for the machine name (mac); the node still checks its own hostname under the hood. Power dialog on the Mac asks you to type 5ce91ee54af6. The confirmation uses the node's hostname, which on the Mac is the unset default. Should offer the machine name (mac) as the confirmation word on remote machines.
24. ✅ RESOLVED — down interfaces collapse to one line marked "down"; modules with nothing to show (fans, UPS, temps, disks) hide unless customizing. Dashboard shows dead interfaces and empty modules. wlp1s0 (down) gets a full sparkline block of flat lines; "Fans: no fan sensors" is a permanent empty card on hosts without fans. Down interfaces should collapse to one line; empty modules should auto-hide unless customizing.
25. ✅ RESOLVED — "load 1.39 1.56 1.55" with a tooltip. Header load average is unlabelled ("1.39 1.56 1.55" floating in the header).
26. ➖ NOT FIXING — the list is intentionally a plain table; the menu anchors inside the row. Cosmetic taste, not a defect. Files list on desktop is a narrow strip in a sea of empty space for small folders; the row menu opens off the right edge of the row. Cosmetic.
27. ✅ RESOLVED — after a 502 the client leaves that machine alone for 5 s: 2 requests in 10 s at a 250 ms refresh (would have been ~80), badge and message intact. Offline machine keeps polling every view at the refresh rate, producing a 502 per request per page (and console errors). The UI copes — badge, message — but the hub should back off to the prober's cadence while a node is down.
28. ✅ RESOLVED with 6. Services logs are noise. The default view of hamlab.service logs is the sudo session spam from bug 6, three lines per poll.

Confirmed working under abuse (for the record)

Path traversal in files, services, cron, ZFS, procs and updates all rejected. /etc and / protected from delete/rename as root. Unit, pool, dataset, snapshot and package names validated. Token issue refused with a wrong password. Oversized prefs rejected. Duplicate machine add refused. Unicode + spaces in file names round-trip. Upload byte-exact. Terminal scrollback survives switching. Offline node: switcher and header show it, pages show the reason. Remote terminals, logs, tasks and files through the proxy all work on all three nodes.

Bug bash — 2026-09-19

Found by driving a scratch instance (same binary, its own state, real fleet behind it) in headless Chromium at 1440 px and iPhone size: a sweep of every page on every machine for page errors, failed requests, overflow and small tap targets; then scripted use of every feature — terminal tabs, Files in a throwaway folder, Processes (killing a process the test spawned), Monitors customisation and widgets, Services, Cron, Updates, Disks, Machines, Settings, embeds, machine switching, session loss — and a hostile pass over the API. Host-changing actions (service control, cron edits, upgrades, power) were opened but not executed. Every fix below was re-checked by the same script that found it.

Bugs — all ✅ fixed and verified

1. ✅ One widget with a bad pattern silenced every custom widget. They share one stream; fan.[ made the whole query a 400, so all of them showed "…" forever with no reason given. The picker now asks the API before adding and shows its answer; a bad pattern already saved shows a notice instead of dots.
2. ✅ A saved layout of the wrong shape could not be loaded ("could not load layout", and the page could no longer be customised). normalize() now accepts anything.
3. ✅ A lost session left pages saying "unauthenticated" for up to 30 s. A 401 from the hub now returns to the login form at once, and logging in returns to the same page. (A 401 relayed from a managed machine is that machine rejecting the hub, and doesn't.)
4. ✅ The machine-name rule in "Add a machine" was an invalid regular expression under the browser's v mode, so it validated nothing and logged a console error on every render.
5. ✅ Two terminals opened in the same minute got the same name ("shell 09:45" twice). Now "shell 09:45 (2)".
6. ✅ Names had no length limit — a 328-character terminal tab, a 5000-character token name. 64 characters, refused with a reason; token names also refuse / \ < > ".
7. ✅ "+ file", "+ folder" and Rename accepted ../x and a/b, quietly acting outside the folder on screen. A name is now one path segment; Move… is how things go elsewhere.
8. ✅ Unsaved file edits were thrown away without a word when you picked another page or machine, or closed the tab. All three now ask; Cancel keeps you where you were.
9. ✅ A mistyped path in Files was remembered, so a reload landed on the error. Only a folder that opened is remembered.
10. ✅ Escape closed nothing: the machine menu, the file menu, the Power dialog, the widget picker. It closes all four.
11. ✅ Reset on Monitors removed every custom widget with no confirmation. It asks, and says how many.
12. ✅ A few bytes of query bought 27 MB of answer ({"k0":"*",…×400} with full=1) — and a read-only token is public on the demo page. A query may return 10 000 values (thirty whole machines); past that it is a 400 that says so.
13. ✅ No ceiling on live streams: 300 from one token were accepted. 32 per client address, 512 in all, then a 429 that says so; closed streams free their places.
14. ✅ Anything under /embed/ lost the anti-framing headers, not just the widget page. Only /embed/widget is frameable.
15. ✅ The new ZFS sensors ran three sudo commands a minute on every machine — including core, which has the tools and no pool — and each wrote three lines to the journal: most of hamlab.service's log was that. Pools are now read with one unprivileged zpool list; a source that finds nothing is asked again ten times less often; and the installer's quiet list covers zpool|zfs list|status.
16. ✅ Phone: the filter boxes on Processes and Files, the path box and the "/" crumb were under 30 px. 36 px like every other control.
17. ✅ Disks → datasets on a phone: rows ended 50 px before their own cells when scrolled sideways (min-width was less than the columns add up to).

Also: tab titles now say where you are ("Services · grid — Ham Lab"), so history and open tabs can be told apart.

Checked and fine

Terminal focus after every switch/close/return; scrollback after reload; exited shells; 30 tabs with the + still reachable; re-fit on resize; Back/Forward; unknown machine or page in the URL; wrong password; XSS through terminal, token and file names (all rendered as text); hidden files, filter, symlinks (broken ones marked), binary files, upload and replace, folder delete needing the typed name, root toggle; process sort, filter and Terminate (asks first); themes; password-change validation; duplicate or unreachable machine; hub terminals not leaking into another machine's page; CORS only on sensor paths; token list never exposing hashes; cookie flags; deep nesting, empty bodies, odd methods and every= values on the sensor API; phone drawer, key bar and picker.

Quality of life — all ✅ done 2026-09-19, each verified in the browser

1. ✅ Selections are in the URL. #/grid/services/ssh.service, #/hub/cron/job:…, #/hub/files/%2Fetc: reload keeps them, links carry them, leaving a page drops them.
2. ✅ Browser Back with unsaved file edits asks, like page and machine changes do.
3. ✅ No more browser prompt()/confirm() boxes — all 30 are in-app dialogs (askYes, askText in ui.tsx): Enter submits, Escape or a tap outside cancels, destructive ones are red and say what they do ("Close", "Delete"), and they fit a phone.
4. ✅ Token creation is one dialog: name, patterns, and the machine's own sensor families as chips to click.
5. ✅ Widgets can be edited — sensor, type and title — in place, from Customize.
6. ✅ The Widgets gallery has a search over IDs, names and labels (so "gitea" finds its container's sensors).
7. ✅ Status sensors keep a history (300 readings each: an hour of container memory, two days of drive wear), so their sparklines have a past.
8. ✅ A stream that fails says why: "the token was rejected", "too many live streams are open", "that machine is unreachable", "can't reach Ham Lab" — in embeds and on Monitors.
9. ✅ Pattern widgets get readable default titles: cpu.core.*.usage → "CPU core usage".
10. ✅ Terminal tabs reorder — drag on a desktop, ‹ › on a touch screen, remembered per machine in the browser — and Alt+Shift+←/→ switches tabs, Alt+Shift+T opens one, with the keyboard left in the terminal each time.
11. ➖ "Log out everywhere" — built, then removed the same day at Mitchell's word: no need for it on a single-user console.
12. ✅ Real clipboard shortcuts in the terminal, from a physical keyboard: Ctrl+C copies when text is selected (and is still ^C when nothing is — the selection clears after a copy, so a second Ctrl+C interrupts), Ctrl+V pastes, Ctrl+Shift+C / Ctrl+Shift+V do the same unconditionally, Ctrl+Insert / Shift+Insert too. It rides the browser's own copy and paste events, which xterm already answers, so it works over plain http on the LAN where navigator.clipboard does not exist; bracketed paste is honoured.