hamlab/PLAN.md

Ham Lab — MVP plan

Proxmox-style web console for a bare-metal Debian 13 host. First target: core (10.0.0.234). MVP is service + shell + persistent terminals; everything else is a stub tab or a roadmap item.

Decisions

Area	Decision	Why
Backend	Go 1.24, single static binary, frontend embedded via embed	core has ~370 MiB free RAM; a Go daemon idles at ~15 MiB vs ~80+ for Node. One file to deploy, no nvm path in the unit. Same binary later installs on grid unchanged.
PTY	creack/pty — server-owned PTYs running core's login shell	Sessions live in the daemon, not the browser, so refresh/tab-switch reattaches. No tmux dependency (not installed on core).
Transport	One WebSocket per attached terminal; REST for everything else	Standard; NPM passes WebSockets with one toggle.
Frontend	React + TypeScript + Vite, xterm.js (+ fit, webgl, web-links addons)	Matches calorie-counter / wedding-website, so no new toolchain to learn.
Auth	Single-user password login, argon2id hash, HttpOnly/Secure/SameSite=Strict session cookie, login rate-limit, Origin check on WS upgrade	This is a root-capable shell on the public internet. Auth is MVP, not roadmap.
Service	hamlab.service, User=core, Restart=always, binary at /usr/local/bin/hamlab	Same pattern as calorie-counter.service. Runs as core so sudo works in the terminal without a password, as on the SSH path.
Listen	10.0.0.234:8100 (free)	NPM is a container, so loopback-only won't do. Firewalling to NPM is a later hardening step.
Data	/home/core/apps/hamlab/ — config.yaml, state.db (SQLite)	apps/ is the nightly rsync target to grid, so it's backed up for free.
Ingress	NPM proxy host hamlab.bigham.dev → :8100, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a second API call (NPM drops them on create-with-new-cert)	*.bigham.dev already resolves; the proxy host alone makes it live.
Repo	Gitea mitchell/hamlab, private, main	Created 2026-09-17.

Layout

┌──────────────────────────────────────────────────────────────┐
│ header: Ham Lab · core · load 0.42 · ● connected · [logout]    │
├──────────────┬───────────────────────────────────────────────┤
│ Dashboard    │ shell 1 ● │ backups │ +                      │
│ Processes    ├───────────────────────────────────────────────┤
│ Terminal     │                                               │
│ Services     │        main content (xterm)                   │
│ Cron         │                                               │
│ …            │                                               │
├──────────────┴───────────────────────────────────────────────┤
│ footer: v0.1.0 · up 12d · hamlab.bigham.dev                   │
└──────────────────────────────────────────────────────────────┘

• Left rail: flat list of pages, active one highlighted. No nested leaves.
• Terminal page: tab strip across the top of the pane, one tab per session, ● when a process is running, + pinned at the right edge. Active tab carries rename and close; double-click renames, middle-click closes. (Until 2026-09-18 the sessions were leaves under a collapsible Terminal group in the rail.)
• Non-MVP groups render as disabled leaves with "coming soon" — the rail shape is the product; stubs keep it honest about scope.
• Mobile: rail becomes a slide-in drawer; xterm fits the viewport; a sticky key bar above the keyboard (Esc Tab Ctrl Alt ↑ ↓ ← → |) because phone keyboards have none of those. This is what makes it usable as a PWA rather than a novelty.

Terminal semantics

• A session is a PTY + shell owned by the daemon, with an ID, name, cwd hint, and a scrollback ring buffer (256 KiB). Attaching replays the buffer, then streams.
• Any number of browser tabs can attach to the same session (all get output; all can type). Resize follows the most recent attacher.
• Switching tabs in the UI just detaches/attaches; the shell keeps running.
• Closing a session sends SIGHUP to the process group and drops it. Shell exit closes the session automatically (leaf shows "exited" until dismissed).
• Session names persist in state.db so the sidebar order survives a service restart. The shells themselves do not survive a daemon restart in the MVP — see roadmap (tmux backing).
• Idle sessions cost one shell process each; no automatic reaping in MVP.

API (MVP)

POST /api/login            {password}            → 204 + cookie
POST /api/logout                                 → 204
GET  /api/me                                     → {host, user, version, uptime, load}
GET  /api/terminals                              → [{id, name, running, created}]
POST /api/terminals        {name?}               → {id, ...}
PATCH /api/terminals/:id   {name}                → 204
DELETE /api/terminals/:id                        → 204
WS   /api/terminals/:id/attach   binary frames = raw bytes; JSON text frames = {resize}
GET  /healthz                                    → 200 (unauthenticated, for NPM)

Everything under /api except login and /healthz requires the session cookie.

Project layout

hamlab/
  cmd/hamlab/main.go          serve | passwd | version
  internal/
    auth/                     argon2id, sessions, rate limit
    term/                     PTY manager, ring buffer, attach/detach
    api/                      chi router, handlers, WS
    state/                    SQLite (modernc, no cgo)
    web/                      go:embed of web/dist
  web/                        Vite + React + TS + xterm.js, PWA manifest + SW
  deploy/
    hamlab.service
    install.sh                build → /usr/local/bin/hamlab → systemctl restart
  Makefile                    build / dev / deploy / lint / test

Milestones

M0 — skeleton (service boots, login works)

• Go module, chi, config loading, hamlab passwd, argon2id login, session cookie.
• hamlab.service on core, :8100, serves a placeholder page.
• make deploy from core itself.

M1 — terminals (the MVP feature)

• PTY manager with ring buffer, create/list/rename/close, WS attach with resize.
• React shell: header, rail, footer, Terminal group, xterm main pane.
• Persistence of session list across refresh and rail switching.

M2 — polish + PWA + ingress

• Mobile drawer, key bar, fit-on-rotate.
• manifest.webmanifest, icons (Ham logo on black, same recipe as HamTV), service worker caching the app shell only (never /api).
• NPM proxy host for hamlab.bigham.dev; verify WS through NPM; force-SSL/HSTS/HTTP2.
• Stub groups for Services / Cron / ZFS / Dashboard.

Done when: from the phone, open the PWA, log in, open two terminals, run something long in one, switch away and back, refresh — it's all still there.

Roadmap (not MVP — notes only)

• Updates tab — shipped 2026-09-17 (internal/updates, internal/tasks, Updates.tsx): upgradable packages with current → candidate, suite, security badge and description; index age, last upgrade (from /var/log/apt/history.log), unattended-upgrades on/off, reboot-required banner with the packages that caused it; Refresh index, Upgrade all (dist-upgrade), Upgrade selected; changelog per package on demand. Runs go through the new tasks package: background command, output kept, followed live over /api/tasks/{id}/stream, cancellable, listed at /api/tasks (the seed of a Task History tab). apt is fully supported; dnf/pacman list only.

• Power — header ⏻ button (amber when a reboot is pending), dialog that requires the hostname typed back, then systemctl reboot / poweroff. /api/me carries reboot_needed so the header knows.

• tmux-backed sessions so shells survive a daemon restart / redeploy. Would spawn tmux new -A -s hamlab-<id>; needs mouse on and status-bar handling in xterm.

• Services tab — systemctl list-units, per-unit page: status, last 200 journal lines (live-tail via WS), start/stop/restart/enable buttons, unit file viewer.

• Cron tab — shipped 2026-09-17 (internal/cron, Cron.tsx): one list built from /etc/crontab, /etc/cron.d/*, every user crontab (via sudo), the scripts in the run-parts directories those drive, plus systemd timers. Next run computed by an in-house cron-expression evaluator (tested: ranges, steps, names, dom-OR-dow rule, @ aliases); last run from the journal's (user) CMD (command) lines (cron logs via syslog to journald; reading needs sudo/adm). Commented-out jobs show as disabled; job-shaped lines inside comment blocks (Debian's header examples) are ignored. Per job: Run now (background, as the job's user, sh -c with crontab % rules, output captured), Enable/Disable (comments the line in place, refuses if the file changed underneath), Edit source (textarea; validated line-by-line; user crontabs installed with crontab -, system files written via sudo tee), Cron log, and Output file when the command redirects to one. Timers: Run now (starts the activated service), start/stop/enable/disable, live logs of the service via the Services tab's tail.

• Disks tab — shipped 2026-09-17 (internal/storage, Disks.tsx), replacing the planned ZFS tab: block devices from lsblk -J (kind, model, size, temperature from the device's hwmon node, partitions with filesystem usage, live read/write/busy from the metrics sampler); SMART via smartctl -j when smartmontools is present (one button installs it as a task otherwise): health, temperature, power-on time, self-test status, ATA attribute table with the failure-predicting IDs flagged, NVMe endurance and error counters, short/long self-tests. Spinning disks are read with -n standby so the tab never wakes a sleeping drive; "Wake and read" is explicit. ZFS section when the tools and module are there: pools with health, capacity, fragmentation, last scan and vdev tree; scrub start/pause/stop; datasets tree with used/refer/in-snapshots/ snapshot count/compressratio/mount; per-dataset snapshot list and "Snapshot now". Verified against grid (raidz2 tank, 10 datasets, 515 snapshots, 8 WD40EZAZ + 970 EVO Plus SMART). Snapshot destroy and dataset property editing are deliberately not there yet.

• Dashboard — first cut shipped 2026-09-17 (internal/metrics, Dashboard.tsx): CPU (total + per core), memory/swap, load, uptime, hwmon temps and fans read straight from /sys (no lm-sensors), per-interface network rates, per-disk read/write/busy from /proc/diskstats, filesystem usage, UPS via sudo -n pwrstat -status. Sampled every 2 s into an in-memory ring of 10 min; the UI seeds from GET /api/metrics/history then polls GET /api/metrics. Customizable: a Customize mode reorders and hides modules (CPU, Memory, Load, Uptime, Temperatures, Fans, Network, Disks, Filesystems, UPS) and hides individual rows (an interface, a mount, a disk, a sensor); the layout is stored server-side in state.json via GET/PUT /api/prefs/dashboard so every device sees the same dashboard. The update interval (100 ms – 2 s) is part of the layout and drives both the browser's polling and the daemon's sampling clock; the chart window is always 300 samples, so 30 s at 100 ms and 10 min at 2 s. Still to do: persistent history (SQLite ring, e.g. 10 s × 7 days), SMART (smartctl -j), longer time ranges.

• Sensors — shipped 2026-09-19 (internal/sensors, internal/metrics/sensors.go, internal/api/status_sensors.go, web/src/sensors.ts, widgets.tsx, Embed.tsx; reference in SENSORS.md): everything measured is a sensor with a stable dotted ID; catalog, single value, history, a shaped query (the client sends the object it wants and gets it back filled in — GraphQL's idea without GraphQL, since sensors differ per machine and a static schema would collapse to sensor(id:)), the same query as a live SSE stream, Prometheus /metrics, read-only sensors:read tokens (never full tokens in a URL), status sensors (services, updates, ZFS, SMART), a widget library, custom widgets on the Monitors tab (was "Dashboard"; the view id and URL are still dashboard), and a frameable /embed/widget for Glance and friends. Remaining: port the stock Monitors modules onto sensors; per-unit service and container sensors; pattern-restricted tokens; one-request fleet query.

• Multi-host — Phase 1 shipped 2026-09-18 (internal/nodes, internal/tlsauto, MachineSwitcher.tsx, Machines.tsx). One binary, hub and spoke: a host whose config.json lists nodes is a hub; any other is a standalone node with its full UI. Node side: "tls": "auto" makes a self-signed cert in the data dir and serves HTTPS; POST /api/tokens {password,name} trades the login password for a long-lived bearer token (SHA-256 stored in tokens), accepted by the auth middleware alongside the cookie; DELETE /api/tokens/{name} revokes. Hub side: POST /api/nodes {name,url, password} peeks the node's certificate (trust on first use, fingerprint pinned), obtains a token named hub:<hostname>, stores {name,url,fingerprint,token}; a 15 s prober keeps each node's status; /api/nodes/{name}/… is a reverse proxy to the node's /api/… (bearer swapped in, cookie and Origin stripped, streaming flush) that carries WebSockets, so terminals, log tails and task output work remotely. Frontend: setMachine() rewrites every /api/… path; the rail has a machine switcher at the top and a Machines page to add/remove/open. Pages remount on switch; terminals live on the node. Nodes: grid, wyse. Phase 3 (update from hub) shipped 2026-09-18: node POST /api/self-update takes a binary, checks it answers version, keeps hamlab.prev in the data dir, installs over the executable via sudo install, replies, then restarts the service; the hub pushes its own executable with POST /api/machines/{name}/update or /api/machines-update-all (architecture checked first via /api/me.arch), the Machines page shows an outdated badge and Update buttons, and hamlab push-nodes does the same from the CLI — make deploy runs it last, so one deploy on the hub updates every machine. First install on a new host is still by hand (scp binary + install.sh, hamlab passwd, tls auto). Still to do: Phase 2 Datacenter overview; optional firewall rule limiting a node's port to the hub. Cross-architecture (2026-09-18): make deploy also builds bin/hamlab-linux-arm64, install.sh installs it beside the binary as hamlab-arm64, and the hub picks the sibling matching a node's /api/me.arch when pushing — the Mac mini (Fedora Asahi, aarch64, node mac) updates like the rest.

• Portability + install — shipped 2026-09-18 (internal/setup, internal/priv, install.sh, make release): the binary installs itself (hamlab install / uninstall; systemd or OpenRC; root, sudo or doas), all builds static on amd64, arm64, arm, 386 and riscv64, install.sh as the curl-able bootstrap with checksums. Every privileged call goes through priv instead of a literal sudo -n. Updates tab acts on dnf, zypper, pacman and apk as well as apt. Services list falls back to the plain table on systemd < 246; Disks falls back to /sys/block without lsblk. Listener uses IP_FREEBIND. Self-update: pre-flight check against the live config, atomic replace, exec in place. Config default moved to ~/.config/hamlab (existing ~/apps/hamlab still found by the installer and the CLI). Tested in containers: Debian 12, Ubuntu 22.04, Fedora 40, Rocky 8, openSUSE Leap 15.5, Alpine 3.18/3.20 (OpenRC + doas, non-root), plus the four lab hosts. Not done: an OpenRC backend for the Services tab; runit/s6/sysvinit service files; public hosting for install.sh (the Gitea repo is private, so the one-liner needs HAMLAB_TOKEN).

• Auth hardening — WebAuthn passkeys (phone Face ID), TOTP fallback, session list + revoke, optional NPM access list restricting to WireGuard/LAN.

• Files tab — shipped 2026-09-18 (internal/files, Files.tsx): browse from / with breadcrumbs, an editable path box, sort, hidden-file toggle and a filter; rows show type icon, size, modified, perms and owner, link targets and broken links. Text files (≤ 2 MiB, sniffed) open in an editor with Ctrl+S / Save; images preview; everything else downloads. Upload via drag-and-drop or picker with per-file progress (streamed, no size limit). New folder / file, rename, move, chmod, copy path, delete (folders need the name typed back), and "open in terminal", which cds the active shell (or a fresh one) to the folder. Sudo toggle: root access re-executes the same binary as hamlab fs <op> under sudo -n, streaming bodies over stdin/stdout, so both modes share one implementation and one set of rules (absolute paths only; /, /etc, /usr, … are never deleted or moved whole). Every write is logged with the caller's IP.

• Docker read-only view — docker ps/logs; dockhand stays the source of truth.

• Notifications — push on service failure / SMART warning / UPS event.

• Terminal extras — search, copy-mode, font/theme, split panes, upload via drag-drop into cwd.

• Settings + themes — shipped 2026-09-18 (Settings.tsx, themes.ts): Settings page with Appearance (Dark, Light, Cyberpunk, Retro — CSS variables on <html> plus a matching xterm palette, applied before first paint from localStorage and synced hub-side via GET/PUT /api/prefs/ui so the choice follows the login), Account (change password, POST /api/password, current password required), Access tokens (list/revoke the tokens hubs hold), About (version, host, arch, uptime, certificate).

Open questions

• Idle-session reaping policy (never / after N days) — leaning never; it's a single-user box.
• Whether sudo -i-style root sessions get their own leaf colour. Cosmetic.