hamlab/PLAN.md

Ham Lab — MVP plan

Proxmox-style web console for a bare-metal Debian 13 host. First target: core (10.0.0.234). MVP is service + shell + persistent terminals; everything else is a stub tab or a roadmap item.

Decisions

Area	Decision	Why
Backend	Go 1.24, single static binary, frontend embedded via embed	core has ~370 MiB free RAM; a Go daemon idles at ~15 MiB vs ~80+ for Node. One file to deploy, no nvm path in the unit. Same binary later installs on grid unchanged.
PTY	creack/pty — server-owned PTYs running core's login shell	Sessions live in the daemon, not the browser, so refresh/tab-switch reattaches. No tmux dependency (not installed on core).
Transport	One WebSocket per attached terminal; REST for everything else	Standard; NPM passes WebSockets with one toggle.
Frontend	React + TypeScript + Vite, xterm.js (+ fit, webgl, web-links addons)	Matches calorie-counter / wedding-website, so no new toolchain to learn.
Auth	Single-user password login, argon2id hash, HttpOnly/Secure/SameSite=Strict session cookie, login rate-limit, Origin check on WS upgrade	This is a root-capable shell on the public internet. Auth is MVP, not roadmap.
Service	hamlab.service, User=core, Restart=always, binary at /usr/local/bin/hamlab	Same pattern as calorie-counter.service. Runs as core so sudo works in the terminal without a password, as on the SSH path.
Listen	10.0.0.234:8100 (free)	NPM is a container, so loopback-only won't do. Firewalling to NPM is a later hardening step.
Data	/home/core/apps/hamlab/ — config.yaml, state.db (SQLite)	apps/ is the nightly rsync target to grid, so it's backed up for free.
Ingress	NPM proxy host hamlab.bigham.dev → :8100, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a second API call (NPM drops them on create-with-new-cert)	*.bigham.dev already resolves; the proxy host alone makes it live.
Repo	Gitea mitchell/hamlab, private, main	Created 2026-09-17.
Machines	A view shows one machine: the one in the switcher. No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it.	Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still proxies for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected.

Layout

┌──────────────────────────────────────────────────────────────┐
│ header: Ham Lab · core · load 0.42 · ● connected · [logout]    │
├──────────────┬───────────────────────────────────────────────┤
│ Dashboard    │ shell 1 ● │ backups │ +                      │
│ Processes    ├───────────────────────────────────────────────┤
│ Terminal     │                                               │
│ Services     │        main content (xterm)                   │
│ Cron         │                                               │
│ …            │                                               │
├──────────────┴───────────────────────────────────────────────┤
│ footer: v0.1.0 · up 12d · hamlab.bigham.dev                   │
└──────────────────────────────────────────────────────────────┘

• Left rail: flat list of pages, active one highlighted. No nested leaves.
• Terminal page: tab strip across the top of the pane, one tab per session, ● when a process is running, + pinned at the right edge. Active tab carries rename and close; double-click renames, middle-click closes. (Until 2026-09-18 the sessions were leaves under a collapsible Terminal group in the rail.)
• Non-MVP groups render as disabled leaves with "coming soon" — the rail shape is the product; stubs keep it honest about scope.
• Mobile: rail becomes a slide-in drawer; xterm fits the viewport; a sticky key bar above the keyboard (Esc Tab Ctrl Alt ↑ ↓ ← → |) because phone keyboards have none of those. This is what makes it usable as a PWA rather than a novelty.

Terminal semantics

• A session is a PTY + shell owned by the daemon, with an ID, name, cwd hint, and a scrollback ring buffer (256 KiB). Attaching replays the buffer, then streams.
• Any number of browser tabs can attach to the same session (all get output; all can type). Resize follows the most recent attacher.
• Switching tabs in the UI just detaches/attaches; the shell keeps running.
• Closing a session sends SIGHUP to the process group and drops it. Shell exit closes the session automatically (leaf shows "exited" until dismissed).
• Session names persist in state.db so the sidebar order survives a service restart. The shells themselves do not survive a daemon restart in the MVP — see roadmap (tmux backing).
• Idle sessions cost one shell process each; no automatic reaping in MVP.

API (MVP)

POST /api/login            {password}            → 204 + cookie
POST /api/logout                                 → 204
GET  /api/me                                     → {host, user, version, uptime, load}
GET  /api/terminals                              → [{id, name, running, created}]
POST /api/terminals        {name?}               → {id, ...}
PATCH /api/terminals/:id   {name}                → 204
DELETE /api/terminals/:id                        → 204
WS   /api/terminals/:id/attach   binary frames = raw bytes; JSON text frames = {resize}
GET  /healthz                                    → 200 (unauthenticated, for NPM)

Everything under /api except login and /healthz requires the session cookie.

Project layout

hamlab/
  cmd/hamlab/main.go          serve | passwd | version
  internal/
    auth/                     argon2id, sessions, rate limit
    term/                     PTY manager, ring buffer, attach/detach
    api/                      chi router, handlers, WS
    state/                    SQLite (modernc, no cgo)
    web/                      go:embed of web/dist
  web/                        Vite + React + TS + xterm.js, PWA manifest + SW
  deploy/
    hamlab.service
    install.sh                build → /usr/local/bin/hamlab → systemctl restart
  Makefile                    build / dev / deploy / lint / test

Milestones

M0 — skeleton (service boots, login works)

• Go module, chi, config loading, hamlab passwd, argon2id login, session cookie.
• hamlab.service on core, :8100, serves a placeholder page.
• make deploy from core itself.

M1 — terminals (the MVP feature)

• PTY manager with ring buffer, create/list/rename/close, WS attach with resize.
• React shell: header, rail, footer, Terminal group, xterm main pane.
• Persistence of session list across refresh and rail switching.

M2 — polish + PWA + ingress

• Mobile drawer, key bar, fit-on-rotate.
• manifest.webmanifest, icons (Ham logo on black, same recipe as HamTV), service worker caching the app shell only (never /api).
• NPM proxy host for hamlab.bigham.dev; verify WS through NPM; force-SSL/HSTS/HTTP2.
• Stub groups for Services / Cron / ZFS / Dashboard.

Done when: from the phone, open the PWA, log in, open two terminals, run something long in one, switch away and back, refresh — it's all still there.

Roadmap (not MVP — notes only)

• Updates tab — shipped 2026-09-17 (internal/updates, internal/tasks, Updates.tsx): upgradable packages with current → candidate, suite, security badge and description; index age, last upgrade (from /var/log/apt/history.log), unattended-upgrades on/off, reboot-required banner with the packages that caused it; Refresh index, Upgrade all (dist-upgrade), Upgrade selected; changelog per package on demand. Runs go through the new tasks package: background command, output kept, followed live over /api/tasks/{id}/stream, cancellable, listed at /api/tasks (the seed of a Task History tab). apt is fully supported; dnf/pacman list only.

• Power — header ⏻ button (amber when a reboot is pending), dialog that requires the hostname typed back, then systemctl reboot / poweroff. /api/me carries reboot_needed so the header knows.

• tmux-backed sessions so shells survive a daemon restart / redeploy. Would spawn tmux new -A -s hamlab-<id>; needs mouse on and status-bar handling in xterm.

• Services tab — systemctl list-units, per-unit page: status, last 200 journal lines (live-tail via WS), start/stop/restart/enable buttons, unit file viewer.

• Cron tab — shipped 2026-09-17 (internal/cron, Cron.tsx): one list built from /etc/crontab, /etc/cron.d/*, every user crontab (via sudo), the scripts in the run-parts directories those drive, plus systemd timers. Next run computed by an in-house cron-expression evaluator (tested: ranges, steps, names, dom-OR-dow rule, @ aliases); last run from the journal's (user) CMD (command) lines (cron logs via syslog to journald; reading needs sudo/adm). Commented-out jobs show as disabled; job-shaped lines inside comment blocks (Debian's header examples) are ignored. Per job: Run now (background, as the job's user, sh -c with crontab % rules, output captured), Enable/Disable (comments the line in place, refuses if the file changed underneath), Edit source (textarea; validated line-by-line; user crontabs installed with crontab -, system files written via sudo tee), Cron log, and Output file when the command redirects to one. Timers: Run now (starts the activated service), start/stop/enable/disable, live logs of the service via the Services tab's tail.

• Disks tab — shipped 2026-09-17 (internal/storage, Disks.tsx), replacing the planned ZFS tab: block devices from lsblk -J (kind, model, size, temperature from the device's hwmon node, partitions with filesystem usage, live read/write/busy from the metrics sampler); SMART via smartctl -j when smartmontools is present (one button installs it as a task otherwise): health, temperature, power-on time, self-test status, ATA attribute table with the failure-predicting IDs flagged, NVMe endurance and error counters, short/long self-tests. Spinning disks are read with -n standby so the tab never wakes a sleeping drive; "Wake and read" is explicit. ZFS section when the tools and module are there: pools with health, capacity, fragmentation, last scan and vdev tree; scrub start/pause/stop; datasets tree with used/refer/in-snapshots/ snapshot count/compressratio/mount; per-dataset snapshot list and "Snapshot now". Verified against grid (raidz2 tank, 10 datasets, 515 snapshots, 8 WD40EZAZ + 970 EVO Plus SMART). Snapshot destroy and dataset property editing are deliberately not there yet.

• Dashboard — first cut shipped 2026-09-17 (internal/metrics, Dashboard.tsx): CPU (total + per core), memory/swap, load, uptime, hwmon temps and fans read straight from /sys (no lm-sensors), per-interface network rates, per-disk read/write/busy from /proc/diskstats, filesystem usage, UPS via sudo -n pwrstat -status. Sampled every 2 s into an in-memory ring of 10 min; the UI (the Dashboard tab since 2026-09-19) reads it all through the sensor API — see Sensors below. Customizable: a Customize mode reorders and hides modules (CPU, Memory, Load, Uptime, Temperatures, Fans, Network, Disks, Filesystems, UPS) and hides individual rows (an interface, a mount, a disk, a sensor); the layout is stored server-side in state.json via GET/PUT /api/prefs/dashboard so every device sees the same dashboard. The update interval (100 ms – 2 s) is part of the layout and drives both the browser's polling and the daemon's sampling clock; the chart window is always 300 samples, so 30 s at 100 ms and 10 min at 2 s. Still to do: persistent history (SQLite ring, e.g. 10 s × 7 days), SMART (smartctl -j), longer time ranges.

• Sensors — shipped 2026-09-19 (internal/sensors, internal/metrics/sensors.go, internal/api/status_sensors.go, web/src/sensors.ts, widgets.tsx, Embed.tsx; reference in SENSORS.md): everything measured is a sensor with a stable dotted ID; catalog, single value, history, a shaped query (the client sends the object it wants and gets it back filled in — GraphQL's idea without GraphQL, since sensors differ per machine and a static schema would collapse to sensor(id:)), the same query as a live SSE stream, Prometheus /metrics, read-only sensors:read tokens (never full tokens in a URL), status sensors (services, updates, ZFS, SMART), a widget library, custom widgets on the Dashboard tab (briefly renamed "Monitors" 2026-09-19, back to "Dashboard" 2026-09-20 at Mitchell's request; the view id and URL were always dashboard), and a frameable /embed/widget for Glance and friends. (An external demo page was deployed at hamlab-demo.bigham.dev and linked from the Widgets tab 2026-09-19; both removed 2026-09-23 at Mitchell's request — the Widgets tab shows the same thing against the machine you are signed in to, so the second copy earned nothing and cost a public read-only token, which was revoked with it. examples/external-demo/ stays as an example to read.) The stock Dashboard modules and the Disks tab read the sensor API too (sampleAdapter.ts); /api/metrics is retired. Also per-service and per-container sensors (memory and CPU from cgroups), tokens limited to ID patterns, and /api/fleet/sensors/query for the whole fleet in one request.

• Alerts — shipped 2026-09-19 (internal/api/alerts.go, Alerts.tsx): rules over sensors (container.*.running == false for 120 s), evaluated by the hub every 15 s for itself and every managed machine in one shaped query each; hamlab.reachable is supplied by the hub so "a machine stopped answering" is a rule like any other. Ten defaults (machine offline, container down/unhealthy, failed service, disk full, pool unhealthy, SMART failing, UPS on battery, CPU hot, memory). Firing and resolved are events: kept (last 200, in state.json), shown on the hub-only Alerts tab with a count on the rail, and POSTed to channels — ntfy (title/priority headers) or a webhook (JSON; Home Assistant, Slack). The rule editor shows what a selector matches across the fleet as you type. Not done: quiet hours, per-rule channels, Web Push straight to the PWA.

• Docker tab — shipped 2026-09-19 (internal/api/docker.go, Docker.tsx): containers grouped by compose stack with live memory and CPU from the container sensors, start / stop / restart (asked first), logs that follow (polled with --since, no held connection), inspect (env names only), and images / volumes / networks tables. Works on every machine through the hub; #/grid/docker/immich_server links to one. Stacks (internal/api/stacks.go): every compose project, found from its containers' labels, with its file to edit and Deploy (up -d), Update (pull, then deploy), Restart and Stop as followed tasks. The file is looked for where the label says and through every container's mounts, because an orchestrator that runs compose inside its own container (Dockhand here; Dockge, Portainer) records a path in its own filesystem; compose is then run inside that container, so .env, relative paths and the compose labels stay exactly as deployed (verified: labels identical after a redeploy). Save validates with compose config first, keeps the previous file as <file>.hamlab-bak, and never deploys by itself. A stack deployed from another host (Dockhand on core driving grid over tcp) has no file on that machine, so the node alone lists it read-only — but the hub fills that in (hubStacks): asked about a node, it resolves the files on itself and runs compose here with DOCKER_HOST pointed at the node's tcp://…:2375, the way it was deployed; the task runs on the hub and the node's page follows it. Also per stack: every file it was deployed from (chips), its .env (values masked until "Reveal and edit"; validated in place, put back if compose objects), what Deploy would change (a line diff against the copy kept at the last deploy from here, <file>.hamlab-deployed, else the copy the last save replaced; .env values never shown), and image update checks — asked for, never polled, since each is a registry request Docker Hub counts; remembered six hours. A save refuses a file changed on disk meanwhile. Making and unmaking (2026-09-20): "+ New stack" — a name, a folder, compose text and an optional .env (written root-only), checked by compose before it is kept, then "Create and deploy" or "Create only". "+ New container" is the same dialog opened on a form (image, ports, volumes, environment, restart, command, host networking) that writes the compose for you: a container made here is a one-service stack, so it stays editable and redeployable, where a bare docker run could only be thrown away and typed again. New stacks go in a home — a folder where two or more stacks are already deployed from (Dockhand's stacks/core here; ~/stacks on a machine with none) — and are deployed the way their neighbours were (inside dockhand, or over tcp to a node from the hub). Homes are also scanned, so a stack with no containers ("not deployed": never deployed, or taken down) is still listed. Down removes a stack's containers; Delete… (type its name) takes it down and moves its own files to <home>/.hamlab-trash/<name>-<time>/ — moved, not erased; volumes and bind-mounted data are never touched. A stopped container can be removed (a running one must be stopped first, on purpose). Images: pull (a followed task), remove (unforced: Docker refuses one still in use) and prune (dangling only). Volumes (2026-09-20): each says which containers use it; one that nothing uses can be deleted — a named one by typing its name (it may be the data of a stack that is only down), an anonymous one by confirming its full name — and "Prune anonymous" clears the nameless leftovers of recreated containers, never named volumes. Images are marked "in use", and "Remove unused" takes every image no container uses. Nothing is ever forced: Docker refuses what is still in use, and that refusal is the answer. Still absent: removing networks.

• Top merged into Processes — 2026-09-23. The two tabs answered the same question from the same data (Top already read /api/procs, the endpoint the Processes table was built on), so at Mitchell's instruction there is now one tab, Processes, showing what Top showed: the btop-style whole-machine screen. Procs.tsx is deleted; the top view id and its route are gone. A rail order saved before this keeps its shape — railorder.ts maps the retired top to procs and drops the duplicate, so the merged page sits where Top sat rather than falling to the bottom.

• Dashboard retired — 2026-09-23, at Mitchell's instruction, the same reasoning that merged Top into Processes: the app repeated itself. The customisable module list is gone (Dashboard.tsx deleted), and with it /api/prefs/dashboard, applyPrefs, and the per-user dashboard blob in state.json. The Processes tab already shows those numbers on one screen. fmtBytes/fmtRate moved to fmt.ts (Disks, Files and Space use them); MONITOR_HISTORY and seriesOf went with the page. The default view after login, after switching machines, and when a hub-only page is left behind, is now Processes. A saved rail order naming dashboard degrades quietly — orderBy drops ids with no page. Widgets stays and says what it is for: a notice at the top explains these are for dashboards outside Ham Lab — Glance, Home Assistant, a wall display — served from /embed/widget with a read-only sensor token, not a second copy of the console's screens. It also closed a real bug. applyPrefs set the machine's sampling rate from intervalMs inside the dashboard layout, and its else branch reset the sampler to the default whenever that field was absent. Since layouts became per-user, one account saving a dashboard could silently reset the sampling rate for the whole machine, overriding the fleet refresh rate set in Settings. The sampler now has one source: refreshMs in the UI prefs, which the hub fans out, remembered per machine as sampler_ms.

• ZFS tab, with scheduled snapshots — shipped 2026-09-24 (internal/storage/snapshots.go, internal/api/zfs.go, ZFS.tsx). Pools, datasets and snapshots moved out of Disks — Disks is the hardware, ZFS is the filesystem on top of it — and the tab gained what it was split out for: snapshot schedules. Hourly, daily, weekly or monthly, per dataset, optionally recursive, with a retention period and a floor that keeps the newest few whatever their age. A missed window (the machine was off) runs once on return rather than being skipped. The naming is zfSnap's, deliberately. grid has run zfSnap v1.11.1 from root cron at 02:30 with a 6-week TTL since before this repo existed, and it works. So rather than invent a second convention for two tools to disagree over, this writes and reads exactly what zfSnap does — tank@2026-09-24_02.30.01--6w, retention encoded in the name. Three consequences: Ham Lab shows and expires the snapshots zfSnap already took; the two can run side by side without fighting, since both only remove what has passed the expiry written into its own name; and changing a schedule never re-dates old snapshots, because each carries the rule it was taken under. The page detects the existing cron job (zfSnap, zfs-auto-snapshot, sanoid, or a bare zfs snapshot) and says so, warning if a Ham Lab schedule is running as well. Nothing is destroyed unseen. Only names that parse as a stamp and a TTL are ever candidates for expiry, so a snapshot taken by hand is invisible to a schedule; Run now says that expired snapshots go with it; and a snapshot deleted by hand is confirmed first and needs its full name echoed back by the caller. (A "what would it delete?" preview shipped and was removed the same day as redundant — Mitchell's call.) The datasets table is a snapshot explorer: each dataset shows how many snapshots it has and what they cost, and opens into the snapshots themselves — when each was taken, when it expires (read from the TTL in its own name, so no server call), its size, and Delete. Deleting takes the same-named snapshot on descendant datasets too, which is how a recursive schedule made it. A schedule is a real cron job (2026-09-24, Mitchell's call after the first cut ran it from a ticker inside the daemon). Ham Lab writes /etc/cron.d/hamlab-zfs, owns it, and rewrites it whole on every change; each line calls hamlab zfs-snapshot <id> as the account Ham Lab runs as, reaching zfs through sudo as the daemon does. Snapshots are backup-shaped and must not stop because a console is down, upgrading, or uninstalled — and a cron line is greppable by anyone looking at the machine the normal way. A cron expression is the single description of when a schedule fires: the four common shapes render to one, custom states one outright, and the next run is read back with the same evaluator the Jobs tab uses on everyone else's lines, so the page, the file and the prediction cannot disagree. A run's outcome is written to zfs-runs/<id>.json beside the config, never to state.json, because the cron job is a separate process and the daemon would clobber it. An existing external job can be adopted — its schedule, retention and dataset read out of the cron line into a new Ham Lab schedule. zfSnap is retired on grid as of that day: Ham Lab now takes the nightly tank snapshot, the package is purged and the old cron line is gone, with the 515 snapshots it had taken left untouched and fully readable here — which is what adopting its naming bought. See hamlet/docs/hosts.md.

• KVM tab — shipped 2026-09-24 (internal/api/kvm.go, KVM.tsx, config.KVM). Paste a device's address; its own web page appears in the tab. Nothing in it knows anything about any particular KVM — the first device here is a Sipeed NanoKVM on grid at 10.0.0.217, but none of that is in the code. Why it proxies rather than framing the device directly. Ham Lab is served over TLS and a KVM usually is not, so a bare http://… iframe is blocked as mixed content before it loads. Serving the device from Ham Lab's own origin (/kvm/<id>/…) fixes that, puts it behind Ham Lab's login rather than a second one, and makes the iframe same-origin. The general problem, and the answer. A device UI asks for its assets by absolute path — /assets/app.js, /api/whatever — which no path prefix can rewrite away, and rewriting bodies per device would not be general. So the prefix is only the entry point: any request Ham Lab does not serve itself is routed to whichever KVM the asking page belongs to, read from the Referer, falling back to a cookie when there is none (a WebSocket upgrade). Ham Lab's own routes always match first, so a device can never shadow the console, and the fallback is gated on a live session. The device never receives Ham Lab's session cookie, and its X-Frame-Options/CSP/HSTS are dropped on the way out since Ham Lab is the origin now. Redirects to the device's root are rewritten back through the prefix. One catch found in testing: Ham Lab's own frame-ancestors 'none' forbids framing even from the same origin, so /kvm/ paths send 'self' instead. Without that the iframe was refused before it loaded. The keyboard needs less than it sounds: a page cannot forward keys into an iframe and does not need to, since one with focus receives them directly. Fullscreen plus navigator.keyboard.lock() is what takes Escape, Tab and Ctrl-W back from the browser; that is Chromium-only, and the tab says which version you are getting rather than pretending. One KVM per machine, and the tab follows the machine switcher (corrected the same day: the first cut was a hub-side list you pointed at machines, which had it backwards). Switch to grid and you get grid's; adding a second for a machine replaces the first. The addresses are still kept on the hub rather than on each machine, because a KVM earns its keep exactly when the machine is down and its own Ham Lab cannot answer — the hub proxies the device either way — and a KVM need not belong to a machine Ham Lab manages at all. Not done: holding the device's own login on the hub, so its page does not ask for a second password.

• Skins — 13 since 2026-09-19 (themes.ts: a palette is one mk(…) line): Dark, Light, Cyberpunk, Retro, HUD (hairlines of cyan on near-black, corner ticks, thin type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light; each repaints the terminal too. Each machine wears a colour (accents.ts): a stripe across the header and its name in the switcher — picked from ten swatches or any colour on the Machines page, else derived from its name — saved with the theme on the hub.

• Multi-host — Phase 1 shipped 2026-09-18 (internal/nodes, internal/tlsauto, MachineSwitcher.tsx, Machines.tsx). One binary, hub and spoke: a host whose config.json lists nodes is a hub; any other is a standalone node with its full UI. Node side: "tls": "auto" makes a self-signed cert in the data dir and serves HTTPS; POST /api/tokens {password,name} trades the login password for a long-lived bearer token (SHA-256 stored in tokens), accepted by the auth middleware alongside the cookie; DELETE /api/tokens/{name} revokes. Hub side: POST /api/nodes {name,url, password} peeks the node's certificate (trust on first use, fingerprint pinned), obtains a token named hub:<hostname>, stores {name,url,fingerprint,token}; a 15 s prober keeps each node's status; /api/nodes/{name}/… is a reverse proxy to the node's /api/… (bearer swapped in, cookie and Origin stripped, streaming flush) that carries WebSockets, so terminals, log tails and task output work remotely. Frontend: setMachine() rewrites every /api/… path; the rail has a machine switcher at the top and a Machines page to add/remove/open. Pages remount on switch; terminals live on the node. Nodes: grid, wyse. Phase 3 (update from hub) shipped 2026-09-18: node POST /api/self-update takes a binary, checks it answers version, keeps hamlab.prev in the data dir, installs over the executable via sudo install, replies, then restarts the service; the hub pushes its own executable with POST /api/machines/{name}/update or /api/machines-update-all (architecture checked first via /api/me.arch), the Machines page shows an outdated badge and Update buttons, and hamlab push-nodes does the same from the CLI — make deploy runs it last, so one deploy on the hub updates every machine. First install on a new host is still by hand (scp binary + install.sh, hamlab passwd, tls auto). Still to do: Phase 2 Datacenter overview; optional firewall rule limiting a node's port to the hub. Cross-architecture (2026-09-18): make deploy also builds bin/hamlab-linux-arm64, install.sh installs it beside the binary as hamlab-arm64, and the hub picks the sibling matching a node's /api/me.arch when pushing — the Mac mini (Fedora Asahi, aarch64, node mac) updates like the rest.

• Portability + install — shipped 2026-09-18 (internal/setup, internal/priv, install.sh, make release): the binary installs itself (hamlab install / uninstall; systemd or OpenRC; root, sudo or doas), all builds static on amd64, arm64, arm, 386 and riscv64, install.sh as the curl-able bootstrap with checksums. Every privileged call goes through priv instead of a literal sudo -n. Updates tab acts on dnf, zypper, pacman and apk as well as apt. Services list falls back to the plain table on systemd < 246; Disks falls back to /sys/block without lsblk. Listener uses IP_FREEBIND. Self-update: pre-flight check against the live config, atomic replace, exec in place. Config default moved to ~/.config/hamlab (existing ~/apps/hamlab still found by the installer and the CLI). Tested in containers: Debian 12, Ubuntu 22.04, Fedora 40, Rocky 8, openSUSE Leap 15.5, Alpine 3.18/3.20 (OpenRC + doas, non-root), plus the four lab hosts. Not done: an OpenRC backend for the Services tab; runit/s6/sysvinit service files; public hosting for install.sh (the Gitea repo is private, so the one-liner needs HAMLAB_TOKEN).

• Users, sessions and settings that follow the account — shipped 2026-09-22 (internal/api/users.go, config.User, store.Session/UserState, Login.tsx, Users.tsx — a hub-only Users tab in the rail, added the same day at Mitchell's request, holds everything about accounts: your own password, your sessions, the accounts; Settings keeps appearance, rates, tokens and About). Rail order is per account too (2026-09-22, railorder.ts, Rail.tsx): "⇅ Arrange" at the foot of the rail puts a handle on every page; drag one up or down (pointer events, so a finger works), let go and the order is saved in the UI prefs with the theme; "Reset" restores the built-in order; pages added in a later build go after the ones the saved order names. The login is a user name and a password. config.json holds users (name, argon2id hash, created, last seen); the old single password_hash becomes the account named after the OS user the console runs as on first boot, its signed-in sessions become that account's, and the machine-wide theme/accents/dashboard become theirs — nothing to redo after the upgrade. A session carries its user, an ID, when it was made and last seen, the IP and the browser; Users → Sessions lists yours ("Safari on iPhone · 10.0.0.5 · seen 2 min ago"), revokes one, or signs out everywhere else. Users → Users lists accounts with their session counts, adds one, removes one (their sessions and settings go too; never yourself, never the last), and resets another's password (they are signed out); removing and resetting take your own password first. Everyone signed in can do everything — a household console, not a tenancy — and each machine keeps its own accounts: a hub reaches a node with its token, so pairing (Machines → Add) now takes a user and password on the node, the user optional while it has one account. Per account: ui (theme, accents) and dashboard in state.json under users.<name>. Still the machine's: refresh rate (it is how often every machine samples — it rides in the UI prefs object for the client but is stored once, machine-wide), graph history, alert rules, terminals (the shells run as one OS user anyway). A login attempt for a missing user takes as long as a wrong password, so the form can't list accounts. Every write's log line now names the account or token (core@10.0.0.5 via …), and Access tokens say who issued them. CLI: hamlab passwd [user] (makes the account if new; reads stdin when piped), hamlab users, hamlab user-rm <user>; hamlab install names the first account after the service user. Not done: a read-only viewer role.

• Space tab — shipped 2026-09-22 (internal/space, internal/api/space.go, Space.tsx), asked for as "exactly WizTree": what is using a disk. The unit is a whole volume, picked from a dropdown at the top: a disk on its own (every filesystem its partitions back) or a ZFS pool as a whole (every mounted dataset) — a pool's member disks are listed under it with the layout (raidz2, 8 disks: sda …) and never offered apart, since one disk's share of a pool means nothing (Mitchell's call, same day, after a first cut that listed the disks as chips). The walk runs as a helper process, hamlab space-walk <roots>, as root through priv when the host allows (so /var/lib/docker and /root count), 32 workers reading directories depth-first, never crossing a filesystem boundary by itself; each root is grafted into one tree at its path. Files under 128 KiB are not kept one by one but summed per folder as "N smaller files" — what keeps a scan of millions of files to tens of MB in the daemon; the tree crosses back as a varint stream (codec.go), and the helper's memory goes with it. core's root: 646k files in ~1.2 s. Progress (files, bytes, current folder) every 250 ms; Cancel sends SIGTERM. The page: picker → Scan/Rescan → Folders (a tree largest-first: size, share of parent as a bar, files, folders, modified; expand in place) over a treemap of the selected folder — squarified, nested to the file, tiles coloured by kind (video, images, audio, archives & disk images, documents, code & config, databases & logs, other; the validated categorical set, dark and light) with folders as labelled boxes; hover for path, size and share; click a folder to drill, breadcrumbs and ↑ to come back. The map asks the server for the folder pruned to what a tile could show (min = 1/6000 of the folder, 24 deep; deeper folders come back as one lump), the list one level at a time. Largest files: the 1,000 biggest with a path filter. Works on every machine through the hub. Smarter and faster (2026-09-23, at Mitchell's request): a scan is kept — in memory and in <data>/space/<target>.scan (the same varint stream) + .json, read back after a restart — and the next scan starts from it: the helper is handed the last tree on stdin and reads a directory again only when something says it changed. On a ZFS pool with snapshots, the scan records the newest snapshot of each dataset at start (bases), and the next one runs zfs diff -H -F <base> per dataset: only the directories it names are read, their ancestors kept but looked into, everything else kept wholesale without a stat. The diffs run side by side and one is given three minutes — timeout -s KILL inside sudo, since Debian's sudo gives the command its own session and a process-group kill from outside never reached zfs — after which that dataset is walked by mtime instead. On tank a diff is metadata-bound like the walk: 8 s to 3 min per dataset for a handful of changes, and immich, which changes constantly, ran past twenty minutes the first time. A refresh of tank took 4 min 19 s against 7 min for the full scan. Elsewhere (ext4, or a dataset with no snapshot) a directory whose mtime — to the nanosecond — is unchanged keeps its list and only its kept files are stat'ed again, since a file can grow without its directory noticing. A cancelled scan keeps what it walked, marks the folders it never got to (partial, "not yet read" in the tree) and Resume reads just those. core: full 1.2 s → refresh 0.3 s (2 folders read, 84,755 kept). The status line says which (method, by_root, read/kept counts); Forget drops the saved result so the next scan is full. The old result stays browsable while a refresh runs. Treemap rethought the same day: every file a tile grouped by folder — folders are only a thin line and a name strip on the first level — instead of nested boxes with headers, which on Linux's single-child folder chains was a staircase of headers around nothing. Not done: delete from here, a size-over-time history, per-user column choices.

• Updater — shipped 2026-09-23 (internal/selfupdate, internal/api/update.go, the Updates card in Settings.tsx, scripts/gitea-release.sh). A Ham Lab asks one URL what the newest build is and installs it on request. update.url in config.json is either a Gitea/Forgejo repo API base (…/api/v1/repos/<owner>/<repo>, whose newest release names the version and carries the binaries as assets) or any static directory holding VERSION, SHA256SUMS and hamlab-linux-<arch> — which is exactly what make release writes and what install.sh already downloads, so the demo site, a folder on a NAS and a Gitea release are all the same thing to it. "off" never looks. update.token is sent when the source is behind a login, so a private Gitea serves its own fleet without the instance being opened to the world. The binary is verified against SHA256SUMS before it goes anywhere near the running one, then takes the same path POST /api/self-update always did (runs version, runs check against this machine's config, atomic replace keeping hamlab.prev, re-exec in place) — that path is now one shared method, installBinary. Version ordering is install.sh's vnum() rule ported to Go and tested against it, so the daemon and the installer never disagree about which of two builds is newer; the commit count since the tag counts, the hash does not. Checked ~90 s after boot and daily (update.check: "never" switches that off), never installed unasked — an update restarts the daemon and drops open terminals, and the dialog says so. Settings → Updates: installed, published, when it last looked, the source, the release notes, Check for updates, Install; the rail badges Settings when a newer build is out (update_available rides on /api/me). Publishing is make gitea-release → scripts/gitea-release.sh, which creates the release and uploads the five architectures, SHA256SUMS, install.sh and VERSION; it is deliberately not part of make deploy, because deploy pushes this build to the machines this hub owns while publishing offers it to every install anywhere, and one tag per deploy would be noise. Re-publishing a version needs --force. Public since 2026-09-23, at Mitchell's instruction and after the trade was put to him twice: Gitea's instance-wide REQUIRE_SIGNIN_VIEW is off and mitchell/hamlab is marked public, so anyone can browse it, clone it and download a release with no account. The other five repos there are each marked private and are invisible — an anonymous request for one is a 404, not a login prompt — registration is still disabled, and new repos still default to private. No token is needed by anything any more (the read-only one was revoked; only hamlab-publish, which creates releases, remains). Gitea resolves releases/download/latest/<asset>, so install.sh and HAMLAB_URL now point straight at the newest release and never need changing. What is public is the source, the history, BUGS.md and this file — which describes the lab and names the console's hostname.

• Auth hardening — WebAuthn passkeys (phone Face ID), TOTP fallback, optional NPM access list restricting to WireGuard/LAN.

• Files tab — shipped 2026-09-18 (internal/files, Files.tsx): browse from / with breadcrumbs, an editable path box, sort, hidden-file toggle and a filter; rows show type icon, size, modified, perms and owner, link targets and broken links. Text files (≤ 2 MiB, sniffed) open in an editor with Ctrl+S / Save; images preview; everything else downloads. Upload via drag-and-drop or picker with per-file progress (streamed, no size limit). New folder / file, rename, move, chmod, copy path, delete (folders need the name typed back), and "open in terminal", which cds the active shell (or a fresh one) to the folder. Sudo toggle: root access re-executes the same binary as hamlab fs <op> under sudo -n, streaming bodies over stdin/stdout, so both modes share one implementation and one set of rules (absolute paths only; /, /etc, /usr, … are never deleted or moved whole). Every write is logged with the caller's IP.

• Docker read-only view — docker ps/logs; dockhand stays the source of truth.

• Notifications — push on service failure / SMART warning / UPS event.

• Terminal extras — search, copy-mode, font/theme, split panes, upload via drag-drop into cwd.

• Settings + themes — shipped 2026-09-18 (Settings.tsx, themes.ts): Settings page with Appearance (Dark, Light, Cyberpunk, Retro — CSS variables on <html> plus a matching xterm palette, applied before first paint from localStorage and synced hub-side via GET/PUT /api/prefs/ui so the choice follows the login), Account (change password, POST /api/password, current password required), Access tokens (list/revoke the tokens hubs hold), About (version, host, arch, uptime, certificate).

To do

• Grow the Files tab until copyparty can go — asked for 2026-09-23 as two items (Files polish, and deprecating copyparty), merged 2026-09-24: they are the same tab and the same work. copyparty on grid (copyparty.service, :3923, /etc/copyparty.conf, serving all of /tank as user core with one account behind NPM) overlaps Files almost entirely. The aim is to stop the service and keep only Ham Lab, with general ease-of-use polish along the way. What Files must gain first, in the order that matters:

  • Multi-select and bulk actions. copyparty selects many rows and moves, deletes or downloads them together; Files acts on one row at a time.
  • Download a folder, and a multi-file download, as a streamed zip or tar — the thing copyparty is reached for most.
  • Resumable uploads. copyparty's up2k chunks a file, survives a dropped connection and skips what is already there; Files streams one PUT, so a phone upload that drops at 90% starts over.
  • A gallery. Thumbnails for a folder of images and video, and play audio/video in place — /tank/media and /tank/immich are most of what is browsed.
  • Search below the current folder by name, since find in a terminal is the fallback today.
  • Decide when built: whether share links (copyparty's public per-path links) are wanted at all — they are the one feature that hands out access without a login, and Ham Lab has no notion of that; if not, say so in the docs rather than leaving a gap unexplained.
  • After: systemctl disable --now copyparty, remove its NPM host, keep /etc/copyparty.conf until a rollback is clearly not needed, then update hamlet's docs/hosts.md bare-metal list, docs/gotchas.md (the "docker ps is not the inventory" note names it) and the grid row in CLAUDE.md.

• Grow the Docker tab until Dockhand can go — asked for 2026-09-23 and widened 2026-09-24 to take in Docker tab polish and feature work, on the same reasoning as the Files item: it is one tab, and the gaps that block retiring the old tool and the things that would simply make it better are the same body of work. The homelab half — the cutover itself, then moving every stack off Dockhand's data dir — is hamlet's docs/pending.md → "Retire Dockhand; Ham Lab becomes the orchestrator", agreed 2026-09-20. This entry is the app half.

  Done 2026-09-24 — the eight Mitchell picked from the inventory, all in the Docker tab: per-container Stats, Health (with the last few probe outputs), Events (Ham Lab keeps its own 4000-entry ring, since docker events forgets after about a minute), Shell (docker exec, handed to the Terminal tab) and Limits (docker update for memory, CPU and restart policy); an updates tab giving the whole machine at once with a per-container indicator in the list; and schedules, which write /etc/cron.d/hamlab-docker and call hamlab docker-update <id> — the shape the ZFS schedules already use. Also done: Ham Lab keeps working when dockhand is removed. It remembers where a container's bind mounts pointed (mounts in state.json) and resolves a stack through that record when no running container explains its path, running compose on the host instead; repoint does the same for a schedule at run time. Without this, removing the container would have made 17 stacks on core and 5 on grid unfindable at once.

  • Left from the Dockhand inventory, both to decide rather than build.
    • Git-sourced stacks. Dockhand's stack_sources table points a stack at a repo and pulls it. Its git_repositories and git_stacks tables are empty, so nothing here uses it — dropping it is the honest answer unless Mitchell says otherwise.
    • Webhooks, OIDC, LDAP, vulnerability scans. All four tables empty. Nothing to move.
  • Known gaps in the tab itself, worth closing whether or not Dockhand goes:
    • Removing networks — volumes and images can be pruned and removed, networks cannot.
    • Logs are polled with --since rather than followed over a held connection. That was deliberate and works, but a long tail is choppier than the Services tab's live tail, which does hold a connection. Worth reconciling the two.
  • The *:2375 question, which this is the moment to settle. A stack deployed to grid or the Mac still goes over tcp://<node>:2375 from the hub. Decide whether the hub should instead ask the node's own Ham Lab to deploy locally, which would end the unauthenticated Docker socket exposure — its own item in docs/pending.md, currently filed under "decided, no action", and this is the change that would make closing it free.

• A Virtualization tab — asked for 2026-09-23, not started, and large. Run and manage LXC containers and virtual machines from the console, the way Proxmox does — which is the comparison Ham Lab has invited since its first line. Nothing exists yet. What a future session has to settle before writing code: which stack (libvirt/KVM + virsh is the obvious one for VMs; LXC or Incus for containers, and Incus covers both); whether it manages what is already installed or installs it; where disk images live (tank on grid, local NVMe elsewhere); console access (SPICE/VNC proxied through the daemon, or a serial console over the existing terminal machinery, which would reuse internal/term); snapshots and clones; and what it does on a host with no virtualisation at all, which is every host here today. Worth scoping as its own plan before starting.

Open questions

• Idle-session reaping policy (never / after N days) — leaning never; it's a single-user box.
• Whether sudo -i-style root sessions get their own leaf colour. Cosmetic.