hamlab/PLAN.md

Ham Lab — MVP plan

Proxmox-style web console for a bare-metal Debian 13 host. First target: core (10.0.0.234). MVP is service + shell + persistent terminals; everything else is a stub tab or a roadmap item.

Decisions

Area	Decision	Why
Backend	Go 1.24, single static binary, frontend embedded via embed	core has ~370 MiB free RAM; a Go daemon idles at ~15 MiB vs ~80+ for Node. One file to deploy, no nvm path in the unit. Same binary later installs on grid unchanged.
PTY	creack/pty — server-owned PTYs running core's login shell	Sessions live in the daemon, not the browser, so refresh/tab-switch reattaches. No tmux dependency (not installed on core).
Transport	One WebSocket per attached terminal; REST for everything else	Standard; NPM passes WebSockets with one toggle.
Frontend	React + TypeScript + Vite, xterm.js (+ fit, webgl, web-links addons)	Matches calorie-counter / wedding-website, so no new toolchain to learn.
Auth	Single-user password login, argon2id hash, HttpOnly/Secure/SameSite=Strict session cookie, login rate-limit, Origin check on WS upgrade	This is a root-capable shell on the public internet. Auth is MVP, not roadmap.
Service	hamlab.service, User=core, Restart=always, binary at /usr/local/bin/hamlab	Same pattern as calorie-counter.service. Runs as core so sudo works in the terminal without a password, as on the SSH path.
Listen	10.0.0.234:8100 (free)	NPM is a container, so loopback-only won't do. Firewalling to NPM is a later hardening step.
Data	/home/core/apps/hamlab/ — config.yaml, state.db (SQLite)	apps/ is the nightly rsync target to grid, so it's backed up for free.
Ingress	NPM proxy host hamlab.bigham.dev → :8100, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a second API call (NPM drops them on create-with-new-cert)	*.bigham.dev already resolves; the proxy host alone makes it live.
Repo	Gitea mitchell/hamlab, private, main	Created 2026-09-17.

Layout

┌──────────────────────────────────────────────────────────────┐
│ header: Ham Lab · core · load 0.42 · ● connected · [logout]    │
├──────────────┬───────────────────────────────────────────────┤
│ Dashboard    │ shell 1 ● │ backups │ +                      │
│ Processes    ├───────────────────────────────────────────────┤
│ Terminal     │                                               │
│ Services     │        main content (xterm)                   │
│ Cron         │                                               │
│ …            │                                               │
├──────────────┴───────────────────────────────────────────────┤
│ footer: v0.1.0 · up 12d · hamlab.bigham.dev                   │
└──────────────────────────────────────────────────────────────┘

• Left rail: flat list of pages, active one highlighted. No nested leaves.
• Terminal page: tab strip across the top of the pane, one tab per session, ● when a process is running, + pinned at the right edge. Active tab carries rename and close; double-click renames, middle-click closes. (Until 2026-09-18 the sessions were leaves under a collapsible Terminal group in the rail.)
• Non-MVP groups render as disabled leaves with "coming soon" — the rail shape is the product; stubs keep it honest about scope.
• Mobile: rail becomes a slide-in drawer; xterm fits the viewport; a sticky key bar above the keyboard (Esc Tab Ctrl Alt ↑ ↓ ← → |) because phone keyboards have none of those. This is what makes it usable as a PWA rather than a novelty.

Terminal semantics

• A session is a PTY + shell owned by the daemon, with an ID, name, cwd hint, and a scrollback ring buffer (256 KiB). Attaching replays the buffer, then streams.
• Any number of browser tabs can attach to the same session (all get output; all can type). Resize follows the most recent attacher.
• Switching tabs in the UI just detaches/attaches; the shell keeps running.
• Closing a session sends SIGHUP to the process group and drops it. Shell exit closes the session automatically (leaf shows "exited" until dismissed).
• Session names persist in state.db so the sidebar order survives a service restart. The shells themselves do not survive a daemon restart in the MVP — see roadmap (tmux backing).
• Idle sessions cost one shell process each; no automatic reaping in MVP.

API (MVP)

POST /api/login            {password}            → 204 + cookie
POST /api/logout                                 → 204
GET  /api/me                                     → {host, user, version, uptime, load}
GET  /api/terminals                              → [{id, name, running, created}]
POST /api/terminals        {name?}               → {id, ...}
PATCH /api/terminals/:id   {name}                → 204
DELETE /api/terminals/:id                        → 204
WS   /api/terminals/:id/attach   binary frames = raw bytes; JSON text frames = {resize}
GET  /healthz                                    → 200 (unauthenticated, for NPM)

Everything under /api except login and /healthz requires the session cookie.

Project layout

hamlab/
  cmd/hamlab/main.go          serve | passwd | version
  internal/
    auth/                     argon2id, sessions, rate limit
    term/                     PTY manager, ring buffer, attach/detach
    api/                      chi router, handlers, WS
    state/                    SQLite (modernc, no cgo)
    web/                      go:embed of web/dist
  web/                        Vite + React + TS + xterm.js, PWA manifest + SW
  deploy/
    hamlab.service
    install.sh                build → /usr/local/bin/hamlab → systemctl restart
  Makefile                    build / dev / deploy / lint / test

Milestones

M0 — skeleton (service boots, login works)

• Go module, chi, config loading, hamlab passwd, argon2id login, session cookie.
• hamlab.service on core, :8100, serves a placeholder page.
• make deploy from core itself.

M1 — terminals (the MVP feature)

• PTY manager with ring buffer, create/list/rename/close, WS attach with resize.
• React shell: header, rail, footer, Terminal group, xterm main pane.
• Persistence of session list across refresh and rail switching.

M2 — polish + PWA + ingress

• Mobile drawer, key bar, fit-on-rotate.
• manifest.webmanifest, icons (Ham logo on black, same recipe as HamTV), service worker caching the app shell only (never /api).
• NPM proxy host for hamlab.bigham.dev; verify WS through NPM; force-SSL/HSTS/HTTP2.
• Stub groups for Services / Cron / ZFS / Dashboard.

Done when: from the phone, open the PWA, log in, open two terminals, run something long in one, switch away and back, refresh — it's all still there.

Roadmap (not MVP — notes only)

• Updates tab — shipped 2026-09-17 (internal/updates, internal/tasks, Updates.tsx): upgradable packages with current → candidate, suite, security badge and description; index age, last upgrade (from /var/log/apt/history.log), unattended-upgrades on/off, reboot-required banner with the packages that caused it; Refresh index, Upgrade all (dist-upgrade), Upgrade selected; changelog per package on demand. Runs go through the new tasks package: background command, output kept, followed live over /api/tasks/{id}/stream, cancellable, listed at /api/tasks (the seed of a Task History tab). apt is fully supported; dnf/pacman list only.

• Power — header ⏻ button (amber when a reboot is pending), dialog that requires the hostname typed back, then systemctl reboot / poweroff. /api/me carries reboot_needed so the header knows.

• tmux-backed sessions so shells survive a daemon restart / redeploy. Would spawn tmux new -A -s hamlab-<id>; needs mouse on and status-bar handling in xterm.

• Services tab — systemctl list-units, per-unit page: status, last 200 journal lines (live-tail via WS), start/stop/restart/enable buttons, unit file viewer.

• Cron tab — shipped 2026-09-17 (internal/cron, Cron.tsx): one list built from /etc/crontab, /etc/cron.d/*, every user crontab (via sudo), the scripts in the run-parts directories those drive, plus systemd timers. Next run computed by an in-house cron-expression evaluator (tested: ranges, steps, names, dom-OR-dow rule, @ aliases); last run from the journal's (user) CMD (command) lines (cron logs via syslog to journald; reading needs sudo/adm). Commented-out jobs show as disabled; job-shaped lines inside comment blocks (Debian's header examples) are ignored. Per job: Run now (background, as the job's user, sh -c with crontab % rules, output captured), Enable/Disable (comments the line in place, refuses if the file changed underneath), Edit source (textarea; validated line-by-line; user crontabs installed with crontab -, system files written via sudo tee), Cron log, and Output file when the command redirects to one. Timers: Run now (starts the activated service), start/stop/enable/disable, live logs of the service via the Services tab's tail.

• Disks tab — shipped 2026-09-17 (internal/storage, Disks.tsx), replacing the planned ZFS tab: block devices from lsblk -J (kind, model, size, temperature from the device's hwmon node, partitions with filesystem usage, live read/write/busy from the metrics sampler); SMART via smartctl -j when smartmontools is present (one button installs it as a task otherwise): health, temperature, power-on time, self-test status, ATA attribute table with the failure-predicting IDs flagged, NVMe endurance and error counters, short/long self-tests. Spinning disks are read with -n standby so the tab never wakes a sleeping drive; "Wake and read" is explicit. ZFS section when the tools and module are there: pools with health, capacity, fragmentation, last scan and vdev tree; scrub start/pause/stop; datasets tree with used/refer/in-snapshots/ snapshot count/compressratio/mount; per-dataset snapshot list and "Snapshot now". Verified against grid (raidz2 tank, 10 datasets, 515 snapshots, 8 WD40EZAZ + 970 EVO Plus SMART). Snapshot destroy and dataset property editing are deliberately not there yet.

• Dashboard — first cut shipped 2026-09-17 (internal/metrics, Dashboard.tsx): CPU (total + per core), memory/swap, load, uptime, hwmon temps and fans read straight from /sys (no lm-sensors), per-interface network rates, per-disk read/write/busy from /proc/diskstats, filesystem usage, UPS via sudo -n pwrstat -status. Sampled every 2 s into an in-memory ring of 10 min; the UI (the Dashboard tab since 2026-09-19) reads it all through the sensor API — see Sensors below. Customizable: a Customize mode reorders and hides modules (CPU, Memory, Load, Uptime, Temperatures, Fans, Network, Disks, Filesystems, UPS) and hides individual rows (an interface, a mount, a disk, a sensor); the layout is stored server-side in state.json via GET/PUT /api/prefs/dashboard so every device sees the same dashboard. The update interval (100 ms – 2 s) is part of the layout and drives both the browser's polling and the daemon's sampling clock; the chart window is always 300 samples, so 30 s at 100 ms and 10 min at 2 s. Still to do: persistent history (SQLite ring, e.g. 10 s × 7 days), SMART (smartctl -j), longer time ranges.

• Sensors — shipped 2026-09-19 (internal/sensors, internal/metrics/sensors.go, internal/api/status_sensors.go, web/src/sensors.ts, widgets.tsx, Embed.tsx; reference in SENSORS.md): everything measured is a sensor with a stable dotted ID; catalog, single value, history, a shaped query (the client sends the object it wants and gets it back filled in — GraphQL's idea without GraphQL, since sensors differ per machine and a static schema would collapse to sensor(id:)), the same query as a live SSE stream, Prometheus /metrics, read-only sensors:read tokens (never full tokens in a URL), status sensors (services, updates, ZFS, SMART), a widget library, custom widgets on the Dashboard tab (briefly renamed "Monitors" 2026-09-19, back to "Dashboard" 2026-09-20 at Mitchell's request; the view id and URL were always dashboard), and a frameable /embed/widget for Glance and friends. (An external demo page was deployed at hamlab-demo.bigham.dev and linked from the Widgets tab 2026-09-19; both removed 2026-09-23 at Mitchell's request — the Widgets tab shows the same thing against the machine you are signed in to, so the second copy earned nothing and cost a public read-only token, which was revoked with it. examples/external-demo/ stays as an example to read.) The stock Dashboard modules and the Disks tab read the sensor API too (sampleAdapter.ts); /api/metrics is retired. Also per-service and per-container sensors (memory and CPU from cgroups), tokens limited to ID patterns, and /api/fleet/sensors/query for the whole fleet in one request.

• Alerts — shipped 2026-09-19 (internal/api/alerts.go, Alerts.tsx): rules over sensors (container.*.running == false for 120 s), evaluated by the hub every 15 s for itself and every managed machine in one shaped query each; hamlab.reachable is supplied by the hub so "a machine stopped answering" is a rule like any other. Ten defaults (machine offline, container down/unhealthy, failed service, disk full, pool unhealthy, SMART failing, UPS on battery, CPU hot, memory). Firing and resolved are events: kept (last 200, in state.json), shown on the hub-only Alerts tab with a count on the rail, and POSTed to channels — ntfy (title/priority headers) or a webhook (JSON; Home Assistant, Slack). The rule editor shows what a selector matches across the fleet as you type. Not done: quiet hours, per-rule channels, Web Push straight to the PWA.

• Docker tab — shipped 2026-09-19 (internal/api/docker.go, Docker.tsx): containers grouped by compose stack with live memory and CPU from the container sensors, start / stop / restart (asked first), logs that follow (polled with --since, no held connection), inspect (env names only), and images / volumes / networks tables. Works on every machine through the hub; #/grid/docker/immich_server links to one. Stacks (internal/api/stacks.go): every compose project, found from its containers' labels, with its file to edit and Deploy (up -d), Update (pull, then deploy), Restart and Stop as followed tasks. The file is looked for where the label says and through every container's mounts, because an orchestrator that runs compose inside its own container (Dockhand here; Dockge, Portainer) records a path in its own filesystem; compose is then run inside that container, so .env, relative paths and the compose labels stay exactly as deployed (verified: labels identical after a redeploy). Save validates with compose config first, keeps the previous file as <file>.hamlab-bak, and never deploys by itself. A stack deployed from another host (Dockhand on core driving grid over tcp) has no file on that machine, so the node alone lists it read-only — but the hub fills that in (hubStacks): asked about a node, it resolves the files on itself and runs compose here with DOCKER_HOST pointed at the node's tcp://…:2375, the way it was deployed; the task runs on the hub and the node's page follows it. Also per stack: every file it was deployed from (chips), its .env (values masked until "Reveal and edit"; validated in place, put back if compose objects), what Deploy would change (a line diff against the copy kept at the last deploy from here, <file>.hamlab-deployed, else the copy the last save replaced; .env values never shown), and image update checks — asked for, never polled, since each is a registry request Docker Hub counts; remembered six hours. A save refuses a file changed on disk meanwhile. Making and unmaking (2026-09-20): "+ New stack" — a name, a folder, compose text and an optional .env (written root-only), checked by compose before it is kept, then "Create and deploy" or "Create only". "+ New container" is the same dialog opened on a form (image, ports, volumes, environment, restart, command, host networking) that writes the compose for you: a container made here is a one-service stack, so it stays editable and redeployable, where a bare docker run could only be thrown away and typed again. New stacks go in a home — a folder where two or more stacks are already deployed from (Dockhand's stacks/core here; ~/stacks on a machine with none) — and are deployed the way their neighbours were (inside dockhand, or over tcp to a node from the hub). Homes are also scanned, so a stack with no containers ("not deployed": never deployed, or taken down) is still listed. Down removes a stack's containers; Delete… (type its name) takes it down and moves its own files to <home>/.hamlab-trash/<name>-<time>/ — moved, not erased; volumes and bind-mounted data are never touched. A stopped container can be removed (a running one must be stopped first, on purpose). Images: pull (a followed task), remove (unforced: Docker refuses one still in use) and prune (dangling only). Volumes (2026-09-20): each says which containers use it; one that nothing uses can be deleted — a named one by typing its name (it may be the data of a stack that is only down), an anonymous one by confirming its full name — and "Prune anonymous" clears the nameless leftovers of recreated containers, never named volumes. Images are marked "in use", and "Remove unused" takes every image no container uses. Nothing is ever forced: Docker refuses what is still in use, and that refusal is the answer. Still absent: removing networks.

• Top merged into Processes — 2026-09-23. The two tabs answered the same question from the same data (Top already read /api/procs, the endpoint the Processes table was built on), so at Mitchell's instruction there is now one tab, Processes, showing what Top showed: the btop-style whole-machine screen. Procs.tsx is deleted; the top view id and its route are gone. A rail order saved before this keeps its shape — railorder.ts maps the retired top to procs and drops the duplicate, so the merged page sits where Top sat rather than falling to the bottom.

• Dashboard retired — 2026-09-23, at Mitchell's instruction, the same reasoning that merged Top into Processes: the app repeated itself. The customisable module list is gone (Dashboard.tsx deleted), and with it /api/prefs/dashboard, applyPrefs, and the per-user dashboard blob in state.json. The Processes tab already shows those numbers on one screen. fmtBytes/fmtRate moved to fmt.ts (Disks, Files and Space use them); MONITOR_HISTORY and seriesOf went with the page. The default view after login, after switching machines, and when a hub-only page is left behind, is now Processes. A saved rail order naming dashboard degrades quietly — orderBy drops ids with no page. Widgets stays and says what it is for: a notice at the top explains these are for dashboards outside Ham Lab — Glance, Home Assistant, a wall display — served from /embed/widget with a read-only sensor token, not a second copy of the console's screens. It also closed a real bug. applyPrefs set the machine's sampling rate from intervalMs inside the dashboard layout, and its else branch reset the sampler to the default whenever that field was absent. Since layouts became per-user, one account saving a dashboard could silently reset the sampling rate for the whole machine, overriding the fleet refresh rate set in Settings. The sampler now has one source: refreshMs in the UI prefs, which the hub fans out, remembered per machine as sampler_ms.

• ZFS tab, with scheduled snapshots — shipped 2026-09-24 (internal/storage/snapshots.go, internal/api/zfs.go, ZFS.tsx). Pools, datasets and snapshots moved out of Disks — Disks is the hardware, ZFS is the filesystem on top of it — and the tab gained what it was split out for: snapshot schedules. Hourly, daily, weekly or monthly, per dataset, optionally recursive, with a retention period and a floor that keeps the newest few whatever their age. A missed window (the machine was off) runs once on return rather than being skipped. The naming is zfSnap's, deliberately. grid has run zfSnap v1.11.1 from root cron at 02:30 with a 6-week TTL since before this repo existed, and it works. So rather than invent a second convention for two tools to disagree over, this writes and reads exactly what zfSnap does — tank@2026-09-24_02.30.01--6w, retention encoded in the name. Three consequences: Ham Lab shows and expires the snapshots zfSnap already took; the two can run side by side without fighting, since both only remove what has passed the expiry written into its own name; and changing a schedule never re-dates old snapshots, because each carries the rule it was taken under. The page detects the existing cron job (zfSnap, zfs-auto-snapshot, sanoid, or a bare zfs snapshot) and says so, warning if a Ham Lab schedule is running as well. Nothing is destroyed unseen. Every schedule answers what would it delete? with the list, sizes and dates, before Run now touches anything; only names that parse as a stamp and a TTL are candidates, so a snapshot taken by hand is invisible to it; and a snapshot deleted by hand needs its full name typed back. The scheduler is a 30 s tick in the daemon, and Run now calls the same code, so the button and the timer cannot drift apart.

• Skins — 13 since 2026-09-19 (themes.ts: a palette is one mk(…) line): Dark, Light, Cyberpunk, Retro, HUD (hairlines of cyan on near-black, corner ticks, thin type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light; each repaints the terminal too. Each machine wears a colour (accents.ts): a stripe across the header and its name in the switcher — picked from ten swatches or any colour on the Machines page, else derived from its name — saved with the theme on the hub.

• Multi-host — Phase 1 shipped 2026-09-18 (internal/nodes, internal/tlsauto, MachineSwitcher.tsx, Machines.tsx). One binary, hub and spoke: a host whose config.json lists nodes is a hub; any other is a standalone node with its full UI. Node side: "tls": "auto" makes a self-signed cert in the data dir and serves HTTPS; POST /api/tokens {password,name} trades the login password for a long-lived bearer token (SHA-256 stored in tokens), accepted by the auth middleware alongside the cookie; DELETE /api/tokens/{name} revokes. Hub side: POST /api/nodes {name,url, password} peeks the node's certificate (trust on first use, fingerprint pinned), obtains a token named hub:<hostname>, stores {name,url,fingerprint,token}; a 15 s prober keeps each node's status; /api/nodes/{name}/… is a reverse proxy to the node's /api/… (bearer swapped in, cookie and Origin stripped, streaming flush) that carries WebSockets, so terminals, log tails and task output work remotely. Frontend: setMachine() rewrites every /api/… path; the rail has a machine switcher at the top and a Machines page to add/remove/open. Pages remount on switch; terminals live on the node. Nodes: grid, wyse. Phase 3 (update from hub) shipped 2026-09-18: node POST /api/self-update takes a binary, checks it answers version, keeps hamlab.prev in the data dir, installs over the executable via sudo install, replies, then restarts the service; the hub pushes its own executable with POST /api/machines/{name}/update or /api/machines-update-all (architecture checked first via /api/me.arch), the Machines page shows an outdated badge and Update buttons, and hamlab push-nodes does the same from the CLI — make deploy runs it last, so one deploy on the hub updates every machine. First install on a new host is still by hand (scp binary + install.sh, hamlab passwd, tls auto). Still to do: Phase 2 Datacenter overview; optional firewall rule limiting a node's port to the hub. Cross-architecture (2026-09-18): make deploy also builds bin/hamlab-linux-arm64, install.sh installs it beside the binary as hamlab-arm64, and the hub picks the sibling matching a node's /api/me.arch when pushing — the Mac mini (Fedora Asahi, aarch64, node mac) updates like the rest.

• Portability + install — shipped 2026-09-18 (internal/setup, internal/priv, install.sh, make release): the binary installs itself (hamlab install / uninstall; systemd or OpenRC; root, sudo or doas), all builds static on amd64, arm64, arm, 386 and riscv64, install.sh as the curl-able bootstrap with checksums. Every privileged call goes through priv instead of a literal sudo -n. Updates tab acts on dnf, zypper, pacman and apk as well as apt. Services list falls back to the plain table on systemd < 246; Disks falls back to /sys/block without lsblk. Listener uses IP_FREEBIND. Self-update: pre-flight check against the live config, atomic replace, exec in place. Config default moved to ~/.config/hamlab (existing ~/apps/hamlab still found by the installer and the CLI). Tested in containers: Debian 12, Ubuntu 22.04, Fedora 40, Rocky 8, openSUSE Leap 15.5, Alpine 3.18/3.20 (OpenRC + doas, non-root), plus the four lab hosts. Not done: an OpenRC backend for the Services tab; runit/s6/sysvinit service files; public hosting for install.sh (the Gitea repo is private, so the one-liner needs HAMLAB_TOKEN).

• Users, sessions and settings that follow the account — shipped 2026-09-22 (internal/api/users.go, config.User, store.Session/UserState, Login.tsx, Users.tsx — a hub-only Users tab in the rail, added the same day at Mitchell's request, holds everything about accounts: your own password, your sessions, the accounts; Settings keeps appearance, rates, tokens and About). Rail order is per account too (2026-09-22, railorder.ts, Rail.tsx): "⇅ Arrange" at the foot of the rail puts a handle on every page; drag one up or down (pointer events, so a finger works), let go and the order is saved in the UI prefs with the theme; "Reset" restores the built-in order; pages added in a later build go after the ones the saved order names. The login is a user name and a password. config.json holds users (name, argon2id hash, created, last seen); the old single password_hash becomes the account named after the OS user the console runs as on first boot, its signed-in sessions become that account's, and the machine-wide theme/accents/dashboard become theirs — nothing to redo after the upgrade. A session carries its user, an ID, when it was made and last seen, the IP and the browser; Users → Sessions lists yours ("Safari on iPhone · 10.0.0.5 · seen 2 min ago"), revokes one, or signs out everywhere else. Users → Users lists accounts with their session counts, adds one, removes one (their sessions and settings go too; never yourself, never the last), and resets another's password (they are signed out); removing and resetting take your own password first. Everyone signed in can do everything — a household console, not a tenancy — and each machine keeps its own accounts: a hub reaches a node with its token, so pairing (Machines → Add) now takes a user and password on the node, the user optional while it has one account. Per account: ui (theme, accents) and dashboard in state.json under users.<name>. Still the machine's: refresh rate (it is how often every machine samples — it rides in the UI prefs object for the client but is stored once, machine-wide), graph history, alert rules, terminals (the shells run as one OS user anyway). A login attempt for a missing user takes as long as a wrong password, so the form can't list accounts. Every write's log line now names the account or token (core@10.0.0.5 via …), and Access tokens say who issued them. CLI: hamlab passwd [user] (makes the account if new; reads stdin when piped), hamlab users, hamlab user-rm <user>; hamlab install names the first account after the service user. Not done: a read-only viewer role.

• Space tab — shipped 2026-09-22 (internal/space, internal/api/space.go, Space.tsx), asked for as "exactly WizTree": what is using a disk. The unit is a whole volume, picked from a dropdown at the top: a disk on its own (every filesystem its partitions back) or a ZFS pool as a whole (every mounted dataset) — a pool's member disks are listed under it with the layout (raidz2, 8 disks: sda …) and never offered apart, since one disk's share of a pool means nothing (Mitchell's call, same day, after a first cut that listed the disks as chips). The walk runs as a helper process, hamlab space-walk <roots>, as root through priv when the host allows (so /var/lib/docker and /root count), 32 workers reading directories depth-first, never crossing a filesystem boundary by itself; each root is grafted into one tree at its path. Files under 128 KiB are not kept one by one but summed per folder as "N smaller files" — what keeps a scan of millions of files to tens of MB in the daemon; the tree crosses back as a varint stream (codec.go), and the helper's memory goes with it. core's root: 646k files in ~1.2 s. Progress (files, bytes, current folder) every 250 ms; Cancel sends SIGTERM. The page: picker → Scan/Rescan → Folders (a tree largest-first: size, share of parent as a bar, files, folders, modified; expand in place) over a treemap of the selected folder — squarified, nested to the file, tiles coloured by kind (video, images, audio, archives & disk images, documents, code & config, databases & logs, other; the validated categorical set, dark and light) with folders as labelled boxes; hover for path, size and share; click a folder to drill, breadcrumbs and ↑ to come back. The map asks the server for the folder pruned to what a tile could show (min = 1/6000 of the folder, 24 deep; deeper folders come back as one lump), the list one level at a time. Largest files: the 1,000 biggest with a path filter. Works on every machine through the hub. Smarter and faster (2026-09-23, at Mitchell's request): a scan is kept — in memory and in <data>/space/<target>.scan (the same varint stream) + .json, read back after a restart — and the next scan starts from it: the helper is handed the last tree on stdin and reads a directory again only when something says it changed. On a ZFS pool with snapshots, the scan records the newest snapshot of each dataset at start (bases), and the next one runs zfs diff -H -F <base> per dataset: only the directories it names are read, their ancestors kept but looked into, everything else kept wholesale without a stat. The diffs run side by side and one is given three minutes — timeout -s KILL inside sudo, since Debian's sudo gives the command its own session and a process-group kill from outside never reached zfs — after which that dataset is walked by mtime instead. On tank a diff is metadata-bound like the walk: 8 s to 3 min per dataset for a handful of changes, and immich, which changes constantly, ran past twenty minutes the first time. A refresh of tank took 4 min 19 s against 7 min for the full scan. Elsewhere (ext4, or a dataset with no snapshot) a directory whose mtime — to the nanosecond — is unchanged keeps its list and only its kept files are stat'ed again, since a file can grow without its directory noticing. A cancelled scan keeps what it walked, marks the folders it never got to (partial, "not yet read" in the tree) and Resume reads just those. core: full 1.2 s → refresh 0.3 s (2 folders read, 84,755 kept). The status line says which (method, by_root, read/kept counts); Forget drops the saved result so the next scan is full. The old result stays browsable while a refresh runs. Treemap rethought the same day: every file a tile grouped by folder — folders are only a thin line and a name strip on the first level — instead of nested boxes with headers, which on Linux's single-child folder chains was a staircase of headers around nothing. Not done: delete from here, a size-over-time history, per-user column choices.

• Updater — shipped 2026-09-23 (internal/selfupdate, internal/api/update.go, the Updates card in Settings.tsx, scripts/gitea-release.sh). A Ham Lab asks one URL what the newest build is and installs it on request. update.url in config.json is either a Gitea/Forgejo repo API base (…/api/v1/repos/<owner>/<repo>, whose newest release names the version and carries the binaries as assets) or any static directory holding VERSION, SHA256SUMS and hamlab-linux-<arch> — which is exactly what make release writes and what install.sh already downloads, so the demo site, a folder on a NAS and a Gitea release are all the same thing to it. "off" never looks. update.token is sent when the source is behind a login, so a private Gitea serves its own fleet without the instance being opened to the world. The binary is verified against SHA256SUMS before it goes anywhere near the running one, then takes the same path POST /api/self-update always did (runs version, runs check against this machine's config, atomic replace keeping hamlab.prev, re-exec in place) — that path is now one shared method, installBinary. Version ordering is install.sh's vnum() rule ported to Go and tested against it, so the daemon and the installer never disagree about which of two builds is newer; the commit count since the tag counts, the hash does not. Checked ~90 s after boot and daily (update.check: "never" switches that off), never installed unasked — an update restarts the daemon and drops open terminals, and the dialog says so. Settings → Updates: installed, published, when it last looked, the source, the release notes, Check for updates, Install; the rail badges Settings when a newer build is out (update_available rides on /api/me). Publishing is make gitea-release → scripts/gitea-release.sh, which creates the release and uploads the five architectures, SHA256SUMS, install.sh and VERSION; it is deliberately not part of make deploy, because deploy pushes this build to the machines this hub owns while publishing offers it to every install anywhere, and one tag per deploy would be noise. Re-publishing a version needs --force. Public since 2026-09-23, at Mitchell's instruction and after the trade was put to him twice: Gitea's instance-wide REQUIRE_SIGNIN_VIEW is off and mitchell/hamlab is marked public, so anyone can browse it, clone it and download a release with no account. The other five repos there are each marked private and are invisible — an anonymous request for one is a 404, not a login prompt — registration is still disabled, and new repos still default to private. No token is needed by anything any more (the read-only one was revoked; only hamlab-publish, which creates releases, remains). Gitea resolves releases/download/latest/<asset>, so install.sh and HAMLAB_URL now point straight at the newest release and never need changing. What is public is the source, the history, BUGS.md and this file — which describes the lab and names the console's hostname.

• Auth hardening — WebAuthn passkeys (phone Face ID), TOTP fallback, optional NPM access list restricting to WireGuard/LAN.

• Files tab — shipped 2026-09-18 (internal/files, Files.tsx): browse from / with breadcrumbs, an editable path box, sort, hidden-file toggle and a filter; rows show type icon, size, modified, perms and owner, link targets and broken links. Text files (≤ 2 MiB, sniffed) open in an editor with Ctrl+S / Save; images preview; everything else downloads. Upload via drag-and-drop or picker with per-file progress (streamed, no size limit). New folder / file, rename, move, chmod, copy path, delete (folders need the name typed back), and "open in terminal", which cds the active shell (or a fresh one) to the folder. Sudo toggle: root access re-executes the same binary as hamlab fs <op> under sudo -n, streaming bodies over stdin/stdout, so both modes share one implementation and one set of rules (absolute paths only; /, /etc, /usr, … are never deleted or moved whole). Every write is logged with the caller's IP.

• Docker read-only view — docker ps/logs; dockhand stays the source of truth.

• Notifications — push on service failure / SMART warning / UPS event.

• Terminal extras — search, copy-mode, font/theme, split panes, upload via drag-drop into cwd.

• Settings + themes — shipped 2026-09-18 (Settings.tsx, themes.ts): Settings page with Appearance (Dark, Light, Cyberpunk, Retro — CSS variables on <html> plus a matching xterm palette, applied before first paint from localStorage and synced hub-side via GET/PUT /api/prefs/ui so the choice follows the login), Account (change password, POST /api/password, current password required), Access tokens (list/revoke the tokens hubs hold), About (version, host, arch, uptime, certificate).

To do

• Deprecate copyparty — asked for 2026-09-23, not started. copyparty on grid (copyparty.service, :3923, /etc/copyparty.conf, serving all of /tank as user core with one account behind NPM) overlaps the Files tab almost entirely. The aim is to stop the service and keep only Ham Lab. What Files must gain first, in the order that matters:

  • Multi-select and bulk actions. copyparty selects many rows and moves, deletes or downloads them together; Files acts on one row at a time.
  • Download a folder, and a multi-file download, as a streamed zip or tar — the thing copyparty is reached for most.
  • Resumable uploads. copyparty's up2k chunks a file, survives a dropped connection and skips what is already there; Files streams one PUT, so a phone upload that drops at 90% starts over.
  • A gallery. Thumbnails for a folder of images and video, and play audio/video in place — /tank/media and /tank/immich are most of what is browsed.
  • Search below the current folder by name, since find in a terminal is the fallback today.
  • Decide when built: whether share links (copyparty's public per-path links) are wanted at all — they are the one feature that hands out access without a login, and Ham Lab has no notion of that; if not, say so in the docs rather than leaving a gap unexplained.
  • After: systemctl disable --now copyparty, remove its NPM host, keep /etc/copyparty.conf until a rollback is clearly not needed, then update hamlet's docs/hosts.md bare-metal list, docs/gotchas.md (the "docker ps is not the inventory" note names it) and the grid row in CLAUDE.md.

• Deprecate Dockhand — asked for 2026-09-23, not started; the homelab half of this (migrating every stack off Dockhand's data dir, then removing the container) is hamlet's docs/pending.md → "Retire Dockhand; Ham Lab becomes the orchestrator", agreed 2026-09-20. This entry is the app half only: what Ham Lab must do before that migration can start. Its Stacks page already creates, edits, deploys, updates, restarts, stops, downs and deletes stacks on every machine, so the gaps are Dockhand's extras:

  • Git-sourced stacks. Dockhand's stack_sources table points a stack at a repo and pulls it. Ham Lab has no equivalent. Decide whether to build it or to drop it — most stacks here have no row in that table anyway, so dropping it may be honest.
  • Scheduled image updates. Ham Lab checks for new images only when asked (a registry request Docker Hub counts) and never updates by itself. If anything relies on Dockhand updating on a schedule, that must move here or be dropped on purpose.
  • Webhooks / per-host environments, if either is in use — inventory before deciding.
  • Deploying from a host path without an orchestrator container. Ham Lab already runs compose inside dockhand because that is how the stacks were deployed; once they move to a neutral home the "via" path is unused for them, but a stack deployed to grid or the Mac still goes over tcp://<node>:2375 from the hub. Decide whether Ham Lab should instead ask the node's own Ham Lab to deploy locally, which would end the *:2375 exposure (its own item in docs/pending.md).
  • Not app work, but the trigger: nothing here blocks starting the migration except the inventory. Do that first.

• A Virtualization tab — asked for 2026-09-23, not started, and large. Run and manage LXC containers and virtual machines from the console, the way Proxmox does — which is the comparison Ham Lab has invited since its first line. Nothing exists yet. What a future session has to settle before writing code: which stack (libvirt/KVM + virsh is the obvious one for VMs; LXC or Incus for containers, and Incus covers both); whether it manages what is already installed or installs it; where disk images live (tank on grid, local NVMe elsewhere); console access (SPICE/VNC proxied through the daemon, or a serial console over the existing terminal machinery, which would reuse internal/term); snapshots and clones; and what it does on a host with no virtualisation at all, which is every host here today. Worth scoping as its own plan before starting.

• Files explorer: cleanup and ease of use — asked for 2026-09-23, not started. General polish and ease of use. The Deprecate copyparty item names the heavier pieces (multi-select and bulk actions, folder download, resumable uploads, a gallery, search) — worth doing the two together, since they are the same tab.

• A KVM tab — asked for 2026-09-24, not started. Dead simple is the brief: paste one HTTP link, the tab shows whatever UI that KVM serves, and the keyboard goes to it. This is KVM the keyboard-video-mouse switch, not KVM the hypervisor — the Virtualization item above means the other one.

  • The device: a Sipeed NanoKVM attached to grid at 10.0.0.217 (MAC 48:da:35:6f:d0:31), web UI on port 80. The only one so far, so store a list, not one URL.
  • Framing works. Probed 2026-09-24: it sends no X-Frame-Options and no frame-ancestors, so it does not refuse to be embedded. No fallback needed.
  • Proxy it through the daemon — that is what makes the simple version possible. The KVM is HTTP only, and Ham Lab is served over TLS, so a bare http://… iframe is blocked as mixed content and the "just paste a link" idea dies there. Serving it from Ham Lab's own origin (/api/kvm/<id>/…, on the proxyNode machinery, which already carries WebSockets — exactly what the video and input streams are) fixes three things at once: no mixed content, the KVM sits behind Ham Lab's login rather than a second one (/api/vm/info answers 401, so it has its own, which the hub can hold once), and the iframe becomes same-origin, which matters for the keyboard below.
  • The keyboard needs less than it sounds. A parent page cannot capture keys and forward them into a cross-origin iframe — the browser forbids synthetic events across origins — but it does not need to: an iframe with focus receives keystrokes directly, and the NanoKVM's own page already handles them. So "capture and give it to the iframe" is really: click-to-focus, keep focus there, and stop the console's own shortcuts from eating keys while it is focused. What is genuinely worth adding is fullscreen plus navigator.keyboard.lock(), which is what stops the browser swallowing Escape, Tab and Ctrl-W so they reach the guest — note it is Chromium-only, so Firefox and iOS Safari get the plain version. Proxying makes the iframe same-origin anyway, so real forwarding stays available if some KVM ever needs it.
  • Where the list lives is the one real design decision. A KVM earns its keep when the machine is down — no OS, a bad kernel, a BIOS screen — which is when that host's Ham Lab cannot answer. So it is hub-side and must work for machines running no Ham Lab at all. A kvm_url in each node's own config.json would fail in exactly the case it is for.
  • Do not give it an NPM host without deciding to: *.bigham.dev is a wildcard to the public IP, so that would put BIOS-level control of grid on the internet behind only the KVM's own login. Behind Ham Lab's proxy it inherits Ham Lab's auth instead.

Open questions

• Idle-session reaping policy (never / after N days) — leaning never; it's a single-user box.
• Whether sudo -i-style root sessions get their own leaf colour. Cosmetic.