# Ham Lab — MVP plan

Proxmox-style web console for a bare-metal Debian 13 host. First target: `core`
(10.0.0.234). MVP is **service + shell + persistent terminals**; everything else is a
stub tab or a roadmap item.

## Decisions

| Area | Decision | Why |
|---|---|---|
| Backend | **Go 1.24**, single static binary, frontend embedded via `embed` | `core` has ~370 MiB free RAM; a Go daemon idles at ~15 MiB vs ~80+ for Node. One file to deploy, no nvm path in the unit. Same binary later installs on `grid` unchanged. |
| PTY | `creack/pty` — server-owned PTYs running `core`'s login shell | Sessions live in the daemon, not the browser, so refresh/tab-switch reattaches. No tmux dependency (not installed on `core`). |
| Transport | One WebSocket per attached terminal; REST for everything else | Standard; NPM passes WebSockets with one toggle. |
| Frontend | **React + TypeScript + Vite**, `xterm.js` (+ fit, webgl, web-links addons) | Matches `calorie-counter` / `wedding-website`, so no new toolchain to learn. |
| Auth | Single-user **password login**, argon2id hash, HttpOnly/Secure/SameSite=Strict session cookie, login rate-limit, Origin check on WS upgrade | This is a root-capable shell on the public internet. Auth is MVP, not roadmap. |
| Service | `hamlab.service`, `User=core`, `Restart=always`, binary at `/usr/local/bin/hamlab` | Same pattern as `calorie-counter.service`. Runs as `core` so `sudo` works in the terminal without a password, as on the SSH path. |
| Listen | `10.0.0.234:8100` (free) | NPM is a container, so loopback-only won't do. Firewalling to NPM is a later hardening step. |
| Data | `/home/core/apps/hamlab/` — `config.yaml`, `state.db` (SQLite) | `apps/` is the nightly rsync target to `grid`, so it's backed up for free. |
| Ingress | NPM proxy host `lab.bigham.dev` → `:8100`, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a **second** API call (NPM drops them on create-with-new-cert) | `*.bigham.dev` already resolves; the proxy host alone makes it live. |
| Repo | Gitea `mitchell/hamlab`, private, `main` | Created 2026-09-17. |
| Guest logins | A **guest** is an account like any other, with `guest: true` and an expiry: `login`, sessions and prefs need no special case, and **it may do everything an account may do**. The expiry is the whole feature — checked on every request and swept once a minute — with revoking as the manual version of it. | Built 2026-09-27. A read-only version was built first and **cut the same day**: Mitchell hands these to people he trusts and revokes them otherwise, so a console with half its tabs missing was solving a problem he doesn't have. Consequences, stated once and accepted: a guest has the Terminal tab, which is a root-capable shell on every host, and can make an account or token of its own that outlives the guest login. The one thing a guest is not is *an account*: it is left out of the Users list, out of "the last account can't be removed", and out of the single account an empty user name resolves to. |
| Machines | **A view shows one machine: the one in the switcher.** No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it. | Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still *proxies* for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected. |
| Capability, not policy | **A tab offers what the selected machine can do, detected live — never what a rule says should run there.** Where the software is missing but installable, Ham Lab installs it; where it cannot be installed or the hardware lacks it, the tab says so in a sentence and offers what does work. Every action lands on the machine in the switcher. | Mitchell, 2026-10-09, after the Virtualization plan's first draft excluded the Mac mini and `hamtv` on the strength of `hamlet/CLAUDE.md`'s deployment rules: *"I want the option to do it anywhere I want, unless the hardware / software literally doesn't support it. Ham Lab needs to be built in a general-purpose way that can run on any Linux system."* Those rules bind the agent's unasked deployments, not the product. |
| Ham Lab's data lives in Ham Lab's folder | **Everything a tab manages sits under the machine's data directory (`config.DataDir()`), never at a convenient path elsewhere.** Existing material is imported by copying. The engine's runtime state (`/var/lib/docker`, `/var/lib/incus`) is the engine's; Ham Lab keeps what would rebuild it. | Mitchell, 2026-10-09, when the Virtualization plan pointed its ISO library at `/tank/software/isos/`: *"ALL data needs to be under a HamLab folder, because it manages its own data."* The same reason `core`'s compose files moved under `apps/hamlab/stacks/` on 2026-10-05. `grid`'s and the Mac's stacks at `~/stacks/<machine>/` were the standing exception; since 2026-10-10 the stacks code offers `<data dir>/stacks` as the default home everywhere and such folders only as discovered extras. |
| No virtualization | **Ham Lab does not manage VMs or system containers** — no Incus, no QEMU, no passthrough, no VM consoles. It is a lightweight console; Docker is its one container engine. | Mitchell, 2026-10-15: *"Virtualization is out of scope for HamLab. This is supposed to be a lightweight application, and VMs and pass through and stuff is too much."* The tab built 2026-10-09 (Incus, phases 1–2, deployed as v0.1.0-196) was cut from `main` the same day and kept on the **`virtualization` branch** (its last commit is `main` as of `a2674be`, with `VIRTUALIZATION.md` and the vendored spice-html5). Don't rebuild it on `main`. |

## Layout

```
┌──────────────────────────────────────────────────────────────┐
│ header: Ham Lab · core · load 0.42 · ● connected · [logout]    │
├──────────────┬───────────────────────────────────────────────┤
│ Dashboard    │ shell 1 ● │ backups │ +                      │
│ Processes    ├───────────────────────────────────────────────┤
│ Terminal     │                                               │
│ Services     │        main content (xterm)                   │
│ Cron         │                                               │
│ …            │                                               │
├──────────────┴───────────────────────────────────────────────┤
│ footer: v0.1.0 · up 12d · lab.bigham.dev                   │
└──────────────────────────────────────────────────────────────┘
```

- Left rail: flat list of pages, active one highlighted. No nested leaves.
- Terminal page: tab strip across the top of the pane, one tab per session, `● ` when
  a process is running, `+` pinned at the right edge. Active tab carries rename and
  close; double-click renames, middle-click closes. (Until 2026-09-18 the sessions
  were leaves under a collapsible Terminal group in the rail.)
- Non-MVP groups render as disabled leaves with "coming soon" — the rail shape is
  the product; stubs keep it honest about scope.
- **Mobile**: rail becomes a slide-in drawer; xterm fits the viewport; a sticky key
  bar above the keyboard (`Esc Tab Ctrl Alt ↑ ↓ ← → |`) because phone keyboards
  have none of those. This is what makes it usable as a PWA rather than a novelty.

## Terminal semantics

- A session is a PTY + shell owned by the daemon, with an ID, name, cwd hint, and a
  **scrollback ring buffer** (256 KiB). Attaching replays the buffer, then streams.
- Any number of browser tabs can attach to the same session (all get output; all can
  type). Resize follows the most recent attacher.
- Switching tabs in the UI just detaches/attaches; the shell keeps running.
- Closing a session sends `SIGHUP` to the process group and drops it. Shell exit
  closes the session automatically (leaf shows "exited" until dismissed).
- Session names persist in `state.db` so the sidebar order survives a service
  restart. **The shells themselves do not survive a daemon restart** in the MVP —
  see roadmap (tmux backing).
- Idle sessions cost one shell process each; no automatic reaping in MVP.

## API (MVP)

```
POST /api/login            {password}            → 204 + cookie
POST /api/logout                                 → 204
GET  /api/me                                     → {host, user, version, uptime, load}
GET  /api/terminals                              → [{id, name, running, created}]
POST /api/terminals        {name?}               → {id, ...}
PATCH /api/terminals/:id   {name}                → 204
DELETE /api/terminals/:id                        → 204
WS   /api/terminals/:id/attach   binary frames = raw bytes; JSON text frames = {resize}
GET  /healthz                                    → 200 (unauthenticated, for NPM)
```

Everything under `/api` except `login` and `/healthz` requires the session cookie.

## Project layout

```
hamlab/
  cmd/hamlab/main.go          serve | passwd | version
  internal/
    auth/                     argon2id, sessions, rate limit
    term/                     PTY manager, ring buffer, attach/detach
    api/                      chi router, handlers, WS
    state/                    SQLite (modernc, no cgo)
    web/                      go:embed of web/dist
  web/                        Vite + React + TS + xterm.js, PWA manifest + SW
  deploy/
    hamlab.service
    install.sh                build → /usr/local/bin/hamlab → systemctl restart
  Makefile                    build / dev / deploy / lint / test
```

## Milestones

**M0 — skeleton** (service boots, login works)
- Go module, chi, config loading, `hamlab passwd`, argon2id login, session cookie.
- `hamlab.service` on `core`, `:8100`, serves a placeholder page.
- `make deploy` from `core` itself.

**M1 — terminals** (the MVP feature)
- PTY manager with ring buffer, create/list/rename/close, WS attach with resize.
- React shell: header, rail, footer, Terminal group, xterm main pane.
- Persistence of session list across refresh and rail switching.

**M2 — polish + PWA + ingress**
- Mobile drawer, key bar, fit-on-rotate.
- `manifest.webmanifest`, icons (Ham logo on black, same recipe as HamTV), service
  worker caching the app shell only (never `/api`).
- NPM proxy host for `lab.bigham.dev`; verify WS through NPM; force-SSL/HSTS/HTTP2.
- Stub groups for Services / Cron / ZFS / Dashboard.

**Done when:** from the phone, open the PWA, log in, open two terminals, run
something long in one, switch away and back, refresh — it's all still there.

## Roadmap (not MVP — notes only)

- **Updates tab** — **shipped 2026-09-17** (`internal/updates`, `internal/tasks`,
  `Updates.tsx`): upgradable packages with current → candidate, suite, security badge
  and description; index age, last upgrade (from `/var/log/apt/history.log`),
  unattended-upgrades on/off, reboot-required banner with the packages that caused it;
  Refresh index, Upgrade all (dist-upgrade), Upgrade selected; changelog per package on
  demand. Runs go through the new **tasks** package: background command, output kept,
  followed live over `/api/tasks/{id}/stream`, cancellable, listed at `/api/tasks` (the
  seed of a Task History tab). apt is fully supported; dnf/pacman list only.
- **Power** — header ⏻ button (amber when a reboot is pending), dialog that requires the
  hostname typed back, then `systemctl reboot` / `poweroff`. `/api/me` carries
  `reboot_needed` so the header knows.

- **tmux-backed sessions** so shells survive a daemon restart / redeploy. Would spawn
  `tmux new -A -s hamlab-<id>`; needs `mouse on` and status-bar handling in xterm.
- **Services tab** — `systemctl list-units`, per-unit page: status, last 200 journal
  lines (live-tail via WS), start/stop/restart/enable buttons, unit file viewer.
- **Cron tab** — **shipped 2026-09-17** (`internal/cron`, `Cron.tsx`): one list built
  from `/etc/crontab`, `/etc/cron.d/*`, every user crontab (via sudo), the scripts in
  the `run-parts` directories those drive, plus systemd timers. Next run computed by an
  in-house cron-expression evaluator (tested: ranges, steps, names, dom-OR-dow rule,
  `@` aliases); last run from the journal's `(user) CMD (command)` lines (cron logs via
  syslog to journald; reading needs sudo/adm). Commented-out jobs show as disabled;
  job-shaped lines inside comment blocks (Debian's header examples) are ignored. Per
  job: Run now (background, as the job's user, `sh -c` with crontab `%` rules, output
  captured), Enable/Disable (comments the line in place, refuses if the file changed
  underneath), Edit source (textarea; validated line-by-line; user crontabs installed
  with `crontab -`, system files written via `sudo tee`), Cron log, and Output file
  when the command redirects to one. Timers: Run now (starts the activated service),
  start/stop/enable/disable, live logs of the service via the Services tab's tail.
- **Disks tab** — **shipped 2026-09-17** (`internal/storage`, `Disks.tsx`), replacing
  the planned ZFS tab: block devices from `lsblk -J` (kind, model, size, temperature
  from the device's hwmon node, partitions with filesystem usage, live read/write/busy
  from the metrics sampler); SMART via `smartctl -j` when smartmontools is present (one
  button installs it as a task otherwise): health, temperature, power-on time, self-test
  status, ATA attribute table with the failure-predicting IDs flagged, NVMe endurance and
  error counters, short/long self-tests. Spinning disks are read with `-n standby` so
  the tab never wakes a sleeping drive; "Wake and read" is explicit. ZFS section when
  the tools and module are there: pools with health, capacity, fragmentation, last scan
  and vdev tree; scrub start/pause/stop; datasets tree with used/refer/in-snapshots/
  snapshot count/compressratio/mount; per-dataset snapshot list and "Snapshot now".
  Verified against `grid` (raidz2 `tank`, 10 datasets, 515 snapshots, 8 WD40EZAZ + 970
  EVO Plus SMART). Snapshot destroy and dataset property editing are deliberately not
  there yet.
- **Dashboard** — **first cut shipped 2026-09-17** (`internal/metrics`, `Dashboard.tsx`):
  CPU (total + per core), memory/swap, load, uptime, hwmon temps and fans read straight
  from `/sys` (no lm-sensors), per-interface network rates, per-disk read/write/busy
  from `/proc/diskstats`, filesystem usage, UPS via `sudo -n pwrstat -status`. Sampled
  every 2 s into an in-memory ring of 10 min; the UI (the **Dashboard** tab
  since 2026-09-19) reads it all through the sensor API — see Sensors below. **Customizable**: a
  Customize mode reorders and hides modules (CPU, Memory, Load, Uptime, Temperatures,
  Fans, Network, Disks, Filesystems, UPS) and hides individual rows (an interface, a
  mount, a disk, a sensor); the layout is stored server-side in `state.json` via
  `GET/PUT /api/prefs/dashboard` so every device sees the same dashboard. The update
  interval (100 ms – 2 s) is part of the layout and drives both the browser's polling and
  the daemon's sampling clock; the chart window is always 300 samples, so 30 s at 100 ms
  and 10 min at 2 s. Still to do: persistent
  history (SQLite ring, e.g. 10 s × 7 days), SMART (`smartctl -j`), longer time ranges.
- **Updating from the Updates tab** — **shipped 2026-09-26**: the "what is out of date" list
  was read-only, so seeing a newer image and doing something about it were different places.
  Each row now has **Update**, and the header has **Update all**. Updating is per stack —
  that is the unit compose deals in — so a row acts on whichever stacks use that image, and
  several stacks run through a queue one at a time rather than recreating half the machine at
  once, each with its task output on screen. A row that succeeds says "updated just now"
  instead of re-checking, since a check is a registry request.
- **Recreate, and the Updates tab at full width** (2026-09-26): a stack's actions gained
  **Recreate** (`up -d --force-recreate`), the only way out of a container Docker refuses to
  start — one pinned to a network that has since been recreated answers "network … not
  found" to Deploy and Update alike, because as far as compose is concerned nothing about it
  changed. Found the hard way when a SiYuan update landed in exactly that state. The Updates
  tab also stopped rendering as a list-and-detail pair: it is a page, not a list, so on a
  wide window it now sits under the tab strip at full width instead of being squeezed into
  the right-hand half beside an empty column.
- **Cron in English** (`crontext.ts`, 2026-09-26): schedules read "daily at 3:15am" rather
  than `15 3 * * *`, with the expression kept beside it. Handles lists, ranges, steps, day
  and month names, `@daily` and friends, weekdays/weekends, midnight and noon — and declines
  to describe anything it is not sure of, because a confident wrong sentence is worse than
  five fields. The Jobs tab had its own smaller version of this; it now uses this one.

- **Hardware tab** — **shipped 2026-09-25** (`internal/hardware`, `internal/api/hardware.go`,
  `Hardware.tsx`): what the machine is physically made of, per machine like every other page.
  Four headline tiles (processor, memory, storage, graphics) over cards for each: the
  processor with its cores, clock range, socket, caches, microcode and the handful of feature
  flags anyone asks about; **every memory slot** with size, type, form factor, rated and
  running speed, maker, part number, serial, rank and voltage, and the empty ones marked
  empty; each drive with model, size, bus, firmware and serial; each display adapter with its
  driver, PCI address and video memory; and the board itself — model, firmware and date,
  serial, kernel. Read from what Debian already has (dmidecode for what only the firmware
  knows, lscpu, lspci, sysfs) and cached for ten minutes, since none of it changes under a
  running kernel; "Re-read" is there for the one thing that does, a drive plugged in.
  Every reader may come back empty and the page says what it could not read and why: a
  machine with no dmidecode, no root, or no PCI bus still gets a page. Verified on `core`
  (SMBIOS, Intel graphics) and on the Mac mini (Asahi: no SMBIOS memory table, a
  device-tree GPU, heterogeneous cores — where lscpu's "cores per socket" counts one
  cluster, so without SMT the thread count is the honest core count).

- **Sensors** — **shipped 2026-09-19** (`internal/sensors`, `internal/metrics/sensors.go`,
  `internal/api/status_sensors.go`, `web/src/sensors.ts`, `widgets.tsx`, `Embed.tsx`;
  reference in `SENSORS.md`): everything measured is a sensor with a stable dotted ID;
  catalog, single value, history, a **shaped query** (the client sends the object it wants
  and gets it back filled in — GraphQL's idea without GraphQL, since sensors differ per
  machine and a static schema would collapse to `sensor(id:)`), the same query as a live
  SSE stream, Prometheus `/metrics`, read-only `sensors:read` tokens (never full tokens in
  a URL), status sensors (services, updates, ZFS, SMART), a widget library, custom widgets
  on the Dashboard tab (briefly renamed "Monitors" 2026-09-19, back to "Dashboard"
  2026-09-20 at Mitchell's request; the view id and URL were always `dashboard`), and a
  frameable `/embed/widget` for Glance and friends. (An external demo page was deployed at
  `hamlab-demo.bigham.dev` and linked from the Widgets tab 2026-09-19; **both removed
  2026-09-23** at Mitchell's request — the Widgets tab shows the same thing against the
  machine you are signed in to, so the second copy earned nothing and cost a public
  read-only token, which was revoked with it. `examples/external-demo/` stays as an example
  to read.) The stock Dashboard modules and the Disks
  tab read the sensor API too (`sampleAdapter.ts`); `/api/metrics` is retired.
  Also per-service and per-container sensors (memory and CPU from cgroups), tokens limited
  to ID patterns, and `/api/fleet/sensors/query` for the whole fleet in one request.
- **Alerts** — **shipped 2026-09-19** (`internal/api/alerts.go`, `Alerts.tsx`): rules over
  sensors (`container.*.running == false for 120 s`), evaluated by the hub every 15 s for
  itself and every managed machine in one shaped query each; `hamlab.reachable` is supplied
  by the hub so "a machine stopped answering" is a rule like any other. Ten defaults
  (machine offline, container down/unhealthy, failed service, disk full, pool unhealthy,
  SMART failing, UPS on battery, CPU hot, memory). Firing and resolved are events: kept
  (last 200, in `state.json`), shown on the hub-only Alerts tab with a count on the rail,
  and POSTed to channels — **ntfy** (title/priority headers) or a **webhook** (JSON; Home
  Assistant, Slack). The rule editor shows what a selector matches across the fleet as
  you type. Not done: quiet hours, per-rule channels, Web Push straight to the PWA.
- **Docker tab** — **shipped 2026-09-19** (`internal/api/docker.go`, `Docker.tsx`):
  containers grouped by compose stack with live memory and CPU from the container
  sensors, start / stop / restart (asked first), logs that follow (polled with `--since`,
  no held connection), inspect (env *names* only), and images / volumes / networks
  tables. Works on every machine through the hub; `#/grid/docker/immich_server` links to
  one. **Stacks** (`internal/api/stacks.go`): every compose project, found from its
  containers' labels, with its file to edit and Deploy (`up -d`), Update (pull, then
  deploy), Restart and Stop as followed tasks. The file is looked for where the label says
  *and through every container's mounts*, because an orchestrator that runs compose inside
  its own container (Dockhand here; Dockge, Portainer) records a path in its own
  filesystem; compose is then run **inside that container**, so `.env`, relative paths
  and the compose labels stay exactly as deployed (verified: labels identical after a
  redeploy). Save validates with `compose config` first, keeps the previous file as
  `<file>.hamlab-bak`, and never deploys by itself. A stack whose file is not on the
  machine that runs it is listed read-only, and that is the end of it: **since 2026-10-10
  the hub holds no files for a node and never points compose at another machine's
  Docker.** The `hubStacks` path — which resolved a node's files on the hub and deployed
  over `tcp://…:2375`, the way Dockhand had — is gone, and every
  `/api/nodes/<node>/docker/…` request is the node's own Ham Lab's to answer through the
  generic proxy. Also per stack: every file it was deployed from (chips), its
  **`.env`** (values masked until "Reveal and edit"; validated in place, put back if
  compose objects), **what Deploy would change** (a line diff against the copy kept at the
  last deploy from here, `<file>.hamlab-deployed`, else the copy the last save replaced;
  `.env` values never shown), and **image update checks** — asked for, never polled, since
  each is a registry request Docker Hub counts; remembered six hours. A save refuses a
  file changed on disk meanwhile.
  **Making and unmaking (2026-09-20):** "+ New stack" — a name, a folder, compose text and
  an optional `.env` (written root-only), checked by compose before it is kept, then
  "Create and deploy" or "Create only". "+ New container" is the same dialog opened on a
  form (image, ports, volumes, environment, restart, command, host networking) that writes
  the compose for you: a container made here is a one-service stack, so it stays editable
  and redeployable, where a bare `docker run` could only be thrown away and typed again.
  New stacks go in a **home**. Since 2026-10-10 the default is Ham Lab's own
  `<data dir>/stacks`, offered first on every machine; a folder where two or more stacks
  are already deployed from is offered as well, and a stack made there is deployed the way
  its neighbours were (inside an orchestrator's container, when that is how they run). Homes
  are also scanned, so a stack with no containers ("not deployed": never deployed, or taken
  down) is still listed. **Down** removes a stack's containers; **Delete…** (type its name)
  takes it down and moves its own files to `<home>/.hamlab-trash/<name>-<time>/` — moved,
  not erased; volumes and bind-mounted data are never touched. A stopped container can be
  **removed** (a running one must be stopped first, on purpose). Images: **pull** (a followed
  task), **remove** (unforced: Docker refuses one still in use) and **prune** (dangling only).
  **Volumes (2026-09-20):** each says which containers use it; one that nothing uses can be
  deleted — a named one by typing its name (it may be the data of a stack that is only
  down), an anonymous one by confirming its full name — and "Prune anonymous" clears the
  nameless leftovers of recreated containers, never named volumes. Images are marked "in
  use", and "Remove unused" takes every image no container uses. Nothing is ever forced:
  Docker refuses what is still in use, and that refusal is the answer. Still absent:
  removing networks.
- **Top merged into Processes** — **2026-09-23.** The two tabs answered the same question
  from the same data (Top already read `/api/procs`, the endpoint the Processes table was
  built on), so at Mitchell's instruction there is now one tab, **Processes**, showing what
  Top showed: the btop-style whole-machine screen. `Procs.tsx` is deleted; the `top` view
  id and its route are gone. A rail order saved before this keeps its shape — `railorder.ts`
  maps the retired `top` to `procs` and drops the duplicate, so the merged page sits where
  Top sat rather than falling to the bottom.
- **GPUs on the Processes tab** — **shipped 2026-10-02** (`internal/metrics/gpu.go`, the
  `GPUBox` in `Top.tsx`, `gpu.*` sensors). Asked for as "GPU activity and temps, like btop
  has it, always on". Each DRM device with a real GPU driver gets a box under the cpu box:
  busy curve, busy meter, a meter per engine (render/video/copy/enhance), temperature, clock,
  power and fan where the card has them, and the processes that have it open with their
  share. The process table gains a `Gpu%` column and sort. **How it is measured, and the
  honest gaps:** device busy is Intel's RC6 idle residency inverted (the counter every GT
  exposes without root; `intel_gpu_top` shows the same number) or amdgpu's own percent;
  per-process and per-engine shares are DRM fdinfo deltas, the way nvtop reads them. Other
  users' fdinfo needs root, so a daemon running as `core` keeps **one** root child
  (`hamlab drm-clients`, over a pipe) rather than a sudo a second. **NVIDIA's proprietary
  driver (2026-10-23)** has neither a sysfs busy counter nor fdinfo engine totals, and its DRM
  card reports `DRIVER=nvidia` (not `nvidia-drm`), so until then such a card was not even
  listed; now it is read through `nvidia-smi` — `--query-gpu` for busy/encoder/decoder, VRAM,
  SM clock, temperature and power, and `pmon -c 1 -s um` for the per-process table (sm/enc/dec
  shares, framebuffer MiB) — two ~20–75 ms calls per sample, selected by PCI address, no root.
  First machine: `link`, the Surface Studio's GTX 965M. The Arc A380 on i915 6.12
  exposes power, energy and fan but **no temperature**, so its box shows none; an integrated
  GPU shows the CPU package temperature marked `*`. Asahi (the Mac mini) has neither a busy
  counter nor fdinfo in 6.19, so it lists the device and says so. No NVIDIA path: nothing here
  has one to test against. **Names come from an embedded PCI ID table** (2026-10-02,
  `internal/metrics/pciids_gpu.txt`, regenerated by `scripts/pci-ids-gpu.sh` from the hub's
  `pci.ids`): `gate` has no `lspci` and showed "Intel GPU 3185" beside `hamtv`'s "Intel UHD
  Graphics 600" for the identical chip. The table is tried first everywhere, so the name — and
  the sensor ID built from it — does not depend on which distro's pciutils a machine has;
  `lspci` is the fallback for a card newer than the table. Two fixes the same evening:
  the gpu box's layout rules had been lost from the stylesheet (v0.1.0-166/167), and on
  `grid` the sampler hung for 42 minutes in Go's poller on the Arc's `fan1_input` —
  `readStr` now uses plain blocking syscalls and each GPU's sysfs reads run under a
  3-second deadline (v0.1.0-168; `hamlet/docs/gotchas.md` has the mechanism).
- **CPU power on the Processes tab** — **shipped 2026-10-10** (v0.1.0-194,
  `internal/metrics/cpupower.go`, the `cpu.power` sensor). Asked for beside the clock speed,
  and that is where it sits: "3600 MHz · 89 W" in the cpu box's header. Watts are the RAPL
  package energy counters' delta between samples, summed over packages; `psys`, a package's
  parts and the MMIO duplicate are left out. The counters are root-only, so a daemon running
  as `core` keeps a second root child, `hamlab cpu-energy`, beside `drm-clients`. Read on all
  four Intel machines; the Mac mini has no CPU-only counter (its SMC reports the whole
  machine's draw instead) and shows nothing.
- **The ARC on the ZFS tab** — **shipped 2026-10-02** (`internal/metrics/arc.go`,
  `internal/storage/arc.go`, `GET /api/zfs/arc`, the Cache section in `ZFS.tsx`, `zfs.arc.*`
  sensors). Asked for as "current ARC usage, including any types of caching services in the
  pool". One card for the ARC: size against its ceiling with the target and floor, sparklines
  of size and hit rate from the sampler's history, what it holds (MRU/MFU/dirty, data vs
  metadata, compression, the ghost lists), the prefetcher and dbuf cache, memory-pressure
  counters, and the since-boot breakdown by demand/prefetch × data/metadata. Then a card per
  pool naming every caching tier — L2ARC, SLOG, special vdev, dedup table, spares — with its
  devices and fill, **or saying plainly that there is none and what that means** (the ZIL on
  the pool's own vdevs, with how much it has written), plus the properties that steer caching
  (`primarycache`, `secondarycache`, `sync`, `logbias`). Read from `/proc/spl/kstat/zfs/*`
  and `/sys/module/zfs/parameters`, with `zpool list -Hpv` and `zfs get` run without sudo
  (they need none) every 30 s at most. The day it shipped it showed `tank`'s ARC pinned at
  its 2 GiB floor under `arc_no_grow`, with a 53 GiB python3 process holding the RAM.
- **Dashboard retired** — **2026-09-23**, at Mitchell's instruction, the same reasoning that
  merged Top into Processes: the app repeated itself. The customisable module list is gone
  (`Dashboard.tsx` deleted), and with it `/api/prefs/dashboard`, `applyPrefs`, and the
  per-user `dashboard` blob in `state.json`. The Processes tab already shows those numbers
  on one screen. `fmtBytes`/`fmtRate` moved to `fmt.ts` (Disks, Files and Space use them);
  `MONITOR_HISTORY` and `seriesOf` went with the page. The default view after login, after
  switching machines, and when a hub-only page is left behind, is now **Processes**. A saved
  rail order naming `dashboard` degrades quietly — `orderBy` drops ids with no page.
  **Widgets stays and says what it is for**: a notice at the top explains these are for
  dashboards *outside* Ham Lab — Glance, Home Assistant, a wall display — served from
  `/embed/widget` with a read-only sensor token, not a second copy of the console's screens.
  **It also closed a real bug.** `applyPrefs` set the machine's sampling rate from
  `intervalMs` inside the dashboard layout, and its `else` branch reset the sampler to the
  default whenever that field was absent. Since layouts became per-user, one account saving
  a dashboard could silently reset the sampling rate for the whole machine, overriding the
  fleet refresh rate set in Settings. The sampler now has one source: `refreshMs` in the UI
  prefs, which the hub fans out, remembered per machine as `sampler_ms`.
- **ZFS tab, with scheduled snapshots** — **shipped 2026-09-24** (`internal/storage/snapshots.go`,
  `internal/api/zfs.go`, `ZFS.tsx`). Pools, datasets and snapshots moved out of Disks —
  Disks is the hardware, ZFS is the filesystem on top of it — and the tab gained what it was
  split out for: **snapshot schedules**. Hourly, daily, weekly or monthly, per dataset,
  optionally recursive, with a **retention period** and a floor that keeps the newest few
  whatever their age. A missed window (the machine was off) runs once on return rather than
  being skipped.
  **The naming is zfSnap's, deliberately.** `grid` has run zfSnap v1.11.1 from root cron at
  02:30 with a 6-week TTL since before this repo existed, and it works. So rather than
  invent a second convention for two tools to disagree over, this writes and reads exactly
  what zfSnap does — `tank@2026-09-24_02.30.01--6w`, **retention encoded in the name**. Three
  consequences: Ham Lab shows and expires the snapshots zfSnap already took; the two can run
  side by side without fighting, since both only remove what has passed the expiry written
  into its own name; and changing a schedule never re-dates old snapshots, because each
  carries the rule it was taken under. The page **detects the existing cron job**
  (zfSnap, zfs-auto-snapshot, sanoid, or a bare `zfs snapshot`) and says so, warning if a Ham
  Lab schedule is running as well.
  **Nothing is destroyed unseen.** Only names that parse as a stamp and a TTL are ever
  candidates for expiry, so a snapshot taken by hand is invisible to a schedule; Run now
  says that expired snapshots go with it; and a snapshot deleted by hand is confirmed first
  and needs its full name echoed back by the caller. (A "what would it delete?" preview
  shipped and was removed the same day as redundant — Mitchell's call.)
  The datasets table is a **snapshot explorer**: each dataset shows how many snapshots it has
  and what they cost, and opens into the snapshots themselves — when each was taken, **when
  it expires** (read from the TTL in its own name, so no server call), its size, and Delete.
  Deleting takes the same-named snapshot on descendant datasets too, which is how a recursive
  schedule made it.
  **A schedule is a real cron job** (2026-09-24, Mitchell's call after the first cut ran it
  from a ticker inside the daemon). Ham Lab writes `/etc/cron.d/hamlab-zfs`, owns it, and
  rewrites it whole on every change; each line calls `hamlab zfs-snapshot <id>` as the
  account Ham Lab runs as, reaching zfs through sudo as the daemon does. Snapshots are
  backup-shaped and must not stop because a console is down, upgrading, or uninstalled — and
  a cron line is greppable by anyone looking at the machine the normal way. **A cron
  expression is the single description of when a schedule fires**: the four common shapes
  render to one, `custom` states one outright, and the next run is read back with the same
  evaluator the Jobs tab uses on everyone else's lines, so the page, the file and the
  prediction cannot disagree. A run's outcome is written to `zfs-runs/<id>.json` beside the
  config, never to `state.json`, because the cron job is a separate process and the daemon
  would clobber it. An existing external job can be **adopted** — its schedule, retention
  and dataset read out of the cron line into a new Ham Lab schedule.
  **zfSnap is retired on `grid`** as of that day: Ham Lab now takes the nightly `tank`
  snapshot, the package is purged and the old cron line is gone, with the 515 snapshots it
  had taken left untouched and fully readable here — which is what adopting its naming
  bought. See `hamlet/docs/hosts.md`.
- **KVM tab** — shipped 2026-09-24, **removed 2026-11-09** at Mitchell's call: not needed —
  the KVMs are reached over WireGuard (see `hamlet/docs/hosts.md`). It proxied a device's web
  page through Ham Lab's origin; the code is in git history before that date.
- **Skins** — 13 since 2026-09-19 (`themes.ts`: a palette is one `mk(…)` line): Dark,
  Light, Cyberpunk, Retro, **HUD** (hairlines of cyan on near-black, corner ticks, thin
  type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light;
  each repaints the terminal too. **Each machine wears a colour** (`accents.ts`): a stripe
  across the header and its name in the switcher — picked from ten swatches or any colour
  on the Machines page, else derived from its name — saved with the theme on the hub.
- **Multi-host** — **Phase 1 shipped 2026-09-18** (`internal/nodes`, `internal/tlsauto`,
  `MachineSwitcher.tsx`, `Machines.tsx`). One binary, hub and spoke: a host whose
  `config.json` lists `nodes` is a hub; any other is a standalone node with its full UI.
  Node side: `"tls": "auto"` makes a self-signed cert in the data dir and serves HTTPS;
  `POST /api/tokens {password,name}` trades the login password for a long-lived bearer
  token (SHA-256 stored in `tokens`), accepted by the auth middleware alongside the
  cookie; `DELETE /api/tokens/{name}` revokes. Hub side: `POST /api/nodes {name,url,
  password}` peeks the node's certificate (trust on first use, fingerprint pinned),
  obtains a token named `hub:<hostname>`, stores `{name,url,fingerprint,token}`; a
  15 s prober keeps each node's status; `/api/nodes/{name}/…` is a reverse proxy to the
  node's `/api/…` (bearer swapped in, cookie and Origin stripped, streaming flush) that
  carries WebSockets, so terminals, log tails and task output work remotely.
  Frontend: `setMachine()` rewrites every `/api/…` path; the rail has a machine
  switcher at the top and a Machines page to add/remove/open. Pages remount on switch;
  terminals live on the node. Nodes: `grid`, `wyse`. **Phase 3 (update from hub)
  shipped 2026-09-18**: node `POST /api/self-update` takes a binary, checks it answers
  `version`, keeps `hamlab.prev` in the data dir, installs over the executable via
  `sudo install`, replies, then restarts the service; the hub pushes its own executable
  with `POST /api/machines/{name}/update` or `/api/machines-update-all` (architecture
  checked first via `/api/me.arch`), the Machines page shows an **outdated** badge and
  Update buttons, and `hamlab push-nodes` does the same from the CLI — `make deploy` runs
  it last, so one deploy on the hub updates every machine. First install on a new host
  is still by hand (scp binary + `install.sh`, `hamlab passwd`, `tls auto`). Still to
  do: Phase 2 Datacenter overview; optional firewall rule limiting a node's port to
  the hub. **Cross-architecture** (2026-09-18): `make deploy` also builds
  `bin/hamlab-linux-arm64`, `install.sh` installs it beside the binary as
  `hamlab-arm64`, and the hub picks the sibling matching a node's `/api/me.arch` when
  pushing — the Mac mini (Fedora Asahi, aarch64, node `mac`) updates like the rest.
- **Portability + install** — **shipped 2026-09-18** (`internal/setup`, `internal/priv`,
  `install.sh`, `make release`): the binary installs itself (`hamlab install` /
  `uninstall`; systemd or OpenRC; root, sudo or doas), all builds static on amd64, arm64,
  arm, 386 and riscv64, `install.sh` as the curl-able bootstrap with checksums. Every
  privileged call goes through `priv` instead of a literal `sudo -n`. Updates tab acts on
  dnf, zypper, pacman and apk as well as apt. Services list falls back to the plain
  table on systemd < 246; Disks falls back to `/sys/block` without `lsblk`. Listener
  uses `IP_FREEBIND`. Self-update: pre-flight `check` against the live config, atomic
  replace, exec in place. Config default moved to `~/.config/hamlab` (existing
  `~/apps/hamlab` still found by the installer and the CLI). Tested in
  containers: Debian 12, Ubuntu 22.04, Fedora 40, Rocky 8, openSUSE Leap 15.5, Alpine
  3.18/3.20 (OpenRC + doas, non-root), plus the four lab hosts. **procd (OpenWrt) added
  2026-09-26** (`internal/setup/hamlab.procd`): detected on `/sbin/procd` + `/etc/rc.common`
  ahead of the OpenRC branch, installed to the same `/etc/init.d/hamlab` path and told apart
  from OpenRC at uninstall by `USE_PROCD=1` in the file. Verified on the new router
  (`gate`, OpenWrt 25.12.5, running as root): `S95`/`K10` symlinks, procd supervising the
  instance, Docker tab and stacks working through the hub. The Updates tab needed no change
  — apk-tools 3 answers `apk version -l '<'` the same way Alpine's does. Not done: an
  OpenRC or procd backend for the Services tab (it is systemd-only and says so cleanly);
  runit/s6/sysvinit service files; public hosting for `install.sh` (the Gitea repo is
  private, so the one-liner needs `HAMLAB_TOKEN`).
- **Users, sessions and settings that follow the account** — **shipped 2026-09-22**
  (`internal/api/users.go`, `config.User`, `store.Session`/`UserState`, `Login.tsx`,
  `Users.tsx` — a hub-only **Users** tab in the rail, added the same day at Mitchell's
  request, holds everything about accounts: your own password, your sessions, the
  accounts; Settings keeps appearance, rates, tokens and About). **Rail order is per account** too (2026-09-22,
  `railorder.ts`, `Rail.tsx`): "⇅ Arrange" at the foot of the rail puts a handle on every
  page; drag one up or down (pointer events, so a finger works), let go and the order is
  saved in the UI prefs with the theme; "Reset" restores the built-in order; pages added
  in a later build go after the ones the saved order names. The login is a user name and a password. `config.json` holds `users`
  (name, argon2id hash, created, last seen); the old single `password_hash` becomes the
  account named after the OS user the console runs as on first boot, its signed-in
  sessions become that account's, and the machine-wide theme/accents/dashboard become
  theirs — nothing to redo after the upgrade. A session carries its user, an ID, when it
  was made and last seen, the IP and the browser; **Users → Sessions** lists yours
  ("Safari on iPhone · 10.0.0.5 · seen 2 min ago"), revokes one, or signs out everywhere
  else. **Users → Users** lists accounts with their session counts, adds one, removes
  one (their sessions and settings go too; never yourself, never the last), and resets
  another's password (they are signed out); removing and resetting take your own
  password first. Everyone signed in can do everything — a household console, not a
  tenancy — and each machine keeps its own accounts: a hub reaches a node with its token,
  so pairing (Machines → Add) now takes a user and password on the node, the user
  optional while it has one account. Per account: `ui` (theme, accents) and `dashboard`
  in `state.json` under `users.<name>`. Still the machine's: refresh rate (it is how
  often every machine samples — it rides in the UI prefs object for the client but is
  stored once, machine-wide), graph history, alert rules, terminals (the shells run as
  one OS user anyway). A login attempt for a missing user takes as long as a wrong
  password, so the form can't list accounts. Every write's log line now names the
  account or token (`core@10.0.0.5 via …`), and Access tokens say who issued them. CLI:
  `hamlab passwd [user]` (makes the account if new; reads stdin when piped), `hamlab
  users`, `hamlab user-rm <user>`; `hamlab install` names the first account after the
  service user. Not done: a read-only viewer role.
- **Space tab** — **shipped 2026-09-22** (`internal/space`, `internal/api/space.go`,
  `Space.tsx`), asked for as "exactly WizTree": what is using a disk. The unit is a whole
  volume, picked from a dropdown at the top: a **disk on its own** (every filesystem its
  partitions back) or a **ZFS pool as a whole** (every mounted dataset) — a pool's member
  disks are listed under it with the layout (raidz2, 8 disks: sda …) and never offered
  apart, since one disk's share of a pool means nothing (Mitchell's call, same day, after a
  first cut that listed the disks as chips). The walk runs as a helper process, `hamlab space-walk <roots>`,
  as root through `priv` when the host allows (so `/var/lib/docker` and `/root` count), 32
  workers reading directories depth-first, never crossing a filesystem boundary by itself;
  each root is grafted into one tree at its path. Files under 128 KiB are not kept one by one
  but summed per folder as "N smaller files" — what keeps a scan of millions of files to tens
  of MB in the daemon; the tree crosses back as a varint stream (`codec.go`), and the helper's
  memory goes with it. core's root: 646k files in ~1.2 s. Progress (files, bytes, current
  folder) every 250 ms; Cancel sends SIGTERM. The page: picker → Scan/Rescan → **Folders**
  (a tree largest-first: size, share of parent as a bar, files, folders, modified; expand in
  place) over a **treemap** of the selected folder — squarified, nested to the file, tiles
  coloured by kind (video, images, audio, archives & disk images, documents, code & config,
  databases & logs, other; the validated categorical set, dark and light) with folders as
  labelled boxes; hover for path, size and share; click a folder to drill, breadcrumbs and ↑
  to come back. The map asks the server for the folder pruned to what a tile could show
  (`min` = 1/6000 of the folder, 24 deep; deeper folders come back as one lump), the list
  one level at a time. **Largest files**: the 1,000 biggest with a path filter. Works on
  every machine through the hub.
  **Smarter and faster (2026-09-23, at Mitchell's request):** a scan is *kept* — in memory
  and in `<data>/space/<target>.scan` (the same varint stream) + `.json`, read back after a
  restart — and the next scan *starts from it*: the helper is handed the last tree on stdin
  and reads a directory again only when something says it changed. On a ZFS pool with
  snapshots, the scan records the newest snapshot of each dataset at start (`bases`), and the
  next one runs `zfs diff -H -F <base>` per dataset: only the directories it names are
  read, their ancestors kept but looked into, everything else kept wholesale without a
  stat. The diffs run side by side and one is given three minutes — `timeout -s KILL`
  inside sudo, since Debian's sudo gives the command its own session and a process-group
  kill from outside never reached zfs — after which that dataset is walked by mtime
  instead. On `tank` a diff is metadata-bound like the walk: 8 s to 3 min per dataset for
  a handful of changes, and `immich`, which changes constantly, ran past twenty minutes
  the first time. A refresh of `tank` took 4 min 19 s against 7 min for the full scan. Elsewhere (ext4, or a dataset with no snapshot) a directory whose mtime — to the
  nanosecond — is unchanged keeps its list and only its kept files are stat'ed again, since
  a file can grow without its directory noticing. A **cancelled** scan keeps what it walked,
  marks the folders it never got to (`partial`, "not yet read" in the tree) and **Resume**
  reads just those. core: full 1.2 s → refresh 0.3 s (2 folders read, 84,755 kept). The
  status line says which (`method`, `by_root`, read/kept counts); **Forget** drops the saved
  result so the next scan is full. The old result stays browsable while a refresh runs.
  **Treemap rethought** the same day: every file a tile grouped by folder — folders are only
  a thin line and a name strip on the first level — instead of nested boxes with headers,
  which on Linux's single-child folder chains was a staircase of headers around nothing.
  Not done: delete from here, a size-over-time history, per-user column choices.
- **Updater** — **shipped 2026-09-23** (`internal/selfupdate`, `internal/api/update.go`,
  the Updates card in `Settings.tsx`, `scripts/gitea-release.sh`). A Ham Lab asks one URL
  what the newest build is and installs it on request. `update.url` in `config.json` is
  either a Gitea/Forgejo repo API base (`…/api/v1/repos/<owner>/<repo>`, whose newest
  release names the version and carries the binaries as assets) or any static directory
  holding `VERSION`, `SHA256SUMS` and `hamlab-linux-<arch>` — which is exactly what
  `make release` writes and what `install.sh` already downloads, so the demo site, a
  folder on a NAS and a Gitea release are all the same thing to it. `"off"` never looks.
  `update.token` is sent when the source is behind a login, so a **private** Gitea serves
  its own fleet without the instance being opened to the world. The binary is verified
  against `SHA256SUMS` before it goes anywhere near the running one, then takes the same
  path `POST /api/self-update` always did (runs `version`, runs `check` against this
  machine's config, atomic replace keeping `hamlab.prev`, re-exec in place) — that path is
  now one shared method, `installBinary`. Version ordering is `install.sh`'s `vnum()` rule
  ported to Go and tested against it, so the daemon and the installer never disagree about
  which of two builds is newer; the commit count since the tag counts, the hash does not.
  Checked ~90 s after boot and daily (`update.check: "never"` switches that off), never
  installed unasked — an update restarts the daemon and drops open terminals, and the
  dialog says so. Settings → **Updates**: installed, published, when it last looked, the
  source, the release notes, Check for updates, Install; the rail badges Settings when a
  newer build is out (`update_available` rides on `/api/me`). Publishing is
  `make gitea-release` → `scripts/gitea-release.sh`, which creates the release and uploads
  the five architectures, `SHA256SUMS`, `install.sh` and `VERSION`; it is deliberately
  **not** part of `make deploy`, because deploy pushes this build to the machines this hub
  owns while publishing offers it to every install anywhere, and one tag per deploy would
  be noise. Re-publishing a version needs `--force`.
  **Public since 2026-09-23**, at Mitchell's instruction and after the trade was put to him
  twice: Gitea's instance-wide `REQUIRE_SIGNIN_VIEW` is off and `mitchell/hamlab` is marked
  public, so anyone can browse it, clone it and download a release with no account. The
  other five repos there are each marked private and are invisible — an anonymous request
  for one is a 404, not a login prompt — registration is still disabled, and new repos
  still default to private. No token is needed by anything any more (the read-only one was
  revoked; only `hamlab-publish`, which creates releases, remains). Gitea resolves
  `releases/download/latest/<asset>`, so `install.sh` and `HAMLAB_URL` now point straight
  at the newest release and never need changing. What is public is the source, the history,
  `BUGS.md` and this file — which describes the lab and names the console's hostname.
- **Auth hardening** — WebAuthn passkeys (phone Face ID), TOTP fallback, optional NPM
  access list restricting to WireGuard/LAN.
- **Files tab** — **shipped 2026-09-18** (`internal/files`, `Files.tsx`): browse from
  `/` with breadcrumbs, an editable path box, sort, hidden-file toggle and a filter;
  rows show type icon, size, modified, perms and owner, link targets and broken links.
  Text files (≤ 2 MiB, sniffed) open in an editor with Ctrl+S / Save; images preview;
  everything else downloads. Upload via drag-and-drop or picker with per-file progress
  (streamed, no size limit). New folder / file, rename, move, chmod, copy path, delete
  (folders need the name typed back), and "open in terminal", which `cd`s the active
  shell (or a fresh one) to the folder. **Sudo toggle**: root access re-executes the
  same binary as `hamlab fs <op>` under `sudo -n`, streaming bodies over stdin/stdout,
  so both modes share one implementation and one set of rules (absolute paths only;
  `/`, `/etc`, `/usr`, … are never deleted or moved whole). Every write is logged with
  the caller's IP.
- **Previews on the Files tab** — **shipped 2026-10-11** (v0.1.0-200, `internal/thumbs`,
  `internal/api/thumbs.go`, `Thumb`/`ThumbNote` in `Files.tsx`). Asked for as a button that
  toggles a file preview, "obviously not for everything", cached, "use whatever GPU is on the
  machine we're looking at, and fall back to CPU", "a single small frame preview so it's
  fast". A **previews** button beside `.hidden` turns the icon column into a 256 px
  thumbnail for every image and video row (the listing says which: `preview: image|video`
  per entry, decided by extension in one place, `thumbs.Kind`), loaded as rows scroll within
  400 px of view — the list scrolls in a box of its own, so it has its own
  IntersectionObserver rather than the browser's window-based lazy loading. A video thumb
  carries a ▶ badge. The setting is in localStorage and follows the browser.
  - **ffmpeg is the one engine**, for images and video alike; `GET /api/files/thumb` is one
    ffprobe and one ffmpeg: `-ss` a tenth of the way in (30 s at most, `-noaccurate_seek`
    so one keyframe is decoded), the first real video stream (cover art is skipped), one
    frame, `scale` to fit 256, MJPEG q4 to a pipe. JPEGs decode at a reduced DCT size
    (`-lowres`, enough halvings to stay at twice the preview) — a 24 MP photo is read as
    1.5 MP. Thumbnails are the only thing Ham Lab asks a GPU to do, and it asks the way the
    Processes tab reads one: **probed, never assumed.** The probe looks ffmpeg and ffprobe
    up, opens every `/dev/dri/renderD*`, pushes a synthetic frame through each with VAAPI,
    and reads `vainfo`'s decode profiles per node where it exists; CUDA is tried the same
    way when `/dev/nvidiactl` is there (untested: nothing here has one). A video goes to the
    first GPU listing its codec; ffmpeg's own software fallback sits behind that for a
    profile the list did not distinguish, and a CPU retry behind a hardware run that fails
    outright. Images are CPU only — one frame through libjpeg at a quarter size beats a
    round trip through a GPU. Measured on `grid`: 1080p h264 0.4 s on the UHD 750 against
    0.7 s on the CPU; the 8K Osmo HEVC 1.0 s against 2.9 s. `grid`'s two GPUs are listed in
    node order and both decode the same set (h264, hevc, vp9, mjpeg, mpeg2 — **no AV1 on
    either**, per vainfo, so AV1 is dav1d on the CPU).
  - **Cached in one database file under the machine's data dir**, `<data dir>/thumbs.db`
    (bbolt — pure Go, so the binary stays static; v0.1.0-202, the first build kept them as
    a tree of hash-named JPEGs under `thumbs/`, replaced the same day when Mitchell asked
    where they were kept: "stored in hamlab, maybe under a DB, up to you"). An entry is
    keyed by path, size and mtime — a changed file gets a new preview, an unchanged one is
    never decoded twice — and carries the engine, codec, source size and time beside the
    picture; a count rides in the file so the previews button's tooltip can say "N kept
    (M MB)" without walking anything. The URL carries `v=<mtime>-<size>` so the browser's
    own cache is right too (7 days, ETag, 304). A file ffmpeg could not decode is kept with
    ffmpeg's reason and no picture, answered as 422, retried only when the file changes; a
    timeout is not kept. A miss never creates the file, so a machine nobody browses with
    previews on never has one; a cache that cannot be opened is logged and previews are
    still made and served, just not kept. Nothing expires: a preview is a few KB, and only
    files somebody scrolled past are ever made. At most 2–4 ffmpegs at once per machine
    (half the cores, clamped), one per file however many rows ask, and a preview's work
    runs on a clock of its own so the row that asked first leaving does not cancel it for
    the rest. Each ffmpeg runs under `nice -n 10`. `hamlet`'s backup job skips the file.
  - **Root mode makes them as root** (`hamlab thumb`, the sibling of `hamlab fs`, caps on
    stdin, JSON out), so a file only root can read gets a preview, and the probe runs as
    root too — root opens every render node whatever groups the daemon has. Which matters:
    a render node is `root:render 0660`, and on `core` and `hamtv` the `core` user is not in
    `render`, so there the daemon's own previews are CPU and the tab says so, naming the GPU
    and the owner, with an **Allow GPU access** button (confirms, then a task writes
    `hamlab.service.d/gpu.conf` adding `render`/`video` as supplementary groups and
    restarts Ham Lab three seconds later, the shape the Incus install used for
    `incus-admin`). Not pressed anywhere yet. Where ffmpeg is missing (the Mac mini, `gate`)
    the note offers **Install ffmpeg with dnf/apk** through the Updates machinery
    (`ffmpeg-free` on Fedora, `ffmpeg` elsewhere); the status re-probes when the task ends.
  - *Not done, on purpose:* no cap on the cache and no clear button (not asked; it only
    holds what was looked at); no tonemapping for HDR sources (a 10-bit HEVC frame comes out
    flat); no preview in the viewer for a video (it still says "binary, download it").
- **Move… is a folder picker** — **shipped 2026-10-11** (v0.1.0-204, `MovePicker` in
  `Files.tsx`, `internal/files/move.go`, `POST /api/files/move`). Asked for as: instead of
  a path prompt, "an interactive file explorer, where I can go back/forward, etc and then
  click some button, and it will move the file/folder to where I was". The picker opens in
  the item's own folder and walks the machine the way the tab does — back and forward
  through where it has been, up, breadcrumbs, a path box, folders only, `.hidden`
  following the tab's toggle, **+ folder** to make the destination on the spot — and
  **Move here** puts the item in the folder being looked at. It says why it can't when it
  can't, with the button greyed: already here, a folder into itself, a name clash in that
  folder (read from the listing before anything is tried). The row being moved shows in
  its own folder, greyed. Rename stays a name prompt: it is a different question.
  - **Across filesystems it copies, then removes.** `os.Rename` is EXDEV between
    filesystems, and on `grid` every dataset is one — `/tank/media` to `/tank/files` was a
    "invalid cross-device link" error before this, and the picker would have made that
    move easy to attempt. `/api/files/move` renames when it can (204) and otherwise runs
    `hamlab fs move` as a task (202 with the task; the tab shows its output and reloads
    when it ends). The copy is built under a hidden name beside its destination
    (`.<name>.hamlab-moving`), renamed into place once complete, and only then is the
    original removed — a failure part-way removes the partial copy and leaves the original
    as it was, and a cancel (SIGTERM to the helper) does the same. Modes, mtimes and
    symlinks are kept; owners too when the backend is root. Devices, pipes and sockets are
    refused outright rather than half-copied. Progress every 500 files or 512 MiB.
    Verified on `grid` ext4 → ZFS with a tree holding a 0600 file, a symlink and an old
    mtime, all intact on the other side.
- **Video and audio play in the viewer** — **shipped 2026-10-11** (v0.1.0-208). Asked for
  as "keep it simple for now, and just send whatever codec the file is in to the client,
  and let the client try to decode and watch it" — and that is the whole design. Opening a
  video or audio file shows a `<video>`/`<audio>` with the browser's own controls, fed the
  file as it is by `/api/files/read`; nothing is transcoded or remuxed, so what plays is
  what the browser decodes (Safari: h264/hevc/aac in mp4/mov; Chrome: those plus vp9/av1 in
  mkv/webm, hevc only where the machine has a hardware decoder; nothing plays MPEG-TS or a
  GoPro `.360`'s raw EAC strip). When the element errors the viewer says so and leaves the
  download. What the server gained is the part a player cannot do without: reads of a
  real file with a known size go through `http.ServeContent`, so **Range requests** (every
  seek, and Safari's first request) and HEAD work, and a download can resume. The root
  backend's helper takes `--offset` for that, and `files.Seeker` opens the backend once, at
  the range asked for, rather than streaming from the top and skipping; seeking costs
  nothing until a read wants somewhere else. `/proc` files, which stat at size 0, keep the
  plain streaming path, as does the editor's capped read. Audio became a preview kind at
  the same time (`thumbs.Kind` → `audio`): its cover art, where it has any, is the row's
  thumbnail, with a ♪ badge.
  - *Not done, on purpose:* no playlist or next/previous, no subtitles. (Transcoding came
    the same evening after all — the next entry — because Safari turned out not to play
    the gaming captures.)
- **Re-encoding for the browser, on the machine's GPU** — **shipped 2026-10-11**
  (v0.1.0-212, `internal/media/{encode,hls,ffrun,inspect}.go`, `internal/api/play.go`,
  `Player.tsx`, `hls.js`). Asked for when Safari would not play the gaming captures:
  "add GPU transcoding to a supported codec when the client browser can't play it. Each
  machine should use its own GPU (if available) and transcode to probably H.264. Maybe we
  need a quality selector, and a codec selector, up to you." What Safari actually refuses
  is the **MKV container** — the captures checked are H.264 High 4:2:0, 18–36 Mbps — so
  the picker has a "repackaged" choice beside the codecs, and a 36 Mbps capture over
  WireGuard is the other reason to re-encode at all.
  - **The viewer tries the file as it is first**, as before. When the browser gives up, or
    on **Re-encode…** in the bar, a picker: **codec** — H.264, HEVC, AV1, or the original
    codec repackaged — each labelled with the engine this machine would use (or greyed as
    "not on this machine"), and **size** — original, 1080p, 720p, 480p; the choice is
    remembered. Play starts the re-encode and the player follows it: "h264 High 2292×960
    60 fps 36 Mbps in matroska → H.264 2292×960 5.0 Mbps on Intel Arc A380 (vaapi) · 7.3× ·
    34% through — seek within what's done", then "all of it is ready". **Change…** drops
    the stream and goes back to the picker.
  - **HLS, made on the machine in the switcher.** `POST /api/files/play` inspects the file
    (ffprobe, as root in root mode) and runs ffmpeg into an *event* playlist of 4 s
    fragmented-MP4 segments under `<data dir>/transcode/<id>/`, answering once the first
    segment exists (0.5–1 s on `grid`); the player reads the playlist and segments from
    `/api/files/play/<id>/…` (only those three names are served), Safari natively and
    everything else through hls.js, loaded only then. ffmpeg runs ahead at the engine's
    pace, so the seekable range grows and the whole file is through long before it is
    watched. Keyframes are forced every 4 s so segments cut cleanly; sound is AAC stereo
    unless it already is AAC; HDR is tone-mapped (`tonemap_vaapi`, or zscale/hable on the
    CPU). Nothing is kept: the same request within ten minutes reuses the stream, the
    player closes it when it goes (and the "Change…" path does), the sweeper closes what
    nobody touched for ten minutes, and the folder is emptied when the daemon starts. At
    most three at once per machine (429 with the reason). Root mode runs ffmpeg as root
    through `hamlab run-ffmpeg`, which ends when the daemon lets go of its stdin — so a
    stream cannot outlive the console.
  - **Engines are probed, never assumed**, the same probe the previews use, now also
    reading each node's encode profiles from vainfo and ffmpeg's own encoder list. The
    recipes for a codec are tried best first until one produces a first segment: a card on
    the bus before the integrated GPU (the A380 before the UHD 750), decoding on the same
    card so frames never leave it, then decoding on the CPU and uploading (a source the
    card cannot decode — AV1 on `grid`), then libx264 / libx265 / SVT-AV1. The size is
    never larger than the source, and the bit rate follows the output size (2/4/8/12/20
    Mbps for 480p/720p/1080p/1440p/2160p, +25 % at 50 fps and up, ×0.6 for HEVC and AV1).
    Measured on `grid` with a 2292×960 60 fps capture: H.264 7.3× realtime, HEVC 720p
    11.7×, AV1 720p 13×, all on the Arc. `core` and `hamtv` re-encode on the CPU until
    *Allow GPU access* is pressed (libx264 at 9.4× for a 720p→480p clip on `core`); the Mac
    and `gate` need ffmpeg first, which the previews note offers to install.
  - *Not done, on purpose:* no bit-rate ladder or adaptive switching, no subtitles, no
    keeping a finished re-encode for next time. (Restart-from-a-seek was left out here at
    first; it came 2026-10-20, the next entry.)
- **Seek anywhere in a re-encode** — **2026-10-20**. Asked for as "there's no way to seek in
  the video and have the service just start transcoding from that point onward".
  - **The playlist is the whole film, up front:** a VOD playlist on a fixed 4 s grid
    (`segCount`, the last 0.3 s left out so sound running past the picture never lists a
    segment no frame starts), so the seek bar is the film's from the first frame.
  - **A segment not on disk** is waited for when the run going now will write it within 3
    segments; otherwise it is a seek: that run is ended and one starts at the segment asked
    for. A run stops half a frame short of the next segment already on disk, never over it.
    When a run ends by itself the next gap is filled, wrapping round, until every segment is
    there. The newest request moves the run; an older one still waiting never moves it back.
    A run we end closes its half-done segment short on the way out, so whatever it finished
    after being asked to stop is deleted.
  - **Runs are interchangeable** because every one, the first included, is made the same way
    (`segArgs`): accurate `-ss` to the segment, timestamps put back with
    `-output_ts_offset start+10` (the +10 keeps every run clear of the muxer's shift for the
    B-frame delay, which a run from the top would otherwise get and a seek wouldn't), kept
    absolute in each fragment (`movflags=+frag_discont`), `-start_number`. Tested on MKV
    and on MPEG-TS with a 2.8 s start time (where `-copyts` was a second out): the init
    segments come out byte-identical, and a seeked run's segments join a from-the-top run's
    within a frame. Only the first run's init is served; a later run writes `init-run.mp4`
    and the log notes if it ever differs.
  - Repackaged ("copy") streams keep the growing event playlist: the source's keyframes
    don't sit on a grid, and repackaging is near-instant anyway.
  - A segment waits at most 45 s, under the hub proxy's 60 s for a node to answer; hls.js
    is told a segment's first byte may take 50 s. The bar reads "7.3× from 12:00 · 43%
    ready — seek anywhere".
  - Tested (`TestSeekRestartsTheEncode`): a 64 s 1080p clip, a request for segment 12 right
    after the start runs from 48 s, the gap is filled, all 16 segments joined behind the one
    init decode with no hole.
- **The player is Ham Share's; a Transcode device setting** — **2026-10-23** (v0.1.0-226,
  `internal/media/{encode,play}.go`, `internal/api/play.go`, `Player.tsx`, `vr.ts`,
  `Settings.tsx`). Asked for as "redo the HamLab one with the one you sent … a 'Transcode
  Device' setting … the A380 on Grid selected … other options being the other transcode
  devices like the iGPUs on the various CPUs that have HamLab installed". **Replaces** the
  previous two entries' design: no codec/size picker, no single-quality re-encode, no
  repackaging; the old streamer (`hls.go`) is gone.
  - **YouTube's way:** Auto (hls.js reads the connection), any quality by hand, or Original
    (the file as it is, `/api/files/read`, falling back to Auto when the browser can't). A
    ladder by size (`Ladder`): flat 2160p/1440p/1080p/720p/480p by the short side, plus the
    file's own size on top when it sits between steps (a 2292×960 capture gets 960p); 360
    files by width, named as YouTube does (2160s 3840 … 720s). 720 and down at ≤30 fps.
    H.264 High + AAC only.
  - **Pieces made when asked for**, 3 s MPEG-TS, every quality's playlist written whole up
    front. One ffmpeg per play: it starts at the piece asked, runs ahead, and is paused
    (SIGSTOP; a root run through `run-ffmpeg`, which now reads `stop`/`cont` on stdin) 10
    pieces ahead; a piece more than 3 past what's made, or at another quality, starts a new
    run there. Keyframes only on the 3 s marks and every run's times offset alike
    (`-output_ts_offset start+10`), so pieces from any run and quality join (`TestPieces`
    checks it; `SRC=… DEVICE=… go test ./internal/media -run Pieces` on a machine with a GPU).
    At most 3 runs a machine, the one asked of least lately ended for a fourth; 2 GiB of
    pieces a play; forgotten after 30 min.
  - **hls.js is patched** (`PatientAbr`): the wait for a piece being made isn't counted
    against the connection, and a piece isn't abandoned before its first byte. Without it
    Auto dropped to the bottom after every seek.
  - **Transcode device** (Settings, hub-side, `store.Transcode`): every machine's devices
    asked live (`GET /api/files/play-devices` on each), unusable ones greyed with the reason
    (hamtv's UHD 600: the render permission; mac and gate: no ffmpeg). Set to **grid's Arc
    A380**. "The machine the file is on" is the other choice. `POST /api/play` on the hub
    opens the play where the setting says; **a machine transcoding another's file reads it
    through the hub** at `/api/play-src/<token>` — a random token good for that one file, 12
    h from its last use, ranges passed through to the file's machine (ffmpeg reconnects).
    If that machine fails, the file's own makes it and the player says so.
  - Recipes per device: same-card decode first, then CPU decode + upload, then libx264 as
    the fallback whatever was asked. A card without `vainfo` (core) is tried anyway: core's
    UHD encodes. **Rotation:** the card path turns a phone's picture itself
    (`transpose_vaapi`, `-noautorotate`; ffmpeg's autorotate never applied to VAAPI frames,
    so iPhone clips came out sideways). **HLG** is shown as it is (tonemap_vaapi refuses it);
    HDR10 is tone-mapped on the card, else zscale/hable before the upload.
  - 360 files: the look-around view, zoom 8°–300° (ordinary lens into stereographic, the
    "little planet"), drag/pinch/wheel, phone motion. Full screen is the element's, or the
    whole window on an iPhone.
  - **Video previews follow the setting too** (same day, asked for as "it should use the
    same video generation device"). On the device's own machine the hub says which device
    by header (`X-Hamlab-Device`); for a file elsewhere the hub asks the file's machine's
    cache first (`?cached=1`), else has the device's machine make it from the hub's one-file
    URL (`POST /api/files/thumb-from`) and keeps it in the file's machine's cache (`PUT
    /api/files/thumb`, refused if the file changed since its listing), so each is made once.
    A failure is kept as a failure. Unreachable → made where the file is, as before. Images
    and audio covers stay on the file's machine's CPU (a JPEG is cheaper there than a GPU
    round trip). Measured: a `hamtv` video previewed on grid's A380 in 0.22 s, then 2 ms
    from hamtv's cache. The previews button's tooltip names the device.
  - Measured (`web/scripts/e2e-player.mjs`, Chromium): an iPhone HLG clip on `core` made on
    grid's A380 — 3.2 s to play, a seek 1.0 s, a quality change 1.5 s; the 8K 360 Disney day on
    grid — 3.1 s, 2160s on Auto, a seek 4.9 s (10 s source keyframes).
- **Files edge sweep** — **2026-10-11** (v0.1.0-214), asked for as "a full bug sweep of
  the Files section, every example and edge case". Every handler and both UI files read
  through with odd inputs in mind, then a battery of ~60 requests against the live hub
  (`sweep.py`, not kept: listings of files, missing and forbidden paths, NUL bytes, names
  with spaces/#/%/?/umlauts, links to files, folders and nothing, a pipe, over-size saves,
  uploads with bad names and clashes, renames and moves onto things, protected paths,
  thumbnails of empty and mislabelled images, playback of an image and an audio-only mp4,
  root-mode variants). One real data bug: **a save over the 8 MiB cap truncated the file
  and kept the first 8 MiB, answering 204** — the body is now read whole first and refused
  with 413, the file untouched. The rest: a link to a file showed the link's own time; a
  link to nothing read as an empty 200; a bad codec or size was answered by inspecting
  the file; a failed stream was handed back to the next identical request; a player
  leaving mid-start made the recipe loop try the next engine; a root-mode ffmpeg's end
  could wait forever; a page closing kept its stream for ten minutes (a beacon now
  releases it at once); the path box took only absolute paths (it resolves like a shell
  now, in the tab and the Move picker, with a test); a dropped folder became a nameless
  empty file; the upload toast counted intentions; the viewer called a pipe a binary
  file. Not bugs, left alone: `rm -rf` of a mount point empties it (the console is
  root-capable by design); a folder of ten thousand rows is slow (no virtual list).
- **Whole-console sweep** — **2026-10-13** (v0.1.0-216), asked for as "a full sweep of
  Ham Lab again, look for any bugs and fix them". Four nets, then reading: the Go race
  detector over the whole suite (clean), staticcheck's development build (the release does
  not know Go 1.26; it found only dead code and one capitalised error string), a
  garbage-in battery over every route safe to poke — 202 routes, bad identifiers, empty
  and malformed bodies, with the destructive ones held back — and a headless Chromium pass
  through all twenty tabs on the hub and on `grid`, collecting exceptions, console errors
  and failed requests (none). By hand: the auth middleware and sessions, token issue, the
  hub→node and KVM proxies, every validation regex before a shell, the cron and cron.d
  writers, config and state atomicity, every WebSocket and SSE loop's exit, the terminal
  manager, the task manager, and every destructive confirmation in the UI. Fixed, all
  small: docker inspect/health of a missing container was a 500 (404 now, a daemon that
  is down 503), `/api/virt/instances/{name}` without Incus was a 502 (409 like its
  siblings), a ZFS snapshot delete with a bad name was a 500 (400; 404 for one that does
  not exist), the task list read a task's Running flag outside its lock (a real data race,
  never seen), the Virtualization export config was written in place rather than
  atomically, and the dead code. Nothing in auth, the proxies or the shell boundaries.
  One self-inflicted finding: the battery's skip list left out `POST
  /api/files/thumbs/gpu-access`, so the sweep pressed *Allow GPU access* on `core` three
  times over — the drop-in landed and the hub restarted (that is what the headless pass
  saw as a refused connection). Kept, since it is the fix the tab offers; `core` makes
  previews and re-encodes on its UHD iGPU now. The handler refuses a second run while one
  is in flight (409), since the second `systemd-run` only collided with the first's timer.
- **Docker read-only view** — `docker ps`/logs; dockhand stays the source of truth.
- **Notifications** — push on service failure / SMART warning / UPS event.
- **Terminal extras** — search, copy-mode, font/theme, split panes, upload via
  drag-drop into cwd.

- **Settings + themes** — **shipped 2026-09-18** (`Settings.tsx`, `themes.ts`): Settings
  page with Appearance (Dark, Light, Cyberpunk, Retro — CSS variables on `<html>` plus a
  matching xterm palette, applied before first paint from localStorage and synced hub-side
  via `GET/PUT /api/prefs/ui` so the choice follows the login), Account (change password,
  `POST /api/password`, current password required), Access tokens (list/revoke the tokens
  hubs hold), About (version, host, arch, uptime, certificate).

## Network tab (2026-09-27)

A **Network** tab, per machine like every other: the interfaces as the kernel has them
(addresses, MAC, MTU, negotiated link speed, carrier, byte totals, live rates from the
sampler's own sensors), the default route and resolvers, what the internet sees of this
connection, and an **internet speed test**.

Decisions worth keeping:

- **Measured by the daemon, not the browser.** The test runs on the machine in the switcher,
  so the figure is that machine's line — the router's, if you switch to `gate` — and it
  carries on if the phone locks or the tab closes. The page polls a run it is not driving.
- **No tool to install.** It measures against Cloudflare's public speed endpoints
  (`__down`, `__up`, `cdn-cgi/trace`), which is what makes it work identically on Debian and
  on OpenWrt, where `opkg install` costs flash. That is the feature's one external
  dependency, and it is named in `internal/network/speedtest.go`.
- **Several streams, HTTP/2 off, and the ramp thrown away.** One stream over one connection
  measures a congestion window; HTTP/2 would put every stream back on one connection and do
  the same. The first second of each phase is measured and discarded.
- **Latency is the round trip less the far end's own Server-Timing.** A zero-byte response
  is mostly Cloudflare's worker running — 20 to 45 ms of it against a few ms of wire. Left
  in, it reported 31 ms for a 7 ms line. Jitter is the mean change between consecutive
  probes, not a standard deviation, so one slow probe doesn't run away with it.
- **A refused phase is not a number.** The far end rate-limits by volume, and a test moves a
  few hundred megabytes; several in a row earn 429s, and the block observed while building this
  lasted over twenty minutes. So: a 20-second cooldown counted from the last *attempt* (a
  refused test still used the line), a stream that retries twice before giving up, the rate
  computed over the window in which bytes were actually still arriving, and a plain sentence
  when it was cut short. Coasting to the end of the clock with nothing arriving reported a
  third of the truth.
- **A second download source, for when the first refuses.** A plain 100 MB file on Linode's
  Seattle mirror, which has no opinion about volume. It is only reached after a refusal, and
  every result names the source it used — that mirror is in one place, so on a machine far from
  Seattle it would read low, and a number that quietly came from somewhere else would be a lie.
  Upload stays on Cloudflare, which has never been the one to refuse.
- **The last result is published as sensors** (`net.internet.*`), so the line is a widget, a
  graph, an alert or a Prometheus series like anything else. Each carries the time the test
  ran, not the time it was asked for.
- **Not scheduled.** Nothing takes a speed test on its own. A nightly one recorded to the
  history is the obvious next step and deliberately not built yet — the cron shape for it
  already exists twice over (ZFS snapshots, Docker image updates).

## VPN tab (2026-09-27)

A **VPN** tab beside Network, per machine: every WireGuard tunnel on the machine in the
switcher, and the devices on it — name, address, last handshake, where it connects from, and
what it has moved.

- **Per machine is the point, not a limitation.** The tunnel people dial into and a tunnel a
  machine dials out through are on different boxes here (`gate` and `hamtv`), and a combined
  view would invite exactly the question "which box is this on".
- **The kernel is the truth, the config only supplies names.** `wg show all dump` has the
  handshakes and counters and knows nothing but public keys; names come from the `# comment`
  above a peer in a wg-quick file, or a UCI section's `description` on OpenWrt. Both are read
  only to label what the kernel already said.
- **Server or client is worked out, not declared.** A peer allowed `0.0.0.0/0` that has an
  endpoint is a way out, so a tunnel holding one is this machine dialling out; a tunnel with a
  listen port and peers is one being dialled. Anything else says neither rather than guessing.
- **"Online" is a handshake in the last three minutes**, and the tab says so. WireGuard is
  silent when idle; there is no connection to be in, and a tab claiming otherwise would be
  lying twice a day.
- **No secrets are read.** Private keys are in the second column of the dump and are read
  past; pre-shared keys are noted as present and never read.
- **Devices can be created, renamed and removed** (added 2026-09-27, asked for the same day),
  on a tunnel the machine configures — today OpenWrt's UCI, which is where the server here
  lives. A wg-quick host is reported unmanaged and shows no buttons: half-supporting a second
  configuration format is worse than an honest refusal.
  - **Nothing restarts the network.** `uci commit` writes the file and applies nothing; `wg
    set` changes one peer on the live interface. Proven on `gate` by the existing peers'
    transfer counters surviving a create and a delete — a reload would have zeroed them.
  - **The two halves are ordered so they cannot disagree quietly.** Adding writes the
    configuration first and rolls it back if the live tunnel refuses, because a device that
    works only after the next reboot is the worst of the three outcomes. Removing does the
    live tunnel first, since that is the moment access ends, and says so plainly if the
    configuration half then fails.
  - **The keypair is generated on the machine** and the profile, with its QR, is shown once.
    The private key is stored nowhere — the same bargain as a guest login's password.
  - **Removing asks for the name to be typed back**, at both ends. It cannot be undone and it
    sits one tap from a list of devices that all look alike.

## To do

- **Handle `fuse.mergerfs` in the Storage view** (2026-10-25). `grid` has a mergerfs pool at
  `/srv/tank` (the NVMe landing area + `/tank`; `hamlet`'s `docs/tank-landing-plan.md`).
  `fuse.mergerfs` isn't in `remoteFS` or `pseudoFS` (`internal/metrics/metrics.go`), so it shows
  as a local filesystem whose size is `tank` and the NVMe added together. That counts both twice
  in any storage total. Show it, labelled as a pool of its branches, and leave it out of totals.
- **Grow the Files tab** — asked for 2026-09-23 as two items (Files polish, and deprecating
  copyparty), merged 2026-09-24. **copyparty was retired 2026-10-11** at Mitchell's call,
  ahead of these gaps rather than after them: service, files and NPM host all gone, with a
  rollback archive kept on `grid` (`hamlet`'s `docs/hosts.md` has the detail). Files is now
  the only file browser on `grid`, so the list below is no longer parity — it is the polish
  still owed, in the order that matters:
  - **Multi-select and bulk actions.** copyparty selects many rows and moves, deletes or
    downloads them together; Files acts on one row at a time.
  - **Download a folder**, and a multi-file download, as a streamed zip or tar — the thing
    copyparty is reached for most.
  - **Resumable uploads.** copyparty's up2k chunks a file, survives a dropped connection
    and skips what is already there; Files streams one PUT, so a phone upload that drops at
    90% starts over.
  - **A gallery.** ~~Thumbnails for a folder of images and video~~ (the **previews** toggle,
    2026-10-11 — see the shipped entry above; a grid view is not built), and play
    audio/video in place — `/tank/media` and `/tank/immich` are most of what is browsed.
  - **Search** below the current folder by name, since `find` in a terminal is the fallback
    today.
  - *Decide when built:* whether share links (copyparty's public per-path links) are wanted
    at all — they are the one feature that hands out access without a login, and Ham Lab has
    no notion of that; if not, say so in the docs rather than leaving a gap unexplained.

- **Grow the Docker tab until Dockhand can go** — asked for 2026-09-23 and widened
  2026-09-24 to take in Docker tab polish and feature work, on the same reasoning as the
  Files item: it is one tab, and the gaps that block retiring the old tool and the things
  that would simply make it better are the same body of work. The *homelab* half —
  the cutover itself, then moving every stack off Dockhand's data dir — is `hamlet`'s
  `docs/pending.md` → "Retire Dockhand; Ham Lab becomes the orchestrator", agreed
  2026-09-20. This entry is the *app* half.

  **Done 2026-09-24 — the eight Mitchell picked from the inventory**, all in the Docker tab:
  per-container **Stats**, **Health** (with the last few probe outputs), **Events** (Ham Lab
  keeps its own 4000-entry ring, since `docker events` forgets after about a minute),
  **Shell** (`docker exec`, handed to the Terminal tab) and **Limits** (`docker update` for
  memory, CPU and restart policy); an **updates** tab giving the whole machine at once with
  a per-container indicator in the list; and **schedules**, which write
  `/etc/cron.d/hamlab-docker` and call
  `hamlab docker-update <id>` — the shape the ZFS schedules already use.
  Also done: Ham Lab keeps working when `dockhand` is removed. It remembers where a
  container's bind mounts pointed (`mounts` in `state.json`) and resolves a stack through
  that record when no running container explains its path, running compose on the host
  instead; `repoint` does the same for a schedule at run time. Without this, removing the
  container would have made 17 stacks on `core` and 5 on `grid` unfindable at once.

  - **Left from the Dockhand inventory, both to decide rather than build.**
    - *Git-sourced stacks.* Dockhand's `stack_sources` table points a stack at a repo and
      pulls it. Its `git_repositories` and `git_stacks` tables are **empty**, so nothing
      here uses it — dropping it is the honest answer unless Mitchell says otherwise.
    - *Webhooks, OIDC, LDAP, vulnerability scans.* All four tables empty. Nothing to move.

  **Done 2026-09-25 — the two gaps in the tab itself.**
  - *Logs are followed over a held connection*, the way the Services tab follows a journal
    (`internal/api/docker_logs.go`, `/api/docker/containers/{id}/logs/tail`): one
    `docker logs -f` per viewer, stdout and stderr both, stderr coloured as an error. A
    reconnect resumes from the last line's timestamp, so a container that restarts or a
    phone that slept doesn't replay what is already on screen; the client dedupes by line
    rather than by "later than", because `--since` is inclusive and two lines can share a
    timestamp. Docker replays the opening backlog down two pipes in no particular order, so
    the first 300 ms is held and sorted before it is sent. The polled `?since=` endpoint
    stays — it is what the resume is built on. Reconnects back off to 15 s, since a
    container that has stopped ends the stream immediately.
  - *Networks can be removed and pruned* — `POST /api/docker/networks/{remove,prune}`, the
    shape images and volumes already had. The overview says who is attached to each one, so
    the ✕ is only offered on a network nothing uses and never on Docker's own three. Prune
    is deliberately untested against this host: `core` has several networks belonging to
    stacks that are merely down (`pihole_default`, `transmission_default`, `gluetun_default`
    among them), and pruning would take them and hand them new subnets on the next deploy.

  - **The `*:2375` question — settled, and the code removed 2026-10-10.** The hub asks the
    node's own Ham Lab; nothing probes or uses a node's Docker port. `hubStacks`,
    `nodeDocker`, `taskOnHub`, the hub-side `/api/nodes/{node}/docker/…` routes and the
    `remote` / `held_by` fields are gone. A schedule saved before this that still carries a
    `remote` is refused with a message, never run against the wrong Docker. `PORTABILITY.md`
    §3 has the audit it came from. Deployed as v0.1.0-191 and checked on all five machines
    through the hub.

- ~~**A Virtualization tab**~~ — built 2026-10-09, **cut 2026-10-15**: out of scope (see Decisions). The code, its plan and its tests are on the `virtualization` branch.


## Open questions

- Idle-session reaping policy (never / after N days) — leaning never; it's a
  single-user box.
- Whether `sudo -i`-style root sessions get their own leaf colour. Cosmetic.