# Ham Lab — MVP plan

Proxmox-style web console for a bare-metal Debian 13 host. First target: `core`
(10.0.0.234). MVP is **service + shell + persistent terminals**; everything else is a
stub tab or a roadmap item.

## Decisions

| Area | Decision | Why |
|---|---|---|
| Backend | **Go 1.24**, single static binary, frontend embedded via `embed` | `core` has ~370 MiB free RAM; a Go daemon idles at ~15 MiB vs ~80+ for Node. One file to deploy, no nvm path in the unit. Same binary later installs on `grid` unchanged. |
| PTY | `creack/pty` — server-owned PTYs running `core`'s login shell | Sessions live in the daemon, not the browser, so refresh/tab-switch reattaches. No tmux dependency (not installed on `core`). |
| Transport | One WebSocket per attached terminal; REST for everything else | Standard; NPM passes WebSockets with one toggle. |
| Frontend | **React + TypeScript + Vite**, `xterm.js` (+ fit, webgl, web-links addons) | Matches `calorie-counter` / `wedding-website`, so no new toolchain to learn. |
| Auth | Single-user **password login**, argon2id hash, HttpOnly/Secure/SameSite=Strict session cookie, login rate-limit, Origin check on WS upgrade | This is a root-capable shell on the public internet. Auth is MVP, not roadmap. |
| Service | `hamlab.service`, `User=core`, `Restart=always`, binary at `/usr/local/bin/hamlab` | Same pattern as `calorie-counter.service`. Runs as `core` so `sudo` works in the terminal without a password, as on the SSH path. |
| Listen | `10.0.0.234:8100` (free) | NPM is a container, so loopback-only won't do. Firewalling to NPM is a later hardening step. |
| Data | `/home/core/apps/hamlab/` — `config.yaml`, `state.db` (SQLite) | `apps/` is the nightly rsync target to `grid`, so it's backed up for free. |
| Ingress | NPM proxy host `hamlab.bigham.dev` → `:8100`, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a **second** API call (NPM drops them on create-with-new-cert) | `*.bigham.dev` already resolves; the proxy host alone makes it live. |
| Repo | Gitea `mitchell/hamlab`, private, `main` | Created 2026-09-17. |

## Layout

```
┌──────────────────────────────────────────────────────────────┐
│ header: Ham Lab · core · load 0.42 · ● connected · [logout]    │
├──────────────┬───────────────────────────────────────────────┤
│ Dashboard    │ shell 1 ● │ backups │ +                      │
│ Processes    ├───────────────────────────────────────────────┤
│ Terminal     │                                               │
│ Services     │        main content (xterm)                   │
│ Cron         │                                               │
│ …            │                                               │
├──────────────┴───────────────────────────────────────────────┤
│ footer: v0.1.0 · up 12d · hamlab.bigham.dev                   │
└──────────────────────────────────────────────────────────────┘
```

- Left rail: flat list of pages, active one highlighted. No nested leaves.
- Terminal page: tab strip across the top of the pane, one tab per session, `● ` when
  a process is running, `+` pinned at the right edge. Active tab carries rename and
  close; double-click renames, middle-click closes. (Until 2026-09-18 the sessions
  were leaves under a collapsible Terminal group in the rail.)
- Non-MVP groups render as disabled leaves with "coming soon" — the rail shape is
  the product; stubs keep it honest about scope.
- **Mobile**: rail becomes a slide-in drawer; xterm fits the viewport; a sticky key
  bar above the keyboard (`Esc Tab Ctrl Alt ↑ ↓ ← → |`) because phone keyboards
  have none of those. This is what makes it usable as a PWA rather than a novelty.

## Terminal semantics

- A session is a PTY + shell owned by the daemon, with an ID, name, cwd hint, and a
  **scrollback ring buffer** (256 KiB). Attaching replays the buffer, then streams.
- Any number of browser tabs can attach to the same session (all get output; all can
  type). Resize follows the most recent attacher.
- Switching tabs in the UI just detaches/attaches; the shell keeps running.
- Closing a session sends `SIGHUP` to the process group and drops it. Shell exit
  closes the session automatically (leaf shows "exited" until dismissed).
- Session names persist in `state.db` so the sidebar order survives a service
  restart. **The shells themselves do not survive a daemon restart** in the MVP —
  see roadmap (tmux backing).
- Idle sessions cost one shell process each; no automatic reaping in MVP.

## API (MVP)

```
POST /api/login            {password}            → 204 + cookie
POST /api/logout                                 → 204
GET  /api/me                                     → {host, user, version, uptime, load}
GET  /api/terminals                              → [{id, name, running, created}]
POST /api/terminals        {name?}               → {id, ...}
PATCH /api/terminals/:id   {name}                → 204
DELETE /api/terminals/:id                        → 204
WS   /api/terminals/:id/attach   binary frames = raw bytes; JSON text frames = {resize}
GET  /healthz                                    → 200 (unauthenticated, for NPM)
```

Everything under `/api` except `login` and `/healthz` requires the session cookie.

## Project layout

```
hamlab/
  cmd/hamlab/main.go          serve | passwd | version
  internal/
    auth/                     argon2id, sessions, rate limit
    term/                     PTY manager, ring buffer, attach/detach
    api/                      chi router, handlers, WS
    state/                    SQLite (modernc, no cgo)
    web/                      go:embed of web/dist
  web/                        Vite + React + TS + xterm.js, PWA manifest + SW
  deploy/
    hamlab.service
    install.sh                build → /usr/local/bin/hamlab → systemctl restart
  Makefile                    build / dev / deploy / lint / test
```

## Milestones

**M0 — skeleton** (service boots, login works)
- Go module, chi, config loading, `hamlab passwd`, argon2id login, session cookie.
- `hamlab.service` on `core`, `:8100`, serves a placeholder page.
- `make deploy` from `core` itself.

**M1 — terminals** (the MVP feature)
- PTY manager with ring buffer, create/list/rename/close, WS attach with resize.
- React shell: header, rail, footer, Terminal group, xterm main pane.
- Persistence of session list across refresh and rail switching.

**M2 — polish + PWA + ingress**
- Mobile drawer, key bar, fit-on-rotate.
- `manifest.webmanifest`, icons (Ham logo on black, same recipe as HamTV), service
  worker caching the app shell only (never `/api`).
- NPM proxy host for `hamlab.bigham.dev`; verify WS through NPM; force-SSL/HSTS/HTTP2.
- Stub groups for Services / Cron / ZFS / Dashboard.

**Done when:** from the phone, open the PWA, log in, open two terminals, run
something long in one, switch away and back, refresh — it's all still there.

## Roadmap (not MVP — notes only)

- **Updates tab** — **shipped 2026-09-17** (`internal/updates`, `internal/tasks`,
  `Updates.tsx`): upgradable packages with current → candidate, suite, security badge
  and description; index age, last upgrade (from `/var/log/apt/history.log`),
  unattended-upgrades on/off, reboot-required banner with the packages that caused it;
  Refresh index, Upgrade all (dist-upgrade), Upgrade selected; changelog per package on
  demand. Runs go through the new **tasks** package: background command, output kept,
  followed live over `/api/tasks/{id}/stream`, cancellable, listed at `/api/tasks` (the
  seed of a Task History tab). apt is fully supported; dnf/pacman list only.
- **Power** — header ⏻ button (amber when a reboot is pending), dialog that requires the
  hostname typed back, then `systemctl reboot` / `poweroff`. `/api/me` carries
  `reboot_needed` so the header knows.

- **tmux-backed sessions** so shells survive a daemon restart / redeploy. Would spawn
  `tmux new -A -s hamlab-<id>`; needs `mouse on` and status-bar handling in xterm.
- **Services tab** — `systemctl list-units`, per-unit page: status, last 200 journal
  lines (live-tail via WS), start/stop/restart/enable buttons, unit file viewer.
- **Cron tab** — **shipped 2026-09-17** (`internal/cron`, `Cron.tsx`): one list built
  from `/etc/crontab`, `/etc/cron.d/*`, every user crontab (via sudo), the scripts in
  the `run-parts` directories those drive, plus systemd timers. Next run computed by an
  in-house cron-expression evaluator (tested: ranges, steps, names, dom-OR-dow rule,
  `@` aliases); last run from the journal's `(user) CMD (command)` lines (cron logs via
  syslog to journald; reading needs sudo/adm). Commented-out jobs show as disabled;
  job-shaped lines inside comment blocks (Debian's header examples) are ignored. Per
  job: Run now (background, as the job's user, `sh -c` with crontab `%` rules, output
  captured), Enable/Disable (comments the line in place, refuses if the file changed
  underneath), Edit source (textarea; validated line-by-line; user crontabs installed
  with `crontab -`, system files written via `sudo tee`), Cron log, and Output file
  when the command redirects to one. Timers: Run now (starts the activated service),
  start/stop/enable/disable, live logs of the service via the Services tab's tail.
- **Disks tab** — **shipped 2026-09-17** (`internal/storage`, `Disks.tsx`), replacing
  the planned ZFS tab: block devices from `lsblk -J` (kind, model, size, temperature
  from the device's hwmon node, partitions with filesystem usage, live read/write/busy
  from the metrics sampler); SMART via `smartctl -j` when smartmontools is present (one
  button installs it as a task otherwise): health, temperature, power-on time, self-test
  status, ATA attribute table with the failure-predicting IDs flagged, NVMe endurance and
  error counters, short/long self-tests. Spinning disks are read with `-n standby` so
  the tab never wakes a sleeping drive; "Wake and read" is explicit. ZFS section when
  the tools and module are there: pools with health, capacity, fragmentation, last scan
  and vdev tree; scrub start/pause/stop; datasets tree with used/refer/in-snapshots/
  snapshot count/compressratio/mount; per-dataset snapshot list and "Snapshot now".
  Verified against `grid` (raidz2 `tank`, 10 datasets, 515 snapshots, 8 WD40EZAZ + 970
  EVO Plus SMART). Snapshot destroy and dataset property editing are deliberately not
  there yet.
- **Dashboard** — **first cut shipped 2026-09-17** (`internal/metrics`, `Dashboard.tsx`):
  CPU (total + per core), memory/swap, load, uptime, hwmon temps and fans read straight
  from `/sys` (no lm-sensors), per-interface network rates, per-disk read/write/busy
  from `/proc/diskstats`, filesystem usage, UPS via `sudo -n pwrstat -status`. Sampled
  every 2 s into an in-memory ring of 10 min; the UI (the **Monitors** tab
  since 2026-09-19) reads it all through the sensor API — see Sensors below. **Customizable**: a
  Customize mode reorders and hides modules (CPU, Memory, Load, Uptime, Temperatures,
  Fans, Network, Disks, Filesystems, UPS) and hides individual rows (an interface, a
  mount, a disk, a sensor); the layout is stored server-side in `state.json` via
  `GET/PUT /api/prefs/dashboard` so every device sees the same dashboard. The update
  interval (100 ms – 2 s) is part of the layout and drives both the browser's polling and
  the daemon's sampling clock; the chart window is always 300 samples, so 30 s at 100 ms
  and 10 min at 2 s. Still to do: persistent
  history (SQLite ring, e.g. 10 s × 7 days), SMART (`smartctl -j`), longer time ranges.
- **Sensors** — **shipped 2026-09-19** (`internal/sensors`, `internal/metrics/sensors.go`,
  `internal/api/status_sensors.go`, `web/src/sensors.ts`, `widgets.tsx`, `Embed.tsx`;
  reference in `SENSORS.md`): everything measured is a sensor with a stable dotted ID;
  catalog, single value, history, a **shaped query** (the client sends the object it wants
  and gets it back filled in — GraphQL's idea without GraphQL, since sensors differ per
  machine and a static schema would collapse to `sensor(id:)`), the same query as a live
  SSE stream, Prometheus `/metrics`, read-only `sensors:read` tokens (never full tokens in
  a URL), status sensors (services, updates, ZFS, SMART), a widget library, custom widgets
  on the Monitors tab (was "Dashboard"; the view id and URL are still `dashboard`), and a
  frameable `/embed/widget` for Glance and friends. The stock Monitors modules and the Disks
  tab read the sensor API too (`sampleAdapter.ts`); `/api/metrics` is retired.
  Also per-service and per-container sensors (memory and CPU from cgroups), tokens limited
  to ID patterns, and `/api/fleet/sensors/query` for the whole fleet in one request.
- **Alerts** — **shipped 2026-09-19** (`internal/api/alerts.go`, `Alerts.tsx`): rules over
  sensors (`container.*.running == false for 120 s`), evaluated by the hub every 15 s for
  itself and every managed machine in one shaped query each; `hamlab.reachable` is supplied
  by the hub so "a machine stopped answering" is a rule like any other. Ten defaults
  (machine offline, container down/unhealthy, failed service, disk full, pool unhealthy,
  SMART failing, UPS on battery, CPU hot, memory). Firing and resolved are events: kept
  (last 200, in `state.json`), shown on the hub-only Alerts tab with a count on the rail,
  and POSTed to channels — **ntfy** (title/priority headers) or a **webhook** (JSON; Home
  Assistant, Slack). The rule editor shows what a selector matches across the fleet as
  you type. Not done: quiet hours, per-rule channels, Web Push straight to the PWA.
- **Docker tab** — **shipped 2026-09-19** (`internal/api/docker.go`, `Docker.tsx`):
  containers grouped by compose stack with live memory and CPU from the container
  sensors, start / stop / restart (asked first), logs that follow (polled with `--since`,
  no held connection), inspect (env *names* only), and images / volumes / networks
  tables. Works on every machine through the hub; `#/grid/docker/immich_server` links to
  one. **Stacks** (`internal/api/stacks.go`): every compose project, found from its
  containers' labels, with its file to edit and Deploy (`up -d`), Update (pull, then
  deploy), Restart and Stop as followed tasks. The file is looked for where the label says
  *and through every container's mounts*, because an orchestrator that runs compose inside
  its own container (Dockhand here; Dockge, Portainer) records a path in its own
  filesystem; compose is then run **inside that container**, so `.env`, relative paths
  and the compose labels stay exactly as deployed (verified: labels identical after a
  redeploy). Save validates with `compose config` first, keeps the previous file as
  `<file>.hamlab-bak`, and never deploys by itself. A stack deployed from another host
  (Dockhand on `core` driving `grid` over tcp) has no file on that machine, so the node
  alone lists it read-only — but **the hub fills that in** (`hubStacks`): asked about a
  node, it resolves the files on itself and runs compose here with `DOCKER_HOST` pointed
  at the node's `tcp://…:2375`, the way it was deployed; the task runs on the hub and the
  node's page follows it. Also per stack: every file it was deployed from (chips), its
  **`.env`** (values masked until "Reveal and edit"; validated in place, put back if
  compose objects), **what Deploy would change** (a line diff against the copy kept at the
  last deploy from here, `<file>.hamlab-deployed`, else the copy the last save replaced;
  `.env` values never shown), and **image update checks** — asked for, never polled, since
  each is a registry request Docker Hub counts; remembered six hours. A save refuses a
  file changed on disk meanwhile. Removing containers, images and volumes is still absent.
- **Skins** — 13 since 2026-09-19 (`themes.ts`: a palette is one `mk(…)` line): Dark,
  Light, Cyberpunk, Retro, **HUD** (hairlines of cyan on near-black, corner ticks, thin
  type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light;
  each repaints the terminal too. **Each machine wears a colour** (`accents.ts`): a stripe
  across the header and its name in the switcher — picked from ten swatches or any colour
  on the Machines page, else derived from its name — saved with the theme on the hub.
- **Multi-host** — **Phase 1 shipped 2026-09-18** (`internal/nodes`, `internal/tlsauto`,
  `MachineSwitcher.tsx`, `Machines.tsx`). One binary, hub and spoke: a host whose
  `config.json` lists `nodes` is a hub; any other is a standalone node with its full UI.
  Node side: `"tls": "auto"` makes a self-signed cert in the data dir and serves HTTPS;
  `POST /api/tokens {password,name}` trades the login password for a long-lived bearer
  token (SHA-256 stored in `tokens`), accepted by the auth middleware alongside the
  cookie; `DELETE /api/tokens/{name}` revokes. Hub side: `POST /api/nodes {name,url,
  password}` peeks the node's certificate (trust on first use, fingerprint pinned),
  obtains a token named `hub:<hostname>`, stores `{name,url,fingerprint,token}`; a
  15 s prober keeps each node's status; `/api/nodes/{name}/…` is a reverse proxy to the
  node's `/api/…` (bearer swapped in, cookie and Origin stripped, streaming flush) that
  carries WebSockets, so terminals, log tails and task output work remotely.
  Frontend: `setMachine()` rewrites every `/api/…` path; the rail has a machine
  switcher at the top and a Machines page to add/remove/open. Pages remount on switch;
  terminals live on the node. Nodes: `grid`, `wyse`. **Phase 3 (update from hub)
  shipped 2026-09-18**: node `POST /api/self-update` takes a binary, checks it answers
  `version`, keeps `hamlab.prev` in the data dir, installs over the executable via
  `sudo install`, replies, then restarts the service; the hub pushes its own executable
  with `POST /api/machines/{name}/update` or `/api/machines-update-all` (architecture
  checked first via `/api/me.arch`), the Machines page shows an **outdated** badge and
  Update buttons, and `hamlab push-nodes` does the same from the CLI — `make deploy` runs
  it last, so one deploy on the hub updates every machine. First install on a new host
  is still by hand (scp binary + `install.sh`, `hamlab passwd`, `tls auto`). Still to
  do: Phase 2 Datacenter overview; optional firewall rule limiting a node's port to
  the hub. **Cross-architecture** (2026-09-18): `make deploy` also builds
  `bin/hamlab-linux-arm64`, `install.sh` installs it beside the binary as
  `hamlab-arm64`, and the hub picks the sibling matching a node's `/api/me.arch` when
  pushing — the Mac mini (Fedora Asahi, aarch64, node `mac`) updates like the rest.
- **Portability + install** — **shipped 2026-09-18** (`internal/setup`, `internal/priv`,
  `install.sh`, `make release`): the binary installs itself (`hamlab install` /
  `uninstall`; systemd or OpenRC; root, sudo or doas), all builds static on amd64, arm64,
  arm, 386 and riscv64, `install.sh` as the curl-able bootstrap with checksums. Every
  privileged call goes through `priv` instead of a literal `sudo -n`. Updates tab acts on
  dnf, zypper, pacman and apk as well as apt. Services list falls back to the plain
  table on systemd < 246; Disks falls back to `/sys/block` without `lsblk`. Listener
  uses `IP_FREEBIND`. Self-update: pre-flight `check` against the live config, atomic
  replace, exec in place. Config default moved to `~/.config/hamlab` (existing
  `~/apps/hamlab` still found by the installer and the CLI). Tested in
  containers: Debian 12, Ubuntu 22.04, Fedora 40, Rocky 8, openSUSE Leap 15.5, Alpine
  3.18/3.20 (OpenRC + doas, non-root), plus the four lab hosts. Not done: an OpenRC
  backend for the Services tab; runit/s6/sysvinit service files; public hosting for
  `install.sh` (the Gitea repo is private, so the one-liner needs `HAMLAB_TOKEN`).
- **Auth hardening** — WebAuthn passkeys (phone Face ID), TOTP fallback, session
  list + revoke, optional NPM access list restricting to WireGuard/LAN.
- **Files tab** — **shipped 2026-09-18** (`internal/files`, `Files.tsx`): browse from
  `/` with breadcrumbs, an editable path box, sort, hidden-file toggle and a filter;
  rows show type icon, size, modified, perms and owner, link targets and broken links.
  Text files (≤ 2 MiB, sniffed) open in an editor with Ctrl+S / Save; images preview;
  everything else downloads. Upload via drag-and-drop or picker with per-file progress
  (streamed, no size limit). New folder / file, rename, move, chmod, copy path, delete
  (folders need the name typed back), and "open in terminal", which `cd`s the active
  shell (or a fresh one) to the folder. **Sudo toggle**: root access re-executes the
  same binary as `hamlab fs <op>` under `sudo -n`, streaming bodies over stdin/stdout,
  so both modes share one implementation and one set of rules (absolute paths only;
  `/`, `/etc`, `/usr`, … are never deleted or moved whole). Every write is logged with
  the caller's IP.
- **Docker read-only view** — `docker ps`/logs; dockhand stays the source of truth.
- **Notifications** — push on service failure / SMART warning / UPS event.
- **Terminal extras** — search, copy-mode, font/theme, split panes, upload via
  drag-drop into cwd.

- **Settings + themes** — **shipped 2026-09-18** (`Settings.tsx`, `themes.ts`): Settings
  page with Appearance (Dark, Light, Cyberpunk, Retro — CSS variables on `<html>` plus a
  matching xterm palette, applied before first paint from localStorage and synced hub-side
  via `GET/PUT /api/prefs/ui` so the choice follows the login), Account (change password,
  `POST /api/password`, current password required), Access tokens (list/revoke the tokens
  hubs hold), About (version, host, arch, uptime, certificate).

## Open questions

- Idle-session reaping policy (never / after N days) — leaning never; it's a
  single-user box.
- Whether `sudo -i`-style root sessions get their own leaf colour. Cosmetic.