Mitchell-JohnsonandClaude Opus 5
94800b8207
The Updates tab could tell you eleven images were behind and offer nothing to do about it; updating meant knowing which stack each image belonged to and going to find it. Now each row has Update and the header has Update all. Updating is per stack, because that is the unit compose deals in, so a row acts on whichever stacks use that image — postgres:15-alpine is wikijs's, postgres:17 is vikunja's — and Update all walks the affected stacks through a queue one at a time, with each task's output on screen, rather than recreating half the machine at once. A stack that finishes says "updated just now" rather than re-checking, since every check is a registry request and Docker Hub counts them. Also: cron expressions read as English. "daily at 3:15am", with 15 3 * * * kept beside it. Jobs.tsx had a smaller version of this for its own list; it now uses the shared one, which handles lists, ranges, steps, names, weekdays/weekends, midnight and noon, and says nothing at all when it is not certain — the caller falls back to the expression. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
671 lines
52 KiB
Markdown
671 lines
52 KiB
Markdown
# Ham Lab — MVP plan
|
||
|
||
Proxmox-style web console for a bare-metal Debian 13 host. First target: `core`
|
||
(10.0.0.234). MVP is **service + shell + persistent terminals**; everything else is a
|
||
stub tab or a roadmap item.
|
||
|
||
## Decisions
|
||
|
||
| Area | Decision | Why |
|
||
|---|---|---|
|
||
| Backend | **Go 1.24**, single static binary, frontend embedded via `embed` | `core` has ~370 MiB free RAM; a Go daemon idles at ~15 MiB vs ~80+ for Node. One file to deploy, no nvm path in the unit. Same binary later installs on `grid` unchanged. |
|
||
| PTY | `creack/pty` — server-owned PTYs running `core`'s login shell | Sessions live in the daemon, not the browser, so refresh/tab-switch reattaches. No tmux dependency (not installed on `core`). |
|
||
| Transport | One WebSocket per attached terminal; REST for everything else | Standard; NPM passes WebSockets with one toggle. |
|
||
| Frontend | **React + TypeScript + Vite**, `xterm.js` (+ fit, webgl, web-links addons) | Matches `calorie-counter` / `wedding-website`, so no new toolchain to learn. |
|
||
| Auth | Single-user **password login**, argon2id hash, HttpOnly/Secure/SameSite=Strict session cookie, login rate-limit, Origin check on WS upgrade | This is a root-capable shell on the public internet. Auth is MVP, not roadmap. |
|
||
| Service | `hamlab.service`, `User=core`, `Restart=always`, binary at `/usr/local/bin/hamlab` | Same pattern as `calorie-counter.service`. Runs as `core` so `sudo` works in the terminal without a password, as on the SSH path. |
|
||
| Listen | `10.0.0.234:8100` (free) | NPM is a container, so loopback-only won't do. Firewalling to NPM is a later hardening step. |
|
||
| Data | `/home/core/apps/hamlab/` — `config.yaml`, `state.db` (SQLite) | `apps/` is the nightly rsync target to `grid`, so it's backed up for free. |
|
||
| Ingress | NPM proxy host `hamlab.bigham.dev` → `:8100`, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a **second** API call (NPM drops them on create-with-new-cert) | `*.bigham.dev` already resolves; the proxy host alone makes it live. |
|
||
| Repo | Gitea `mitchell/hamlab`, private, `main` | Created 2026-09-17. |
|
||
| Machines | **A view shows one machine: the one in the switcher.** No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it. | Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still *proxies* for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected. |
|
||
|
||
## Layout
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────┐
|
||
│ header: Ham Lab · core · load 0.42 · ● connected · [logout] │
|
||
├──────────────┬───────────────────────────────────────────────┤
|
||
│ Dashboard │ shell 1 ● │ backups │ + │
|
||
│ Processes ├───────────────────────────────────────────────┤
|
||
│ Terminal │ │
|
||
│ Services │ main content (xterm) │
|
||
│ Cron │ │
|
||
│ … │ │
|
||
├──────────────┴───────────────────────────────────────────────┤
|
||
│ footer: v0.1.0 · up 12d · hamlab.bigham.dev │
|
||
└──────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
- Left rail: flat list of pages, active one highlighted. No nested leaves.
|
||
- Terminal page: tab strip across the top of the pane, one tab per session, `● ` when
|
||
a process is running, `+` pinned at the right edge. Active tab carries rename and
|
||
close; double-click renames, middle-click closes. (Until 2026-09-18 the sessions
|
||
were leaves under a collapsible Terminal group in the rail.)
|
||
- Non-MVP groups render as disabled leaves with "coming soon" — the rail shape is
|
||
the product; stubs keep it honest about scope.
|
||
- **Mobile**: rail becomes a slide-in drawer; xterm fits the viewport; a sticky key
|
||
bar above the keyboard (`Esc Tab Ctrl Alt ↑ ↓ ← → |`) because phone keyboards
|
||
have none of those. This is what makes it usable as a PWA rather than a novelty.
|
||
|
||
## Terminal semantics
|
||
|
||
- A session is a PTY + shell owned by the daemon, with an ID, name, cwd hint, and a
|
||
**scrollback ring buffer** (256 KiB). Attaching replays the buffer, then streams.
|
||
- Any number of browser tabs can attach to the same session (all get output; all can
|
||
type). Resize follows the most recent attacher.
|
||
- Switching tabs in the UI just detaches/attaches; the shell keeps running.
|
||
- Closing a session sends `SIGHUP` to the process group and drops it. Shell exit
|
||
closes the session automatically (leaf shows "exited" until dismissed).
|
||
- Session names persist in `state.db` so the sidebar order survives a service
|
||
restart. **The shells themselves do not survive a daemon restart** in the MVP —
|
||
see roadmap (tmux backing).
|
||
- Idle sessions cost one shell process each; no automatic reaping in MVP.
|
||
|
||
## API (MVP)
|
||
|
||
```
|
||
POST /api/login {password} → 204 + cookie
|
||
POST /api/logout → 204
|
||
GET /api/me → {host, user, version, uptime, load}
|
||
GET /api/terminals → [{id, name, running, created}]
|
||
POST /api/terminals {name?} → {id, ...}
|
||
PATCH /api/terminals/:id {name} → 204
|
||
DELETE /api/terminals/:id → 204
|
||
WS /api/terminals/:id/attach binary frames = raw bytes; JSON text frames = {resize}
|
||
GET /healthz → 200 (unauthenticated, for NPM)
|
||
```
|
||
|
||
Everything under `/api` except `login` and `/healthz` requires the session cookie.
|
||
|
||
## Project layout
|
||
|
||
```
|
||
hamlab/
|
||
cmd/hamlab/main.go serve | passwd | version
|
||
internal/
|
||
auth/ argon2id, sessions, rate limit
|
||
term/ PTY manager, ring buffer, attach/detach
|
||
api/ chi router, handlers, WS
|
||
state/ SQLite (modernc, no cgo)
|
||
web/ go:embed of web/dist
|
||
web/ Vite + React + TS + xterm.js, PWA manifest + SW
|
||
deploy/
|
||
hamlab.service
|
||
install.sh build → /usr/local/bin/hamlab → systemctl restart
|
||
Makefile build / dev / deploy / lint / test
|
||
```
|
||
|
||
## Milestones
|
||
|
||
**M0 — skeleton** (service boots, login works)
|
||
- Go module, chi, config loading, `hamlab passwd`, argon2id login, session cookie.
|
||
- `hamlab.service` on `core`, `:8100`, serves a placeholder page.
|
||
- `make deploy` from `core` itself.
|
||
|
||
**M1 — terminals** (the MVP feature)
|
||
- PTY manager with ring buffer, create/list/rename/close, WS attach with resize.
|
||
- React shell: header, rail, footer, Terminal group, xterm main pane.
|
||
- Persistence of session list across refresh and rail switching.
|
||
|
||
**M2 — polish + PWA + ingress**
|
||
- Mobile drawer, key bar, fit-on-rotate.
|
||
- `manifest.webmanifest`, icons (Ham logo on black, same recipe as HamTV), service
|
||
worker caching the app shell only (never `/api`).
|
||
- NPM proxy host for `hamlab.bigham.dev`; verify WS through NPM; force-SSL/HSTS/HTTP2.
|
||
- Stub groups for Services / Cron / ZFS / Dashboard.
|
||
|
||
**Done when:** from the phone, open the PWA, log in, open two terminals, run
|
||
something long in one, switch away and back, refresh — it's all still there.
|
||
|
||
## Roadmap (not MVP — notes only)
|
||
|
||
- **Updates tab** — **shipped 2026-09-17** (`internal/updates`, `internal/tasks`,
|
||
`Updates.tsx`): upgradable packages with current → candidate, suite, security badge
|
||
and description; index age, last upgrade (from `/var/log/apt/history.log`),
|
||
unattended-upgrades on/off, reboot-required banner with the packages that caused it;
|
||
Refresh index, Upgrade all (dist-upgrade), Upgrade selected; changelog per package on
|
||
demand. Runs go through the new **tasks** package: background command, output kept,
|
||
followed live over `/api/tasks/{id}/stream`, cancellable, listed at `/api/tasks` (the
|
||
seed of a Task History tab). apt is fully supported; dnf/pacman list only.
|
||
- **Power** — header ⏻ button (amber when a reboot is pending), dialog that requires the
|
||
hostname typed back, then `systemctl reboot` / `poweroff`. `/api/me` carries
|
||
`reboot_needed` so the header knows.
|
||
|
||
- **tmux-backed sessions** so shells survive a daemon restart / redeploy. Would spawn
|
||
`tmux new -A -s hamlab-<id>`; needs `mouse on` and status-bar handling in xterm.
|
||
- **Services tab** — `systemctl list-units`, per-unit page: status, last 200 journal
|
||
lines (live-tail via WS), start/stop/restart/enable buttons, unit file viewer.
|
||
- **Cron tab** — **shipped 2026-09-17** (`internal/cron`, `Cron.tsx`): one list built
|
||
from `/etc/crontab`, `/etc/cron.d/*`, every user crontab (via sudo), the scripts in
|
||
the `run-parts` directories those drive, plus systemd timers. Next run computed by an
|
||
in-house cron-expression evaluator (tested: ranges, steps, names, dom-OR-dow rule,
|
||
`@` aliases); last run from the journal's `(user) CMD (command)` lines (cron logs via
|
||
syslog to journald; reading needs sudo/adm). Commented-out jobs show as disabled;
|
||
job-shaped lines inside comment blocks (Debian's header examples) are ignored. Per
|
||
job: Run now (background, as the job's user, `sh -c` with crontab `%` rules, output
|
||
captured), Enable/Disable (comments the line in place, refuses if the file changed
|
||
underneath), Edit source (textarea; validated line-by-line; user crontabs installed
|
||
with `crontab -`, system files written via `sudo tee`), Cron log, and Output file
|
||
when the command redirects to one. Timers: Run now (starts the activated service),
|
||
start/stop/enable/disable, live logs of the service via the Services tab's tail.
|
||
- **Disks tab** — **shipped 2026-09-17** (`internal/storage`, `Disks.tsx`), replacing
|
||
the planned ZFS tab: block devices from `lsblk -J` (kind, model, size, temperature
|
||
from the device's hwmon node, partitions with filesystem usage, live read/write/busy
|
||
from the metrics sampler); SMART via `smartctl -j` when smartmontools is present (one
|
||
button installs it as a task otherwise): health, temperature, power-on time, self-test
|
||
status, ATA attribute table with the failure-predicting IDs flagged, NVMe endurance and
|
||
error counters, short/long self-tests. Spinning disks are read with `-n standby` so
|
||
the tab never wakes a sleeping drive; "Wake and read" is explicit. ZFS section when
|
||
the tools and module are there: pools with health, capacity, fragmentation, last scan
|
||
and vdev tree; scrub start/pause/stop; datasets tree with used/refer/in-snapshots/
|
||
snapshot count/compressratio/mount; per-dataset snapshot list and "Snapshot now".
|
||
Verified against `grid` (raidz2 `tank`, 10 datasets, 515 snapshots, 8 WD40EZAZ + 970
|
||
EVO Plus SMART). Snapshot destroy and dataset property editing are deliberately not
|
||
there yet.
|
||
- **Dashboard** — **first cut shipped 2026-09-17** (`internal/metrics`, `Dashboard.tsx`):
|
||
CPU (total + per core), memory/swap, load, uptime, hwmon temps and fans read straight
|
||
from `/sys` (no lm-sensors), per-interface network rates, per-disk read/write/busy
|
||
from `/proc/diskstats`, filesystem usage, UPS via `sudo -n pwrstat -status`. Sampled
|
||
every 2 s into an in-memory ring of 10 min; the UI (the **Dashboard** tab
|
||
since 2026-09-19) reads it all through the sensor API — see Sensors below. **Customizable**: a
|
||
Customize mode reorders and hides modules (CPU, Memory, Load, Uptime, Temperatures,
|
||
Fans, Network, Disks, Filesystems, UPS) and hides individual rows (an interface, a
|
||
mount, a disk, a sensor); the layout is stored server-side in `state.json` via
|
||
`GET/PUT /api/prefs/dashboard` so every device sees the same dashboard. The update
|
||
interval (100 ms – 2 s) is part of the layout and drives both the browser's polling and
|
||
the daemon's sampling clock; the chart window is always 300 samples, so 30 s at 100 ms
|
||
and 10 min at 2 s. Still to do: persistent
|
||
history (SQLite ring, e.g. 10 s × 7 days), SMART (`smartctl -j`), longer time ranges.
|
||
- **Updating from the Updates tab** — **shipped 2026-09-26**: the "what is out of date" list
|
||
was read-only, so seeing a newer image and doing something about it were different places.
|
||
Each row now has **Update**, and the header has **Update all**. Updating is per stack —
|
||
that is the unit compose deals in — so a row acts on whichever stacks use that image, and
|
||
several stacks run through a queue one at a time rather than recreating half the machine at
|
||
once, each with its task output on screen. A row that succeeds says "updated just now"
|
||
instead of re-checking, since a check is a registry request.
|
||
- **Cron in English** (`crontext.ts`, 2026-09-26): schedules read "daily at 3:15am" rather
|
||
than `15 3 * * *`, with the expression kept beside it. Handles lists, ranges, steps, day
|
||
and month names, `@daily` and friends, weekdays/weekends, midnight and noon — and declines
|
||
to describe anything it is not sure of, because a confident wrong sentence is worse than
|
||
five fields. The Jobs tab had its own smaller version of this; it now uses this one.
|
||
|
||
- **Hardware tab** — **shipped 2026-09-25** (`internal/hardware`, `internal/api/hardware.go`,
|
||
`Hardware.tsx`): what the machine is physically made of, per machine like every other page.
|
||
Four headline tiles (processor, memory, storage, graphics) over cards for each: the
|
||
processor with its cores, clock range, socket, caches, microcode and the handful of feature
|
||
flags anyone asks about; **every memory slot** with size, type, form factor, rated and
|
||
running speed, maker, part number, serial, rank and voltage, and the empty ones marked
|
||
empty; each drive with model, size, bus, firmware and serial; each display adapter with its
|
||
driver, PCI address and video memory; and the board itself — model, firmware and date,
|
||
serial, kernel. Read from what Debian already has (dmidecode for what only the firmware
|
||
knows, lscpu, lspci, sysfs) and cached for ten minutes, since none of it changes under a
|
||
running kernel; "Re-read" is there for the one thing that does, a drive plugged in.
|
||
Every reader may come back empty and the page says what it could not read and why: a
|
||
machine with no dmidecode, no root, or no PCI bus still gets a page. Verified on `core`
|
||
(SMBIOS, Intel graphics) and on the Mac mini (Asahi: no SMBIOS memory table, a
|
||
device-tree GPU, heterogeneous cores — where lscpu's "cores per socket" counts one
|
||
cluster, so without SMT the thread count is the honest core count).
|
||
|
||
- **Sensors** — **shipped 2026-09-19** (`internal/sensors`, `internal/metrics/sensors.go`,
|
||
`internal/api/status_sensors.go`, `web/src/sensors.ts`, `widgets.tsx`, `Embed.tsx`;
|
||
reference in `SENSORS.md`): everything measured is a sensor with a stable dotted ID;
|
||
catalog, single value, history, a **shaped query** (the client sends the object it wants
|
||
and gets it back filled in — GraphQL's idea without GraphQL, since sensors differ per
|
||
machine and a static schema would collapse to `sensor(id:)`), the same query as a live
|
||
SSE stream, Prometheus `/metrics`, read-only `sensors:read` tokens (never full tokens in
|
||
a URL), status sensors (services, updates, ZFS, SMART), a widget library, custom widgets
|
||
on the Dashboard tab (briefly renamed "Monitors" 2026-09-19, back to "Dashboard"
|
||
2026-09-20 at Mitchell's request; the view id and URL were always `dashboard`), and a
|
||
frameable `/embed/widget` for Glance and friends. (An external demo page was deployed at
|
||
`hamlab-demo.bigham.dev` and linked from the Widgets tab 2026-09-19; **both removed
|
||
2026-09-23** at Mitchell's request — the Widgets tab shows the same thing against the
|
||
machine you are signed in to, so the second copy earned nothing and cost a public
|
||
read-only token, which was revoked with it. `examples/external-demo/` stays as an example
|
||
to read.) The stock Dashboard modules and the Disks
|
||
tab read the sensor API too (`sampleAdapter.ts`); `/api/metrics` is retired.
|
||
Also per-service and per-container sensors (memory and CPU from cgroups), tokens limited
|
||
to ID patterns, and `/api/fleet/sensors/query` for the whole fleet in one request.
|
||
- **Alerts** — **shipped 2026-09-19** (`internal/api/alerts.go`, `Alerts.tsx`): rules over
|
||
sensors (`container.*.running == false for 120 s`), evaluated by the hub every 15 s for
|
||
itself and every managed machine in one shaped query each; `hamlab.reachable` is supplied
|
||
by the hub so "a machine stopped answering" is a rule like any other. Ten defaults
|
||
(machine offline, container down/unhealthy, failed service, disk full, pool unhealthy,
|
||
SMART failing, UPS on battery, CPU hot, memory). Firing and resolved are events: kept
|
||
(last 200, in `state.json`), shown on the hub-only Alerts tab with a count on the rail,
|
||
and POSTed to channels — **ntfy** (title/priority headers) or a **webhook** (JSON; Home
|
||
Assistant, Slack). The rule editor shows what a selector matches across the fleet as
|
||
you type. Not done: quiet hours, per-rule channels, Web Push straight to the PWA.
|
||
- **Docker tab** — **shipped 2026-09-19** (`internal/api/docker.go`, `Docker.tsx`):
|
||
containers grouped by compose stack with live memory and CPU from the container
|
||
sensors, start / stop / restart (asked first), logs that follow (polled with `--since`,
|
||
no held connection), inspect (env *names* only), and images / volumes / networks
|
||
tables. Works on every machine through the hub; `#/grid/docker/immich_server` links to
|
||
one. **Stacks** (`internal/api/stacks.go`): every compose project, found from its
|
||
containers' labels, with its file to edit and Deploy (`up -d`), Update (pull, then
|
||
deploy), Restart and Stop as followed tasks. The file is looked for where the label says
|
||
*and through every container's mounts*, because an orchestrator that runs compose inside
|
||
its own container (Dockhand here; Dockge, Portainer) records a path in its own
|
||
filesystem; compose is then run **inside that container**, so `.env`, relative paths
|
||
and the compose labels stay exactly as deployed (verified: labels identical after a
|
||
redeploy). Save validates with `compose config` first, keeps the previous file as
|
||
`<file>.hamlab-bak`, and never deploys by itself. A stack deployed from another host
|
||
(Dockhand on `core` driving `grid` over tcp) has no file on that machine, so the node
|
||
alone lists it read-only — but **the hub fills that in** (`hubStacks`): asked about a
|
||
node, it resolves the files on itself and runs compose here with `DOCKER_HOST` pointed
|
||
at the node's `tcp://…:2375`, the way it was deployed; the task runs on the hub and the
|
||
node's page follows it. Also per stack: every file it was deployed from (chips), its
|
||
**`.env`** (values masked until "Reveal and edit"; validated in place, put back if
|
||
compose objects), **what Deploy would change** (a line diff against the copy kept at the
|
||
last deploy from here, `<file>.hamlab-deployed`, else the copy the last save replaced;
|
||
`.env` values never shown), and **image update checks** — asked for, never polled, since
|
||
each is a registry request Docker Hub counts; remembered six hours. A save refuses a
|
||
file changed on disk meanwhile.
|
||
**Making and unmaking (2026-09-20):** "+ New stack" — a name, a folder, compose text and
|
||
an optional `.env` (written root-only), checked by compose before it is kept, then
|
||
"Create and deploy" or "Create only". "+ New container" is the same dialog opened on a
|
||
form (image, ports, volumes, environment, restart, command, host networking) that writes
|
||
the compose for you: a container made here is a one-service stack, so it stays editable
|
||
and redeployable, where a bare `docker run` could only be thrown away and typed again.
|
||
New stacks go in a **home** — a folder where two or more stacks are already deployed from
|
||
(Dockhand's `stacks/core` here; `~/stacks` on a machine with none) — and are deployed the
|
||
way their neighbours were (inside `dockhand`, or over tcp to a node from the hub). Homes
|
||
are also scanned, so a stack with no containers ("not deployed": never deployed, or taken
|
||
down) is still listed. **Down** removes a stack's containers; **Delete…** (type its name)
|
||
takes it down and moves its own files to `<home>/.hamlab-trash/<name>-<time>/` — moved,
|
||
not erased; volumes and bind-mounted data are never touched. A stopped container can be
|
||
**removed** (a running one must be stopped first, on purpose). Images: **pull** (a followed
|
||
task), **remove** (unforced: Docker refuses one still in use) and **prune** (dangling only).
|
||
**Volumes (2026-09-20):** each says which containers use it; one that nothing uses can be
|
||
deleted — a named one by typing its name (it may be the data of a stack that is only
|
||
down), an anonymous one by confirming its full name — and "Prune anonymous" clears the
|
||
nameless leftovers of recreated containers, never named volumes. Images are marked "in
|
||
use", and "Remove unused" takes every image no container uses. Nothing is ever forced:
|
||
Docker refuses what is still in use, and that refusal is the answer. Still absent:
|
||
removing networks.
|
||
- **Top merged into Processes** — **2026-09-23.** The two tabs answered the same question
|
||
from the same data (Top already read `/api/procs`, the endpoint the Processes table was
|
||
built on), so at Mitchell's instruction there is now one tab, **Processes**, showing what
|
||
Top showed: the btop-style whole-machine screen. `Procs.tsx` is deleted; the `top` view
|
||
id and its route are gone. A rail order saved before this keeps its shape — `railorder.ts`
|
||
maps the retired `top` to `procs` and drops the duplicate, so the merged page sits where
|
||
Top sat rather than falling to the bottom.
|
||
- **Dashboard retired** — **2026-09-23**, at Mitchell's instruction, the same reasoning that
|
||
merged Top into Processes: the app repeated itself. The customisable module list is gone
|
||
(`Dashboard.tsx` deleted), and with it `/api/prefs/dashboard`, `applyPrefs`, and the
|
||
per-user `dashboard` blob in `state.json`. The Processes tab already shows those numbers
|
||
on one screen. `fmtBytes`/`fmtRate` moved to `fmt.ts` (Disks, Files and Space use them);
|
||
`MONITOR_HISTORY` and `seriesOf` went with the page. The default view after login, after
|
||
switching machines, and when a hub-only page is left behind, is now **Processes**. A saved
|
||
rail order naming `dashboard` degrades quietly — `orderBy` drops ids with no page.
|
||
**Widgets stays and says what it is for**: a notice at the top explains these are for
|
||
dashboards *outside* Ham Lab — Glance, Home Assistant, a wall display — served from
|
||
`/embed/widget` with a read-only sensor token, not a second copy of the console's screens.
|
||
**It also closed a real bug.** `applyPrefs` set the machine's sampling rate from
|
||
`intervalMs` inside the dashboard layout, and its `else` branch reset the sampler to the
|
||
default whenever that field was absent. Since layouts became per-user, one account saving
|
||
a dashboard could silently reset the sampling rate for the whole machine, overriding the
|
||
fleet refresh rate set in Settings. The sampler now has one source: `refreshMs` in the UI
|
||
prefs, which the hub fans out, remembered per machine as `sampler_ms`.
|
||
- **ZFS tab, with scheduled snapshots** — **shipped 2026-09-24** (`internal/storage/snapshots.go`,
|
||
`internal/api/zfs.go`, `ZFS.tsx`). Pools, datasets and snapshots moved out of Disks —
|
||
Disks is the hardware, ZFS is the filesystem on top of it — and the tab gained what it was
|
||
split out for: **snapshot schedules**. Hourly, daily, weekly or monthly, per dataset,
|
||
optionally recursive, with a **retention period** and a floor that keeps the newest few
|
||
whatever their age. A missed window (the machine was off) runs once on return rather than
|
||
being skipped.
|
||
**The naming is zfSnap's, deliberately.** `grid` has run zfSnap v1.11.1 from root cron at
|
||
02:30 with a 6-week TTL since before this repo existed, and it works. So rather than
|
||
invent a second convention for two tools to disagree over, this writes and reads exactly
|
||
what zfSnap does — `tank@2026-09-24_02.30.01--6w`, **retention encoded in the name**. Three
|
||
consequences: Ham Lab shows and expires the snapshots zfSnap already took; the two can run
|
||
side by side without fighting, since both only remove what has passed the expiry written
|
||
into its own name; and changing a schedule never re-dates old snapshots, because each
|
||
carries the rule it was taken under. The page **detects the existing cron job**
|
||
(zfSnap, zfs-auto-snapshot, sanoid, or a bare `zfs snapshot`) and says so, warning if a Ham
|
||
Lab schedule is running as well.
|
||
**Nothing is destroyed unseen.** Only names that parse as a stamp and a TTL are ever
|
||
candidates for expiry, so a snapshot taken by hand is invisible to a schedule; Run now
|
||
says that expired snapshots go with it; and a snapshot deleted by hand is confirmed first
|
||
and needs its full name echoed back by the caller. (A "what would it delete?" preview
|
||
shipped and was removed the same day as redundant — Mitchell's call.)
|
||
The datasets table is a **snapshot explorer**: each dataset shows how many snapshots it has
|
||
and what they cost, and opens into the snapshots themselves — when each was taken, **when
|
||
it expires** (read from the TTL in its own name, so no server call), its size, and Delete.
|
||
Deleting takes the same-named snapshot on descendant datasets too, which is how a recursive
|
||
schedule made it.
|
||
**A schedule is a real cron job** (2026-09-24, Mitchell's call after the first cut ran it
|
||
from a ticker inside the daemon). Ham Lab writes `/etc/cron.d/hamlab-zfs`, owns it, and
|
||
rewrites it whole on every change; each line calls `hamlab zfs-snapshot <id>` as the
|
||
account Ham Lab runs as, reaching zfs through sudo as the daemon does. Snapshots are
|
||
backup-shaped and must not stop because a console is down, upgrading, or uninstalled — and
|
||
a cron line is greppable by anyone looking at the machine the normal way. **A cron
|
||
expression is the single description of when a schedule fires**: the four common shapes
|
||
render to one, `custom` states one outright, and the next run is read back with the same
|
||
evaluator the Jobs tab uses on everyone else's lines, so the page, the file and the
|
||
prediction cannot disagree. A run's outcome is written to `zfs-runs/<id>.json` beside the
|
||
config, never to `state.json`, because the cron job is a separate process and the daemon
|
||
would clobber it. An existing external job can be **adopted** — its schedule, retention
|
||
and dataset read out of the cron line into a new Ham Lab schedule.
|
||
**zfSnap is retired on `grid`** as of that day: Ham Lab now takes the nightly `tank`
|
||
snapshot, the package is purged and the old cron line is gone, with the 515 snapshots it
|
||
had taken left untouched and fully readable here — which is what adopting its naming
|
||
bought. See `hamlet/docs/hosts.md`.
|
||
- **KVM tab** — **shipped 2026-09-24** (`internal/api/kvm.go`, `KVM.tsx`, `config.KVM`).
|
||
Paste a device's address; its own web page appears in the tab. **Nothing in it knows
|
||
anything about any particular KVM** — the first device here is a Sipeed NanoKVM on `grid`
|
||
at `10.0.0.217`, but none of that is in the code.
|
||
**Why it proxies rather than framing the device directly.** Ham Lab is served over TLS and
|
||
a KVM usually is not, so a bare `http://…` iframe is blocked as mixed content before it
|
||
loads. Serving the device from Ham Lab's own origin (`/kvm/<id>/…`) fixes that, puts it
|
||
behind **Ham Lab's** login rather than a second one, and makes the iframe same-origin.
|
||
**The general problem, and the answer.** A device UI asks for its assets by absolute path
|
||
— `/assets/app.js`, `/api/whatever` — which no path prefix can rewrite away, and rewriting
|
||
bodies per device would not be general. So the prefix is only the entry point: any request
|
||
Ham Lab does not serve itself is routed to whichever KVM the asking page belongs to, read
|
||
from the **Referer**, falling back to a **cookie** when there is none (a WebSocket upgrade).
|
||
Ham Lab's own routes always match first, so a device can never shadow the console, and the
|
||
fallback is gated on a live session. The device never receives Ham Lab's session cookie,
|
||
and its `X-Frame-Options`/CSP/HSTS are dropped on the way out since Ham Lab is the origin
|
||
now. Redirects to the device's root are rewritten back through the prefix.
|
||
**One catch found in testing:** Ham Lab's own `frame-ancestors 'none'` forbids framing even
|
||
from the same origin, so `/kvm/` paths send `'self'` instead. Without that the iframe was
|
||
refused before it loaded.
|
||
**The keyboard** needs less than it sounds: a page cannot forward keys into an iframe and
|
||
does not need to, since one with focus receives them directly. Fullscreen plus
|
||
`navigator.keyboard.lock()` is what takes Escape, Tab and Ctrl-W back from the browser;
|
||
that is Chromium-only, and the tab says which version you are getting rather than
|
||
pretending.
|
||
**One KVM per machine, and the tab follows the machine switcher** (corrected the same day:
|
||
the first cut was a hub-side list you pointed at machines, which had it backwards). Switch
|
||
to `grid` and you get grid's; adding a second for a machine replaces the first. The
|
||
addresses are still kept **on the hub** rather than on each machine, because a KVM earns
|
||
its keep exactly when the machine is down and its own Ham Lab cannot answer — the hub
|
||
proxies the device either way — and a KVM need not belong to a machine Ham Lab manages at
|
||
all. Not done: holding the device's own login on the hub, so its page does not ask for a
|
||
second password.
|
||
- **Skins** — 13 since 2026-09-19 (`themes.ts`: a palette is one `mk(…)` line): Dark,
|
||
Light, Cyberpunk, Retro, **HUD** (hairlines of cyan on near-black, corner ticks, thin
|
||
type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light;
|
||
each repaints the terminal too. **Each machine wears a colour** (`accents.ts`): a stripe
|
||
across the header and its name in the switcher — picked from ten swatches or any colour
|
||
on the Machines page, else derived from its name — saved with the theme on the hub.
|
||
- **Multi-host** — **Phase 1 shipped 2026-09-18** (`internal/nodes`, `internal/tlsauto`,
|
||
`MachineSwitcher.tsx`, `Machines.tsx`). One binary, hub and spoke: a host whose
|
||
`config.json` lists `nodes` is a hub; any other is a standalone node with its full UI.
|
||
Node side: `"tls": "auto"` makes a self-signed cert in the data dir and serves HTTPS;
|
||
`POST /api/tokens {password,name}` trades the login password for a long-lived bearer
|
||
token (SHA-256 stored in `tokens`), accepted by the auth middleware alongside the
|
||
cookie; `DELETE /api/tokens/{name}` revokes. Hub side: `POST /api/nodes {name,url,
|
||
password}` peeks the node's certificate (trust on first use, fingerprint pinned),
|
||
obtains a token named `hub:<hostname>`, stores `{name,url,fingerprint,token}`; a
|
||
15 s prober keeps each node's status; `/api/nodes/{name}/…` is a reverse proxy to the
|
||
node's `/api/…` (bearer swapped in, cookie and Origin stripped, streaming flush) that
|
||
carries WebSockets, so terminals, log tails and task output work remotely.
|
||
Frontend: `setMachine()` rewrites every `/api/…` path; the rail has a machine
|
||
switcher at the top and a Machines page to add/remove/open. Pages remount on switch;
|
||
terminals live on the node. Nodes: `grid`, `wyse`. **Phase 3 (update from hub)
|
||
shipped 2026-09-18**: node `POST /api/self-update` takes a binary, checks it answers
|
||
`version`, keeps `hamlab.prev` in the data dir, installs over the executable via
|
||
`sudo install`, replies, then restarts the service; the hub pushes its own executable
|
||
with `POST /api/machines/{name}/update` or `/api/machines-update-all` (architecture
|
||
checked first via `/api/me.arch`), the Machines page shows an **outdated** badge and
|
||
Update buttons, and `hamlab push-nodes` does the same from the CLI — `make deploy` runs
|
||
it last, so one deploy on the hub updates every machine. First install on a new host
|
||
is still by hand (scp binary + `install.sh`, `hamlab passwd`, `tls auto`). Still to
|
||
do: Phase 2 Datacenter overview; optional firewall rule limiting a node's port to
|
||
the hub. **Cross-architecture** (2026-09-18): `make deploy` also builds
|
||
`bin/hamlab-linux-arm64`, `install.sh` installs it beside the binary as
|
||
`hamlab-arm64`, and the hub picks the sibling matching a node's `/api/me.arch` when
|
||
pushing — the Mac mini (Fedora Asahi, aarch64, node `mac`) updates like the rest.
|
||
- **Portability + install** — **shipped 2026-09-18** (`internal/setup`, `internal/priv`,
|
||
`install.sh`, `make release`): the binary installs itself (`hamlab install` /
|
||
`uninstall`; systemd or OpenRC; root, sudo or doas), all builds static on amd64, arm64,
|
||
arm, 386 and riscv64, `install.sh` as the curl-able bootstrap with checksums. Every
|
||
privileged call goes through `priv` instead of a literal `sudo -n`. Updates tab acts on
|
||
dnf, zypper, pacman and apk as well as apt. Services list falls back to the plain
|
||
table on systemd < 246; Disks falls back to `/sys/block` without `lsblk`. Listener
|
||
uses `IP_FREEBIND`. Self-update: pre-flight `check` against the live config, atomic
|
||
replace, exec in place. Config default moved to `~/.config/hamlab` (existing
|
||
`~/apps/hamlab` still found by the installer and the CLI). Tested in
|
||
containers: Debian 12, Ubuntu 22.04, Fedora 40, Rocky 8, openSUSE Leap 15.5, Alpine
|
||
3.18/3.20 (OpenRC + doas, non-root), plus the four lab hosts. Not done: an OpenRC
|
||
backend for the Services tab; runit/s6/sysvinit service files; public hosting for
|
||
`install.sh` (the Gitea repo is private, so the one-liner needs `HAMLAB_TOKEN`).
|
||
- **Users, sessions and settings that follow the account** — **shipped 2026-09-22**
|
||
(`internal/api/users.go`, `config.User`, `store.Session`/`UserState`, `Login.tsx`,
|
||
`Users.tsx` — a hub-only **Users** tab in the rail, added the same day at Mitchell's
|
||
request, holds everything about accounts: your own password, your sessions, the
|
||
accounts; Settings keeps appearance, rates, tokens and About). **Rail order is per account** too (2026-09-22,
|
||
`railorder.ts`, `Rail.tsx`): "⇅ Arrange" at the foot of the rail puts a handle on every
|
||
page; drag one up or down (pointer events, so a finger works), let go and the order is
|
||
saved in the UI prefs with the theme; "Reset" restores the built-in order; pages added
|
||
in a later build go after the ones the saved order names. The login is a user name and a password. `config.json` holds `users`
|
||
(name, argon2id hash, created, last seen); the old single `password_hash` becomes the
|
||
account named after the OS user the console runs as on first boot, its signed-in
|
||
sessions become that account's, and the machine-wide theme/accents/dashboard become
|
||
theirs — nothing to redo after the upgrade. A session carries its user, an ID, when it
|
||
was made and last seen, the IP and the browser; **Users → Sessions** lists yours
|
||
("Safari on iPhone · 10.0.0.5 · seen 2 min ago"), revokes one, or signs out everywhere
|
||
else. **Users → Users** lists accounts with their session counts, adds one, removes
|
||
one (their sessions and settings go too; never yourself, never the last), and resets
|
||
another's password (they are signed out); removing and resetting take your own
|
||
password first. Everyone signed in can do everything — a household console, not a
|
||
tenancy — and each machine keeps its own accounts: a hub reaches a node with its token,
|
||
so pairing (Machines → Add) now takes a user and password on the node, the user
|
||
optional while it has one account. Per account: `ui` (theme, accents) and `dashboard`
|
||
in `state.json` under `users.<name>`. Still the machine's: refresh rate (it is how
|
||
often every machine samples — it rides in the UI prefs object for the client but is
|
||
stored once, machine-wide), graph history, alert rules, terminals (the shells run as
|
||
one OS user anyway). A login attempt for a missing user takes as long as a wrong
|
||
password, so the form can't list accounts. Every write's log line now names the
|
||
account or token (`core@10.0.0.5 via …`), and Access tokens say who issued them. CLI:
|
||
`hamlab passwd [user]` (makes the account if new; reads stdin when piped), `hamlab
|
||
users`, `hamlab user-rm <user>`; `hamlab install` names the first account after the
|
||
service user. Not done: a read-only viewer role.
|
||
- **Space tab** — **shipped 2026-09-22** (`internal/space`, `internal/api/space.go`,
|
||
`Space.tsx`), asked for as "exactly WizTree": what is using a disk. The unit is a whole
|
||
volume, picked from a dropdown at the top: a **disk on its own** (every filesystem its
|
||
partitions back) or a **ZFS pool as a whole** (every mounted dataset) — a pool's member
|
||
disks are listed under it with the layout (raidz2, 8 disks: sda …) and never offered
|
||
apart, since one disk's share of a pool means nothing (Mitchell's call, same day, after a
|
||
first cut that listed the disks as chips). The walk runs as a helper process, `hamlab space-walk <roots>`,
|
||
as root through `priv` when the host allows (so `/var/lib/docker` and `/root` count), 32
|
||
workers reading directories depth-first, never crossing a filesystem boundary by itself;
|
||
each root is grafted into one tree at its path. Files under 128 KiB are not kept one by one
|
||
but summed per folder as "N smaller files" — what keeps a scan of millions of files to tens
|
||
of MB in the daemon; the tree crosses back as a varint stream (`codec.go`), and the helper's
|
||
memory goes with it. core's root: 646k files in ~1.2 s. Progress (files, bytes, current
|
||
folder) every 250 ms; Cancel sends SIGTERM. The page: picker → Scan/Rescan → **Folders**
|
||
(a tree largest-first: size, share of parent as a bar, files, folders, modified; expand in
|
||
place) over a **treemap** of the selected folder — squarified, nested to the file, tiles
|
||
coloured by kind (video, images, audio, archives & disk images, documents, code & config,
|
||
databases & logs, other; the validated categorical set, dark and light) with folders as
|
||
labelled boxes; hover for path, size and share; click a folder to drill, breadcrumbs and ↑
|
||
to come back. The map asks the server for the folder pruned to what a tile could show
|
||
(`min` = 1/6000 of the folder, 24 deep; deeper folders come back as one lump), the list
|
||
one level at a time. **Largest files**: the 1,000 biggest with a path filter. Works on
|
||
every machine through the hub.
|
||
**Smarter and faster (2026-09-23, at Mitchell's request):** a scan is *kept* — in memory
|
||
and in `<data>/space/<target>.scan` (the same varint stream) + `.json`, read back after a
|
||
restart — and the next scan *starts from it*: the helper is handed the last tree on stdin
|
||
and reads a directory again only when something says it changed. On a ZFS pool with
|
||
snapshots, the scan records the newest snapshot of each dataset at start (`bases`), and the
|
||
next one runs `zfs diff -H -F <base>` per dataset: only the directories it names are
|
||
read, their ancestors kept but looked into, everything else kept wholesale without a
|
||
stat. The diffs run side by side and one is given three minutes — `timeout -s KILL`
|
||
inside sudo, since Debian's sudo gives the command its own session and a process-group
|
||
kill from outside never reached zfs — after which that dataset is walked by mtime
|
||
instead. On `tank` a diff is metadata-bound like the walk: 8 s to 3 min per dataset for
|
||
a handful of changes, and `immich`, which changes constantly, ran past twenty minutes
|
||
the first time. A refresh of `tank` took 4 min 19 s against 7 min for the full scan. Elsewhere (ext4, or a dataset with no snapshot) a directory whose mtime — to the
|
||
nanosecond — is unchanged keeps its list and only its kept files are stat'ed again, since
|
||
a file can grow without its directory noticing. A **cancelled** scan keeps what it walked,
|
||
marks the folders it never got to (`partial`, "not yet read" in the tree) and **Resume**
|
||
reads just those. core: full 1.2 s → refresh 0.3 s (2 folders read, 84,755 kept). The
|
||
status line says which (`method`, `by_root`, read/kept counts); **Forget** drops the saved
|
||
result so the next scan is full. The old result stays browsable while a refresh runs.
|
||
**Treemap rethought** the same day: every file a tile grouped by folder — folders are only
|
||
a thin line and a name strip on the first level — instead of nested boxes with headers,
|
||
which on Linux's single-child folder chains was a staircase of headers around nothing.
|
||
Not done: delete from here, a size-over-time history, per-user column choices.
|
||
- **Updater** — **shipped 2026-09-23** (`internal/selfupdate`, `internal/api/update.go`,
|
||
the Updates card in `Settings.tsx`, `scripts/gitea-release.sh`). A Ham Lab asks one URL
|
||
what the newest build is and installs it on request. `update.url` in `config.json` is
|
||
either a Gitea/Forgejo repo API base (`…/api/v1/repos/<owner>/<repo>`, whose newest
|
||
release names the version and carries the binaries as assets) or any static directory
|
||
holding `VERSION`, `SHA256SUMS` and `hamlab-linux-<arch>` — which is exactly what
|
||
`make release` writes and what `install.sh` already downloads, so the demo site, a
|
||
folder on a NAS and a Gitea release are all the same thing to it. `"off"` never looks.
|
||
`update.token` is sent when the source is behind a login, so a **private** Gitea serves
|
||
its own fleet without the instance being opened to the world. The binary is verified
|
||
against `SHA256SUMS` before it goes anywhere near the running one, then takes the same
|
||
path `POST /api/self-update` always did (runs `version`, runs `check` against this
|
||
machine's config, atomic replace keeping `hamlab.prev`, re-exec in place) — that path is
|
||
now one shared method, `installBinary`. Version ordering is `install.sh`'s `vnum()` rule
|
||
ported to Go and tested against it, so the daemon and the installer never disagree about
|
||
which of two builds is newer; the commit count since the tag counts, the hash does not.
|
||
Checked ~90 s after boot and daily (`update.check: "never"` switches that off), never
|
||
installed unasked — an update restarts the daemon and drops open terminals, and the
|
||
dialog says so. Settings → **Updates**: installed, published, when it last looked, the
|
||
source, the release notes, Check for updates, Install; the rail badges Settings when a
|
||
newer build is out (`update_available` rides on `/api/me`). Publishing is
|
||
`make gitea-release` → `scripts/gitea-release.sh`, which creates the release and uploads
|
||
the five architectures, `SHA256SUMS`, `install.sh` and `VERSION`; it is deliberately
|
||
**not** part of `make deploy`, because deploy pushes this build to the machines this hub
|
||
owns while publishing offers it to every install anywhere, and one tag per deploy would
|
||
be noise. Re-publishing a version needs `--force`.
|
||
**Public since 2026-09-23**, at Mitchell's instruction and after the trade was put to him
|
||
twice: Gitea's instance-wide `REQUIRE_SIGNIN_VIEW` is off and `mitchell/hamlab` is marked
|
||
public, so anyone can browse it, clone it and download a release with no account. The
|
||
other five repos there are each marked private and are invisible — an anonymous request
|
||
for one is a 404, not a login prompt — registration is still disabled, and new repos
|
||
still default to private. No token is needed by anything any more (the read-only one was
|
||
revoked; only `hamlab-publish`, which creates releases, remains). Gitea resolves
|
||
`releases/download/latest/<asset>`, so `install.sh` and `HAMLAB_URL` now point straight
|
||
at the newest release and never need changing. What is public is the source, the history,
|
||
`BUGS.md` and this file — which describes the lab and names the console's hostname.
|
||
- **Auth hardening** — WebAuthn passkeys (phone Face ID), TOTP fallback, optional NPM
|
||
access list restricting to WireGuard/LAN.
|
||
- **Files tab** — **shipped 2026-09-18** (`internal/files`, `Files.tsx`): browse from
|
||
`/` with breadcrumbs, an editable path box, sort, hidden-file toggle and a filter;
|
||
rows show type icon, size, modified, perms and owner, link targets and broken links.
|
||
Text files (≤ 2 MiB, sniffed) open in an editor with Ctrl+S / Save; images preview;
|
||
everything else downloads. Upload via drag-and-drop or picker with per-file progress
|
||
(streamed, no size limit). New folder / file, rename, move, chmod, copy path, delete
|
||
(folders need the name typed back), and "open in terminal", which `cd`s the active
|
||
shell (or a fresh one) to the folder. **Sudo toggle**: root access re-executes the
|
||
same binary as `hamlab fs <op>` under `sudo -n`, streaming bodies over stdin/stdout,
|
||
so both modes share one implementation and one set of rules (absolute paths only;
|
||
`/`, `/etc`, `/usr`, … are never deleted or moved whole). Every write is logged with
|
||
the caller's IP.
|
||
- **Docker read-only view** — `docker ps`/logs; dockhand stays the source of truth.
|
||
- **Notifications** — push on service failure / SMART warning / UPS event.
|
||
- **Terminal extras** — search, copy-mode, font/theme, split panes, upload via
|
||
drag-drop into cwd.
|
||
|
||
- **Settings + themes** — **shipped 2026-09-18** (`Settings.tsx`, `themes.ts`): Settings
|
||
page with Appearance (Dark, Light, Cyberpunk, Retro — CSS variables on `<html>` plus a
|
||
matching xterm palette, applied before first paint from localStorage and synced hub-side
|
||
via `GET/PUT /api/prefs/ui` so the choice follows the login), Account (change password,
|
||
`POST /api/password`, current password required), Access tokens (list/revoke the tokens
|
||
hubs hold), About (version, host, arch, uptime, certificate).
|
||
|
||
## To do
|
||
|
||
- **Grow the Files tab until copyparty can go** — asked for 2026-09-23 as two items (Files
|
||
polish, and deprecating copyparty), merged 2026-09-24: they are the same tab and the same
|
||
work. `copyparty` on `grid` (`copyparty.service`, `:3923`, `/etc/copyparty.conf`, serving
|
||
all of `/tank` as user `core` with one account behind NPM) overlaps **Files** almost
|
||
entirely. The aim is to stop the service and keep only Ham Lab, with general ease-of-use
|
||
polish along the way. What Files must gain first, in the order that matters:
|
||
- **Multi-select and bulk actions.** copyparty selects many rows and moves, deletes or
|
||
downloads them together; Files acts on one row at a time.
|
||
- **Download a folder**, and a multi-file download, as a streamed zip or tar — the thing
|
||
copyparty is reached for most.
|
||
- **Resumable uploads.** copyparty's up2k chunks a file, survives a dropped connection
|
||
and skips what is already there; Files streams one PUT, so a phone upload that drops at
|
||
90% starts over.
|
||
- **A gallery.** Thumbnails for a folder of images and video, and play audio/video in
|
||
place — `/tank/media` and `/tank/immich` are most of what is browsed.
|
||
- **Search** below the current folder by name, since `find` in a terminal is the fallback
|
||
today.
|
||
- *Decide when built:* whether share links (copyparty's public per-path links) are wanted
|
||
at all — they are the one feature that hands out access without a login, and Ham Lab has
|
||
no notion of that; if not, say so in the docs rather than leaving a gap unexplained.
|
||
- *After:* `systemctl disable --now copyparty`, remove its NPM host, keep
|
||
`/etc/copyparty.conf` until a rollback is clearly not needed, then update
|
||
`hamlet`'s `docs/hosts.md` bare-metal list, `docs/gotchas.md` (the "`docker ps` is not
|
||
the inventory" note names it) and the `grid` row in `CLAUDE.md`.
|
||
|
||
- **Grow the Docker tab until Dockhand can go** — asked for 2026-09-23 and widened
|
||
2026-09-24 to take in Docker tab polish and feature work, on the same reasoning as the
|
||
Files item: it is one tab, and the gaps that block retiring the old tool and the things
|
||
that would simply make it better are the same body of work. The *homelab* half —
|
||
the cutover itself, then moving every stack off Dockhand's data dir — is `hamlet`'s
|
||
`docs/pending.md` → "Retire Dockhand; Ham Lab becomes the orchestrator", agreed
|
||
2026-09-20. This entry is the *app* half.
|
||
|
||
**Done 2026-09-24 — the eight Mitchell picked from the inventory**, all in the Docker tab:
|
||
per-container **Stats**, **Health** (with the last few probe outputs), **Events** (Ham Lab
|
||
keeps its own 4000-entry ring, since `docker events` forgets after about a minute),
|
||
**Shell** (`docker exec`, handed to the Terminal tab) and **Limits** (`docker update` for
|
||
memory, CPU and restart policy); an **updates** tab giving the whole machine at once with
|
||
a per-container indicator in the list; and **schedules**, which write
|
||
`/etc/cron.d/hamlab-docker` and call
|
||
`hamlab docker-update <id>` — the shape the ZFS schedules already use.
|
||
Also done: Ham Lab keeps working when `dockhand` is removed. It remembers where a
|
||
container's bind mounts pointed (`mounts` in `state.json`) and resolves a stack through
|
||
that record when no running container explains its path, running compose on the host
|
||
instead; `repoint` does the same for a schedule at run time. Without this, removing the
|
||
container would have made 17 stacks on `core` and 5 on `grid` unfindable at once.
|
||
|
||
- **Left from the Dockhand inventory, both to decide rather than build.**
|
||
- *Git-sourced stacks.* Dockhand's `stack_sources` table points a stack at a repo and
|
||
pulls it. Its `git_repositories` and `git_stacks` tables are **empty**, so nothing
|
||
here uses it — dropping it is the honest answer unless Mitchell says otherwise.
|
||
- *Webhooks, OIDC, LDAP, vulnerability scans.* All four tables empty. Nothing to move.
|
||
|
||
**Done 2026-09-25 — the two gaps in the tab itself.**
|
||
- *Logs are followed over a held connection*, the way the Services tab follows a journal
|
||
(`internal/api/docker_logs.go`, `/api/docker/containers/{id}/logs/tail`): one
|
||
`docker logs -f` per viewer, stdout and stderr both, stderr coloured as an error. A
|
||
reconnect resumes from the last line's timestamp, so a container that restarts or a
|
||
phone that slept doesn't replay what is already on screen; the client dedupes by line
|
||
rather than by "later than", because `--since` is inclusive and two lines can share a
|
||
timestamp. Docker replays the opening backlog down two pipes in no particular order, so
|
||
the first 300 ms is held and sorted before it is sent. The polled `?since=` endpoint
|
||
stays — it is what the resume is built on. Reconnects back off to 15 s, since a
|
||
container that has stopped ends the stream immediately.
|
||
- *Networks can be removed and pruned* — `POST /api/docker/networks/{remove,prune}`, the
|
||
shape images and volumes already had. The overview says who is attached to each one, so
|
||
the ✕ is only offered on a network nothing uses and never on Docker's own three. Prune
|
||
is deliberately untested against this host: `core` has several networks belonging to
|
||
stacks that are merely down (`pihole_default`, `transmission_default`, `gluetun_default`
|
||
among them), and pruning would take them and hand them new subnets on the next deploy.
|
||
|
||
- **The `*:2375` question, which this is the moment to settle.** A stack deployed to
|
||
`grid` or the Mac still goes over `tcp://<node>:2375` from the hub. Decide whether the
|
||
hub should instead ask the node's own Ham Lab to deploy locally, which would end the
|
||
unauthenticated Docker socket exposure — its own item in `docs/pending.md`, currently
|
||
filed under "decided, no action", and this is the change that would make closing it
|
||
free.
|
||
|
||
- **A Virtualization tab** — asked for 2026-09-23, not started, and **large**. Run and
|
||
manage **LXC containers and virtual machines** from the console, the way Proxmox does —
|
||
which is the comparison Ham Lab has invited since its first line. Nothing exists yet.
|
||
What a future session has to settle before writing code: which stack (libvirt/KVM +
|
||
`virsh` is the obvious one for VMs; LXC or Incus for containers, and Incus covers both);
|
||
whether it manages what is already installed or installs it; where disk images live
|
||
(`tank` on `grid`, local NVMe elsewhere); console access (SPICE/VNC proxied through the
|
||
daemon, or a serial console over the existing terminal machinery, which would reuse
|
||
`internal/term`); snapshots and clones; and what it does on a host with no virtualisation
|
||
at all, which is every host here today. Worth scoping as its own plan before starting.
|
||
|
||
|
||
## Open questions
|
||
|
||
- Idle-session reaping policy (never / after N days) — leaning never; it's a
|
||
single-user box.
|
||
- Whether `sudo -i`-style root sessions get their own leaf colour. Cosmetic.
|