mitchell/hamlab
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 14 Wiki Activity
Files
e772f7fae2d87deca7a40a628df7698be619d1f7
hamlab/PLAN.md
T

64 KiB
Raw Blame History

Ham Lab — MVP plan

Proxmox-style web console for a bare-metal Debian 13 host. First target: core (10.0.0.234). MVP is service + shell + persistent terminals; everything else is a stub tab or a roadmap item.

Decisions

Area Decision Why
Backend Go 1.24, single static binary, frontend embedded via embed core has ~370 MiB free RAM; a Go daemon idles at ~15 MiB vs ~80+ for Node. One file to deploy, no nvm path in the unit. Same binary later installs on grid unchanged.
PTY creack/pty — server-owned PTYs running core's login shell Sessions live in the daemon, not the browser, so refresh/tab-switch reattaches. No tmux dependency (not installed on core).
Transport One WebSocket per attached terminal; REST for everything else Standard; NPM passes WebSockets with one toggle.
Frontend React + TypeScript + Vite, xterm.js (+ fit, webgl, web-links addons) Matches calorie-counter / wedding-website, so no new toolchain to learn.
Auth Single-user password login, argon2id hash, HttpOnly/Secure/SameSite=Strict session cookie, login rate-limit, Origin check on WS upgrade This is a root-capable shell on the public internet. Auth is MVP, not roadmap.
Service hamlab.service, User=core, Restart=always, binary at /usr/local/bin/hamlab Same pattern as calorie-counter.service. Runs as core so sudo works in the terminal without a password, as on the SSH path.
Listen 10.0.0.234:8100 (free) NPM is a container, so loopback-only won't do. Firewalling to NPM is a later hardening step.
Data /home/core/apps/hamlab/ — config.yaml, state.db (SQLite) apps/ is the nightly rsync target to grid, so it's backed up for free.
Ingress NPM proxy host hamlab.bigham.dev → :8100, LE cert, WebSockets on, caching off, force-SSL + HSTS + HTTP/2 set in a second API call (NPM drops them on create-with-new-cert) *.bigham.dev already resolves; the proxy host alone makes it live.
Repo Gitea mitchell/hamlab, private, main Created 2026-09-17.
Guest logins A guest is an account like any other, with guest: true and an expiry: login, sessions and prefs need no special case, and it may do everything an account may do. The expiry is the whole feature — checked on every request and swept once a minute — with revoking as the manual version of it. Built 2026-09-27. A read-only version was built first and cut the same day: Mitchell hands these to people he trusts and revokes them otherwise, so a console with half its tabs missing was solving a problem he doesn't have. Consequences, stated once and accepted: a guest has the Terminal tab, which is a root-capable shell on every host, and can make an account or token of its own that outlives the guest login. The one thing a guest is not is an account: it is left out of the Users list, out of "the last account can't be removed", and out of the single account an empty user name resolves to.
Machines A view shows one machine: the one in the switcher. No hub-side page lists or aggregates what its nodes are doing. To see or change anything on another machine, switch to it. Mitchell, 2026-09-25, cutting a fleet-wide updates list out of the Docker tab. The switcher is the only place a machine is chosen, so there is exactly one answer to "which machine am I looking at" — a page that quietly reaches across breaks that, and it is never obvious which machine an action would land on. The hub still proxies for a node, and hub-only admin (Machines, Users, Settings, Alerts) is unaffected.
Capability, not policy A tab offers what the selected machine can do, detected live — never what a rule says should run there. Where the software is missing but installable, Ham Lab installs it; where it cannot be installed or the hardware lacks it, the tab says so in a sentence and offers what does work. Every action lands on the machine in the switcher. Mitchell, 2026-10-09, after the Virtualization plan's first draft excluded the Mac mini and hamtv on the strength of hamlet/CLAUDE.md's deployment rules: "I want the option to do it anywhere I want, unless the hardware / software literally doesn't support it. Ham Lab needs to be built in a general-purpose way that can run on any Linux system." Those rules bind the agent's unasked deployments, not the product.

Layout

┌──────────────────────────────────────────────────────────────┐
│ header: Ham Lab · core · load 0.42 · ● connected · [logout]    │
├──────────────┬───────────────────────────────────────────────┤
│ Dashboard    │ shell 1 ● │ backups │ +                      │
│ Processes    ├───────────────────────────────────────────────┤
│ Terminal     │                                               │
│ Services     │        main content (xterm)                   │
│ Cron         │                                               │
│ …            │                                               │
├──────────────┴───────────────────────────────────────────────┤
│ footer: v0.1.0 · up 12d · hamlab.bigham.dev                   │
└──────────────────────────────────────────────────────────────┘
  • Left rail: flat list of pages, active one highlighted. No nested leaves.
  • Terminal page: tab strip across the top of the pane, one tab per session, ● when a process is running, + pinned at the right edge. Active tab carries rename and close; double-click renames, middle-click closes. (Until 2026-09-18 the sessions were leaves under a collapsible Terminal group in the rail.)
  • Non-MVP groups render as disabled leaves with "coming soon" — the rail shape is the product; stubs keep it honest about scope.
  • Mobile: rail becomes a slide-in drawer; xterm fits the viewport; a sticky key bar above the keyboard (Esc Tab Ctrl Alt ↑ ↓ ← → |) because phone keyboards have none of those. This is what makes it usable as a PWA rather than a novelty.

Terminal semantics

  • A session is a PTY + shell owned by the daemon, with an ID, name, cwd hint, and a scrollback ring buffer (256 KiB). Attaching replays the buffer, then streams.
  • Any number of browser tabs can attach to the same session (all get output; all can type). Resize follows the most recent attacher.
  • Switching tabs in the UI just detaches/attaches; the shell keeps running.
  • Closing a session sends SIGHUP to the process group and drops it. Shell exit closes the session automatically (leaf shows "exited" until dismissed).
  • Session names persist in state.db so the sidebar order survives a service restart. The shells themselves do not survive a daemon restart in the MVP — see roadmap (tmux backing).
  • Idle sessions cost one shell process each; no automatic reaping in MVP.

API (MVP)

POST /api/login            {password}            → 204 + cookie
POST /api/logout                                 → 204
GET  /api/me                                     → {host, user, version, uptime, load}
GET  /api/terminals                              → [{id, name, running, created}]
POST /api/terminals        {name?}               → {id, ...}
PATCH /api/terminals/:id   {name}                → 204
DELETE /api/terminals/:id                        → 204
WS   /api/terminals/:id/attach   binary frames = raw bytes; JSON text frames = {resize}
GET  /healthz                                    → 200 (unauthenticated, for NPM)

Everything under /api except login and /healthz requires the session cookie.

Project layout

hamlab/
  cmd/hamlab/main.go          serve | passwd | version
  internal/
    auth/                     argon2id, sessions, rate limit
    term/                     PTY manager, ring buffer, attach/detach
    api/                      chi router, handlers, WS
    state/                    SQLite (modernc, no cgo)
    web/                      go:embed of web/dist
  web/                        Vite + React + TS + xterm.js, PWA manifest + SW
  deploy/
    hamlab.service
    install.sh                build → /usr/local/bin/hamlab → systemctl restart
  Makefile                    build / dev / deploy / lint / test

Milestones

M0 — skeleton (service boots, login works)

  • Go module, chi, config loading, hamlab passwd, argon2id login, session cookie.
  • hamlab.service on core, :8100, serves a placeholder page.
  • make deploy from core itself.

M1 — terminals (the MVP feature)

  • PTY manager with ring buffer, create/list/rename/close, WS attach with resize.
  • React shell: header, rail, footer, Terminal group, xterm main pane.
  • Persistence of session list across refresh and rail switching.

M2 — polish + PWA + ingress

  • Mobile drawer, key bar, fit-on-rotate.
  • manifest.webmanifest, icons (Ham logo on black, same recipe as HamTV), service worker caching the app shell only (never /api).
  • NPM proxy host for hamlab.bigham.dev; verify WS through NPM; force-SSL/HSTS/HTTP2.
  • Stub groups for Services / Cron / ZFS / Dashboard.

Done when: from the phone, open the PWA, log in, open two terminals, run something long in one, switch away and back, refresh — it's all still there.

Roadmap (not MVP — notes only)

  • Updates tab — shipped 2026-09-17 (internal/updates, internal/tasks, Updates.tsx): upgradable packages with current → candidate, suite, security badge and description; index age, last upgrade (from /var/log/apt/history.log), unattended-upgrades on/off, reboot-required banner with the packages that caused it; Refresh index, Upgrade all (dist-upgrade), Upgrade selected; changelog per package on demand. Runs go through the new tasks package: background command, output kept, followed live over /api/tasks/{id}/stream, cancellable, listed at /api/tasks (the seed of a Task History tab). apt is fully supported; dnf/pacman list only.

  • Power — header ⏻ button (amber when a reboot is pending), dialog that requires the hostname typed back, then systemctl reboot / poweroff. /api/me carries reboot_needed so the header knows.

  • tmux-backed sessions so shells survive a daemon restart / redeploy. Would spawn tmux new -A -s hamlab-<id>; needs mouse on and status-bar handling in xterm.

  • Services tab — systemctl list-units, per-unit page: status, last 200 journal lines (live-tail via WS), start/stop/restart/enable buttons, unit file viewer.

  • Cron tab — shipped 2026-09-17 (internal/cron, Cron.tsx): one list built from /etc/crontab, /etc/cron.d/*, every user crontab (via sudo), the scripts in the run-parts directories those drive, plus systemd timers. Next run computed by an in-house cron-expression evaluator (tested: ranges, steps, names, dom-OR-dow rule, @ aliases); last run from the journal's (user) CMD (command) lines (cron logs via syslog to journald; reading needs sudo/adm). Commented-out jobs show as disabled; job-shaped lines inside comment blocks (Debian's header examples) are ignored. Per job: Run now (background, as the job's user, sh -c with crontab % rules, output captured), Enable/Disable (comments the line in place, refuses if the file changed underneath), Edit source (textarea; validated line-by-line; user crontabs installed with crontab -, system files written via sudo tee), Cron log, and Output file when the command redirects to one. Timers: Run now (starts the activated service), start/stop/enable/disable, live logs of the service via the Services tab's tail.

  • Disks tab — shipped 2026-09-17 (internal/storage, Disks.tsx), replacing the planned ZFS tab: block devices from lsblk -J (kind, model, size, temperature from the device's hwmon node, partitions with filesystem usage, live read/write/busy from the metrics sampler); SMART via smartctl -j when smartmontools is present (one button installs it as a task otherwise): health, temperature, power-on time, self-test status, ATA attribute table with the failure-predicting IDs flagged, NVMe endurance and error counters, short/long self-tests. Spinning disks are read with -n standby so the tab never wakes a sleeping drive; "Wake and read" is explicit. ZFS section when the tools and module are there: pools with health, capacity, fragmentation, last scan and vdev tree; scrub start/pause/stop; datasets tree with used/refer/in-snapshots/ snapshot count/compressratio/mount; per-dataset snapshot list and "Snapshot now". Verified against grid (raidz2 tank, 10 datasets, 515 snapshots, 8 WD40EZAZ + 970 EVO Plus SMART). Snapshot destroy and dataset property editing are deliberately not there yet.

  • Dashboard — first cut shipped 2026-09-17 (internal/metrics, Dashboard.tsx): CPU (total + per core), memory/swap, load, uptime, hwmon temps and fans read straight from /sys (no lm-sensors), per-interface network rates, per-disk read/write/busy from /proc/diskstats, filesystem usage, UPS via sudo -n pwrstat -status. Sampled every 2 s into an in-memory ring of 10 min; the UI (the Dashboard tab since 2026-09-19) reads it all through the sensor API — see Sensors below. Customizable: a Customize mode reorders and hides modules (CPU, Memory, Load, Uptime, Temperatures, Fans, Network, Disks, Filesystems, UPS) and hides individual rows (an interface, a mount, a disk, a sensor); the layout is stored server-side in state.json via GET/PUT /api/prefs/dashboard so every device sees the same dashboard. The update interval (100 ms – 2 s) is part of the layout and drives both the browser's polling and the daemon's sampling clock; the chart window is always 300 samples, so 30 s at 100 ms and 10 min at 2 s. Still to do: persistent history (SQLite ring, e.g. 10 s × 7 days), SMART (smartctl -j), longer time ranges.

  • Updating from the Updates tab — shipped 2026-09-26: the "what is out of date" list was read-only, so seeing a newer image and doing something about it were different places. Each row now has Update, and the header has Update all. Updating is per stack — that is the unit compose deals in — so a row acts on whichever stacks use that image, and several stacks run through a queue one at a time rather than recreating half the machine at once, each with its task output on screen. A row that succeeds says "updated just now" instead of re-checking, since a check is a registry request.

  • Recreate, and the Updates tab at full width (2026-09-26): a stack's actions gained Recreate (up -d --force-recreate), the only way out of a container Docker refuses to start — one pinned to a network that has since been recreated answers "network … not found" to Deploy and Update alike, because as far as compose is concerned nothing about it changed. Found the hard way when a SiYuan update landed in exactly that state. The Updates tab also stopped rendering as a list-and-detail pair: it is a page, not a list, so on a wide window it now sits under the tab strip at full width instead of being squeezed into the right-hand half beside an empty column.

  • Cron in English (crontext.ts, 2026-09-26): schedules read "daily at 3:15am" rather than 15 3 * * *, with the expression kept beside it. Handles lists, ranges, steps, day and month names, @daily and friends, weekdays/weekends, midnight and noon — and declines to describe anything it is not sure of, because a confident wrong sentence is worse than five fields. The Jobs tab had its own smaller version of this; it now uses this one.

  • Hardware tab — shipped 2026-09-25 (internal/hardware, internal/api/hardware.go, Hardware.tsx): what the machine is physically made of, per machine like every other page. Four headline tiles (processor, memory, storage, graphics) over cards for each: the processor with its cores, clock range, socket, caches, microcode and the handful of feature flags anyone asks about; every memory slot with size, type, form factor, rated and running speed, maker, part number, serial, rank and voltage, and the empty ones marked empty; each drive with model, size, bus, firmware and serial; each display adapter with its driver, PCI address and video memory; and the board itself — model, firmware and date, serial, kernel. Read from what Debian already has (dmidecode for what only the firmware knows, lscpu, lspci, sysfs) and cached for ten minutes, since none of it changes under a running kernel; "Re-read" is there for the one thing that does, a drive plugged in. Every reader may come back empty and the page says what it could not read and why: a machine with no dmidecode, no root, or no PCI bus still gets a page. Verified on core (SMBIOS, Intel graphics) and on the Mac mini (Asahi: no SMBIOS memory table, a device-tree GPU, heterogeneous cores — where lscpu's "cores per socket" counts one cluster, so without SMT the thread count is the honest core count).

  • Sensors — shipped 2026-09-19 (internal/sensors, internal/metrics/sensors.go, internal/api/status_sensors.go, web/src/sensors.ts, widgets.tsx, Embed.tsx; reference in SENSORS.md): everything measured is a sensor with a stable dotted ID; catalog, single value, history, a shaped query (the client sends the object it wants and gets it back filled in — GraphQL's idea without GraphQL, since sensors differ per machine and a static schema would collapse to sensor(id:)), the same query as a live SSE stream, Prometheus /metrics, read-only sensors:read tokens (never full tokens in a URL), status sensors (services, updates, ZFS, SMART), a widget library, custom widgets on the Dashboard tab (briefly renamed "Monitors" 2026-09-19, back to "Dashboard" 2026-09-20 at Mitchell's request; the view id and URL were always dashboard), and a frameable /embed/widget for Glance and friends. (An external demo page was deployed at hamlab-demo.bigham.dev and linked from the Widgets tab 2026-09-19; both removed 2026-09-23 at Mitchell's request — the Widgets tab shows the same thing against the machine you are signed in to, so the second copy earned nothing and cost a public read-only token, which was revoked with it. examples/external-demo/ stays as an example to read.) The stock Dashboard modules and the Disks tab read the sensor API too (sampleAdapter.ts); /api/metrics is retired. Also per-service and per-container sensors (memory and CPU from cgroups), tokens limited to ID patterns, and /api/fleet/sensors/query for the whole fleet in one request.

  • Alerts — shipped 2026-09-19 (internal/api/alerts.go, Alerts.tsx): rules over sensors (container.*.running == false for 120 s), evaluated by the hub every 15 s for itself and every managed machine in one shaped query each; hamlab.reachable is supplied by the hub so "a machine stopped answering" is a rule like any other. Ten defaults (machine offline, container down/unhealthy, failed service, disk full, pool unhealthy, SMART failing, UPS on battery, CPU hot, memory). Firing and resolved are events: kept (last 200, in state.json), shown on the hub-only Alerts tab with a count on the rail, and POSTed to channels — ntfy (title/priority headers) or a webhook (JSON; Home Assistant, Slack). The rule editor shows what a selector matches across the fleet as you type. Not done: quiet hours, per-rule channels, Web Push straight to the PWA.

  • Docker tab — shipped 2026-09-19 (internal/api/docker.go, Docker.tsx): containers grouped by compose stack with live memory and CPU from the container sensors, start / stop / restart (asked first), logs that follow (polled with --since, no held connection), inspect (env names only), and images / volumes / networks tables. Works on every machine through the hub; #/grid/docker/immich_server links to one. Stacks (internal/api/stacks.go): every compose project, found from its containers' labels, with its file to edit and Deploy (up -d), Update (pull, then deploy), Restart and Stop as followed tasks. The file is looked for where the label says and through every container's mounts, because an orchestrator that runs compose inside its own container (Dockhand here; Dockge, Portainer) records a path in its own filesystem; compose is then run inside that container, so .env, relative paths and the compose labels stay exactly as deployed (verified: labels identical after a redeploy). Save validates with compose config first, keeps the previous file as <file>.hamlab-bak, and never deploys by itself. A stack deployed from another host (Dockhand on core driving grid over tcp) has no file on that machine, so the node alone lists it read-only — but the hub fills that in (hubStacks): asked about a node, it resolves the files on itself and runs compose here with DOCKER_HOST pointed at the node's tcp://…:2375, the way it was deployed; the task runs on the hub and the node's page follows it. Also per stack: every file it was deployed from (chips), its .env (values masked until "Reveal and edit"; validated in place, put back if compose objects), what Deploy would change (a line diff against the copy kept at the last deploy from here, <file>.hamlab-deployed, else the copy the last save replaced; .env values never shown), and image update checks — asked for, never polled, since each is a registry request Docker Hub counts; remembered six hours. A save refuses a file changed on disk meanwhile. Making and unmaking (2026-09-20): "+ New stack" — a name, a folder, compose text and an optional .env (written root-only), checked by compose before it is kept, then "Create and deploy" or "Create only". "+ New container" is the same dialog opened on a form (image, ports, volumes, environment, restart, command, host networking) that writes the compose for you: a container made here is a one-service stack, so it stays editable and redeployable, where a bare docker run could only be thrown away and typed again. New stacks go in a home — a folder where two or more stacks are already deployed from (Dockhand's stacks/core here; ~/stacks on a machine with none) — and are deployed the way their neighbours were (inside dockhand, or over tcp to a node from the hub). Homes are also scanned, so a stack with no containers ("not deployed": never deployed, or taken down) is still listed. Down removes a stack's containers; Delete… (type its name) takes it down and moves its own files to <home>/.hamlab-trash/<name>-<time>/ — moved, not erased; volumes and bind-mounted data are never touched. A stopped container can be removed (a running one must be stopped first, on purpose). Images: pull (a followed task), remove (unforced: Docker refuses one still in use) and prune (dangling only). Volumes (2026-09-20): each says which containers use it; one that nothing uses can be deleted — a named one by typing its name (it may be the data of a stack that is only down), an anonymous one by confirming its full name — and "Prune anonymous" clears the nameless leftovers of recreated containers, never named volumes. Images are marked "in use", and "Remove unused" takes every image no container uses. Nothing is ever forced: Docker refuses what is still in use, and that refusal is the answer. Still absent: removing networks.

  • Top merged into Processes — 2026-09-23. The two tabs answered the same question from the same data (Top already read /api/procs, the endpoint the Processes table was built on), so at Mitchell's instruction there is now one tab, Processes, showing what Top showed: the btop-style whole-machine screen. Procs.tsx is deleted; the top view id and its route are gone. A rail order saved before this keeps its shape — railorder.ts maps the retired top to procs and drops the duplicate, so the merged page sits where Top sat rather than falling to the bottom.

  • GPUs on the Processes tab — shipped 2026-10-02 (internal/metrics/gpu.go, the GPUBox in Top.tsx, gpu.* sensors). Asked for as "GPU activity and temps, like btop has it, always on". Each DRM device with a real GPU driver gets a box under the cpu box: busy curve, busy meter, a meter per engine (render/video/copy/enhance), temperature, clock, power and fan where the card has them, and the processes that have it open with their share. The process table gains a Gpu% column and sort. How it is measured, and the honest gaps: device busy is Intel's RC6 idle residency inverted (the counter every GT exposes without root; intel_gpu_top shows the same number) or amdgpu's own percent; per-process and per-engine shares are DRM fdinfo deltas, the way nvtop reads them. Other users' fdinfo needs root, so a daemon running as core keeps one root child (hamlab drm-clients, over a pipe) rather than a sudo a second. The Arc A380 on i915 6.12 exposes power, energy and fan but no temperature, so its box shows none; an integrated GPU shows the CPU package temperature marked *. Asahi (the Mac mini) has neither a busy counter nor fdinfo in 6.19, so it lists the device and says so. No NVIDIA path: nothing here has one to test against. Names come from an embedded PCI ID table (2026-10-02, internal/metrics/pciids_gpu.txt, regenerated by scripts/pci-ids-gpu.sh from the hub's pci.ids): gate has no lspci and showed "Intel GPU 3185" beside hamtv's "Intel UHD Graphics 600" for the identical chip. The table is tried first everywhere, so the name — and the sensor ID built from it — does not depend on which distro's pciutils a machine has; lspci is the fallback for a card newer than the table. Two fixes the same evening: the gpu box's layout rules had been lost from the stylesheet (v0.1.0-166/167), and on grid the sampler hung for 42 minutes in Go's poller on the Arc's fan1_input — readStr now uses plain blocking syscalls and each GPU's sysfs reads run under a 3-second deadline (v0.1.0-168; hamlet/docs/gotchas.md has the mechanism).

  • The ARC on the ZFS tab — shipped 2026-10-02 (internal/metrics/arc.go, internal/storage/arc.go, GET /api/zfs/arc, the Cache section in ZFS.tsx, zfs.arc.* sensors). Asked for as "current ARC usage, including any types of caching services in the pool". One card for the ARC: size against its ceiling with the target and floor, sparklines of size and hit rate from the sampler's history, what it holds (MRU/MFU/dirty, data vs metadata, compression, the ghost lists), the prefetcher and dbuf cache, memory-pressure counters, and the since-boot breakdown by demand/prefetch × data/metadata. Then a card per pool naming every caching tier — L2ARC, SLOG, special vdev, dedup table, spares — with its devices and fill, or saying plainly that there is none and what that means (the ZIL on the pool's own vdevs, with how much it has written), plus the properties that steer caching (primarycache, secondarycache, sync, logbias). Read from /proc/spl/kstat/zfs/* and /sys/module/zfs/parameters, with zpool list -Hpv and zfs get run without sudo (they need none) every 30 s at most. The day it shipped it showed tank's ARC pinned at its 2 GiB floor under arc_no_grow, with a 53 GiB python3 process holding the RAM.

  • Dashboard retired — 2026-09-23, at Mitchell's instruction, the same reasoning that merged Top into Processes: the app repeated itself. The customisable module list is gone (Dashboard.tsx deleted), and with it /api/prefs/dashboard, applyPrefs, and the per-user dashboard blob in state.json. The Processes tab already shows those numbers on one screen. fmtBytes/fmtRate moved to fmt.ts (Disks, Files and Space use them); MONITOR_HISTORY and seriesOf went with the page. The default view after login, after switching machines, and when a hub-only page is left behind, is now Processes. A saved rail order naming dashboard degrades quietly — orderBy drops ids with no page. Widgets stays and says what it is for: a notice at the top explains these are for dashboards outside Ham Lab — Glance, Home Assistant, a wall display — served from /embed/widget with a read-only sensor token, not a second copy of the console's screens. It also closed a real bug. applyPrefs set the machine's sampling rate from intervalMs inside the dashboard layout, and its else branch reset the sampler to the default whenever that field was absent. Since layouts became per-user, one account saving a dashboard could silently reset the sampling rate for the whole machine, overriding the fleet refresh rate set in Settings. The sampler now has one source: refreshMs in the UI prefs, which the hub fans out, remembered per machine as sampler_ms.

  • ZFS tab, with scheduled snapshots — shipped 2026-09-24 (internal/storage/snapshots.go, internal/api/zfs.go, ZFS.tsx). Pools, datasets and snapshots moved out of Disks — Disks is the hardware, ZFS is the filesystem on top of it — and the tab gained what it was split out for: snapshot schedules. Hourly, daily, weekly or monthly, per dataset, optionally recursive, with a retention period and a floor that keeps the newest few whatever their age. A missed window (the machine was off) runs once on return rather than being skipped. The naming is zfSnap's, deliberately. grid has run zfSnap v1.11.1 from root cron at 02:30 with a 6-week TTL since before this repo existed, and it works. So rather than invent a second convention for two tools to disagree over, this writes and reads exactly what zfSnap does — tank@2026-09-24_02.30.01--6w, retention encoded in the name. Three consequences: Ham Lab shows and expires the snapshots zfSnap already took; the two can run side by side without fighting, since both only remove what has passed the expiry written into its own name; and changing a schedule never re-dates old snapshots, because each carries the rule it was taken under. The page detects the existing cron job (zfSnap, zfs-auto-snapshot, sanoid, or a bare zfs snapshot) and says so, warning if a Ham Lab schedule is running as well. Nothing is destroyed unseen. Only names that parse as a stamp and a TTL are ever candidates for expiry, so a snapshot taken by hand is invisible to a schedule; Run now says that expired snapshots go with it; and a snapshot deleted by hand is confirmed first and needs its full name echoed back by the caller. (A "what would it delete?" preview shipped and was removed the same day as redundant — Mitchell's call.) The datasets table is a snapshot explorer: each dataset shows how many snapshots it has and what they cost, and opens into the snapshots themselves — when each was taken, when it expires (read from the TTL in its own name, so no server call), its size, and Delete. Deleting takes the same-named snapshot on descendant datasets too, which is how a recursive schedule made it. A schedule is a real cron job (2026-09-24, Mitchell's call after the first cut ran it from a ticker inside the daemon). Ham Lab writes /etc/cron.d/hamlab-zfs, owns it, and rewrites it whole on every change; each line calls hamlab zfs-snapshot <id> as the account Ham Lab runs as, reaching zfs through sudo as the daemon does. Snapshots are backup-shaped and must not stop because a console is down, upgrading, or uninstalled — and a cron line is greppable by anyone looking at the machine the normal way. A cron expression is the single description of when a schedule fires: the four common shapes render to one, custom states one outright, and the next run is read back with the same evaluator the Jobs tab uses on everyone else's lines, so the page, the file and the prediction cannot disagree. A run's outcome is written to zfs-runs/<id>.json beside the config, never to state.json, because the cron job is a separate process and the daemon would clobber it. An existing external job can be adopted — its schedule, retention and dataset read out of the cron line into a new Ham Lab schedule. zfSnap is retired on grid as of that day: Ham Lab now takes the nightly tank snapshot, the package is purged and the old cron line is gone, with the 515 snapshots it had taken left untouched and fully readable here — which is what adopting its naming bought. See hamlet/docs/hosts.md.

  • KVM tab — shipped 2026-09-24 (internal/api/kvm.go, KVM.tsx, config.KVM). Paste a device's address; its own web page appears in the tab. Nothing in it knows anything about any particular KVM — the first device here is a Sipeed NanoKVM on grid at 10.0.0.217, but none of that is in the code. Why it proxies rather than framing the device directly. Ham Lab is served over TLS and a KVM usually is not, so a bare http://… iframe is blocked as mixed content before it loads. Serving the device from Ham Lab's own origin (/kvm/<id>/…) fixes that, puts it behind Ham Lab's login rather than a second one, and makes the iframe same-origin. The general problem, and the answer. A device UI asks for its assets by absolute path — /assets/app.js, /api/whatever — which no path prefix can rewrite away, and rewriting bodies per device would not be general. So the prefix is only the entry point: any request Ham Lab does not serve itself is routed to whichever KVM the asking page belongs to, read from the Referer, falling back to a cookie when there is none (a WebSocket upgrade). Ham Lab's own routes always match first, so a device can never shadow the console, and the fallback is gated on a live session. The device never receives Ham Lab's session cookie, and its X-Frame-Options/CSP/HSTS are dropped on the way out since Ham Lab is the origin now. Redirects to the device's root are rewritten back through the prefix. One catch found in testing: Ham Lab's own frame-ancestors 'none' forbids framing even from the same origin, so /kvm/ paths send 'self' instead. Without that the iframe was refused before it loaded. The keyboard needs less than it sounds: a page cannot forward keys into an iframe and does not need to, since one with focus receives them directly. Fullscreen plus navigator.keyboard.lock() is what takes Escape, Tab and Ctrl-W back from the browser; that is Chromium-only, and the tab says which version you are getting rather than pretending. One KVM per machine, and the tab follows the machine switcher (corrected the same day: the first cut was a hub-side list you pointed at machines, which had it backwards). Switch to grid and you get grid's; adding a second for a machine replaces the first. The addresses are still kept on the hub rather than on each machine, because a KVM earns its keep exactly when the machine is down and its own Ham Lab cannot answer — the hub proxies the device either way — and a KVM need not belong to a machine Ham Lab manages at all. Not done: holding the device's own login on the hub, so its page does not ask for a second password.

  • Skins — 13 since 2026-09-19 (themes.ts: a palette is one mk(…) line): Dark, Light, Cyberpunk, Retro, HUD (hairlines of cyan on near-black, corner ticks, thin type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light; each repaints the terminal too. Each machine wears a colour (accents.ts): a stripe across the header and its name in the switcher — picked from ten swatches or any colour on the Machines page, else derived from its name — saved with the theme on the hub.

  • Multi-host — Phase 1 shipped 2026-09-18 (internal/nodes, internal/tlsauto, MachineSwitcher.tsx, Machines.tsx). One binary, hub and spoke: a host whose config.json lists nodes is a hub; any other is a standalone node with its full UI. Node side: "tls": "auto" makes a self-signed cert in the data dir and serves HTTPS; POST /api/tokens {password,name} trades the login password for a long-lived bearer token (SHA-256 stored in tokens), accepted by the auth middleware alongside the cookie; DELETE /api/tokens/{name} revokes. Hub side: POST /api/nodes {name,url, password} peeks the node's certificate (trust on first use, fingerprint pinned), obtains a token named hub:<hostname>, stores {name,url,fingerprint,token}; a 15 s prober keeps each node's status; /api/nodes/{name}/… is a reverse proxy to the node's /api/… (bearer swapped in, cookie and Origin stripped, streaming flush) that carries WebSockets, so terminals, log tails and task output work remotely. Frontend: setMachine() rewrites every /api/… path; the rail has a machine switcher at the top and a Machines page to add/remove/open. Pages remount on switch; terminals live on the node. Nodes: grid, wyse. Phase 3 (update from hub) shipped 2026-09-18: node POST /api/self-update takes a binary, checks it answers version, keeps hamlab.prev in the data dir, installs over the executable via sudo install, replies, then restarts the service; the hub pushes its own executable with POST /api/machines/{name}/update or /api/machines-update-all (architecture checked first via /api/me.arch), the Machines page shows an outdated badge and Update buttons, and hamlab push-nodes does the same from the CLI — make deploy runs it last, so one deploy on the hub updates every machine. First install on a new host is still by hand (scp binary + install.sh, hamlab passwd, tls auto). Still to do: Phase 2 Datacenter overview; optional firewall rule limiting a node's port to the hub. Cross-architecture (2026-09-18): make deploy also builds bin/hamlab-linux-arm64, install.sh installs it beside the binary as hamlab-arm64, and the hub picks the sibling matching a node's /api/me.arch when pushing — the Mac mini (Fedora Asahi, aarch64, node mac) updates like the rest.

  • Portability + install — shipped 2026-09-18 (internal/setup, internal/priv, install.sh, make release): the binary installs itself (hamlab install / uninstall; systemd or OpenRC; root, sudo or doas), all builds static on amd64, arm64, arm, 386 and riscv64, install.sh as the curl-able bootstrap with checksums. Every privileged call goes through priv instead of a literal sudo -n. Updates tab acts on dnf, zypper, pacman and apk as well as apt. Services list falls back to the plain table on systemd < 246; Disks falls back to /sys/block without lsblk. Listener uses IP_FREEBIND. Self-update: pre-flight check against the live config, atomic replace, exec in place. Config default moved to ~/.config/hamlab (existing ~/apps/hamlab still found by the installer and the CLI). Tested in containers: Debian 12, Ubuntu 22.04, Fedora 40, Rocky 8, openSUSE Leap 15.5, Alpine 3.18/3.20 (OpenRC + doas, non-root), plus the four lab hosts. procd (OpenWrt) added 2026-09-26 (internal/setup/hamlab.procd): detected on /sbin/procd + /etc/rc.common ahead of the OpenRC branch, installed to the same /etc/init.d/hamlab path and told apart from OpenRC at uninstall by USE_PROCD=1 in the file. Verified on the new router (gate, OpenWrt 25.12.5, running as root): S95/K10 symlinks, procd supervising the instance, Docker tab and stacks working through the hub. The Updates tab needed no change — apk-tools 3 answers apk version -l '<' the same way Alpine's does. Not done: an OpenRC or procd backend for the Services tab (it is systemd-only and says so cleanly); runit/s6/sysvinit service files; public hosting for install.sh (the Gitea repo is private, so the one-liner needs HAMLAB_TOKEN).

  • Users, sessions and settings that follow the account — shipped 2026-09-22 (internal/api/users.go, config.User, store.Session/UserState, Login.tsx, Users.tsx — a hub-only Users tab in the rail, added the same day at Mitchell's request, holds everything about accounts: your own password, your sessions, the accounts; Settings keeps appearance, rates, tokens and About). Rail order is per account too (2026-09-22, railorder.ts, Rail.tsx): "⇅ Arrange" at the foot of the rail puts a handle on every page; drag one up or down (pointer events, so a finger works), let go and the order is saved in the UI prefs with the theme; "Reset" restores the built-in order; pages added in a later build go after the ones the saved order names. The login is a user name and a password. config.json holds users (name, argon2id hash, created, last seen); the old single password_hash becomes the account named after the OS user the console runs as on first boot, its signed-in sessions become that account's, and the machine-wide theme/accents/dashboard become theirs — nothing to redo after the upgrade. A session carries its user, an ID, when it was made and last seen, the IP and the browser; Users → Sessions lists yours ("Safari on iPhone · 10.0.0.5 · seen 2 min ago"), revokes one, or signs out everywhere else. Users → Users lists accounts with their session counts, adds one, removes one (their sessions and settings go too; never yourself, never the last), and resets another's password (they are signed out); removing and resetting take your own password first. Everyone signed in can do everything — a household console, not a tenancy — and each machine keeps its own accounts: a hub reaches a node with its token, so pairing (Machines → Add) now takes a user and password on the node, the user optional while it has one account. Per account: ui (theme, accents) and dashboard in state.json under users.<name>. Still the machine's: refresh rate (it is how often every machine samples — it rides in the UI prefs object for the client but is stored once, machine-wide), graph history, alert rules, terminals (the shells run as one OS user anyway). A login attempt for a missing user takes as long as a wrong password, so the form can't list accounts. Every write's log line now names the account or token (core@10.0.0.5 via …), and Access tokens say who issued them. CLI: hamlab passwd [user] (makes the account if new; reads stdin when piped), hamlab users, hamlab user-rm <user>; hamlab install names the first account after the service user. Not done: a read-only viewer role.

  • Space tab — shipped 2026-09-22 (internal/space, internal/api/space.go, Space.tsx), asked for as "exactly WizTree": what is using a disk. The unit is a whole volume, picked from a dropdown at the top: a disk on its own (every filesystem its partitions back) or a ZFS pool as a whole (every mounted dataset) — a pool's member disks are listed under it with the layout (raidz2, 8 disks: sda …) and never offered apart, since one disk's share of a pool means nothing (Mitchell's call, same day, after a first cut that listed the disks as chips). The walk runs as a helper process, hamlab space-walk <roots>, as root through priv when the host allows (so /var/lib/docker and /root count), 32 workers reading directories depth-first, never crossing a filesystem boundary by itself; each root is grafted into one tree at its path. Files under 128 KiB are not kept one by one but summed per folder as "N smaller files" — what keeps a scan of millions of files to tens of MB in the daemon; the tree crosses back as a varint stream (codec.go), and the helper's memory goes with it. core's root: 646k files in ~1.2 s. Progress (files, bytes, current folder) every 250 ms; Cancel sends SIGTERM. The page: picker → Scan/Rescan → Folders (a tree largest-first: size, share of parent as a bar, files, folders, modified; expand in place) over a treemap of the selected folder — squarified, nested to the file, tiles coloured by kind (video, images, audio, archives & disk images, documents, code & config, databases & logs, other; the validated categorical set, dark and light) with folders as labelled boxes; hover for path, size and share; click a folder to drill, breadcrumbs and ↑ to come back. The map asks the server for the folder pruned to what a tile could show (min = 1/6000 of the folder, 24 deep; deeper folders come back as one lump), the list one level at a time. Largest files: the 1,000 biggest with a path filter. Works on every machine through the hub. Smarter and faster (2026-09-23, at Mitchell's request): a scan is kept — in memory and in <data>/space/<target>.scan (the same varint stream) + .json, read back after a restart — and the next scan starts from it: the helper is handed the last tree on stdin and reads a directory again only when something says it changed. On a ZFS pool with snapshots, the scan records the newest snapshot of each dataset at start (bases), and the next one runs zfs diff -H -F <base> per dataset: only the directories it names are read, their ancestors kept but looked into, everything else kept wholesale without a stat. The diffs run side by side and one is given three minutes — timeout -s KILL inside sudo, since Debian's sudo gives the command its own session and a process-group kill from outside never reached zfs — after which that dataset is walked by mtime instead. On tank a diff is metadata-bound like the walk: 8 s to 3 min per dataset for a handful of changes, and immich, which changes constantly, ran past twenty minutes the first time. A refresh of tank took 4 min 19 s against 7 min for the full scan. Elsewhere (ext4, or a dataset with no snapshot) a directory whose mtime — to the nanosecond — is unchanged keeps its list and only its kept files are stat'ed again, since a file can grow without its directory noticing. A cancelled scan keeps what it walked, marks the folders it never got to (partial, "not yet read" in the tree) and Resume reads just those. core: full 1.2 s → refresh 0.3 s (2 folders read, 84,755 kept). The status line says which (method, by_root, read/kept counts); Forget drops the saved result so the next scan is full. The old result stays browsable while a refresh runs. Treemap rethought the same day: every file a tile grouped by folder — folders are only a thin line and a name strip on the first level — instead of nested boxes with headers, which on Linux's single-child folder chains was a staircase of headers around nothing. Not done: delete from here, a size-over-time history, per-user column choices.

  • Updater — shipped 2026-09-23 (internal/selfupdate, internal/api/update.go, the Updates card in Settings.tsx, scripts/gitea-release.sh). A Ham Lab asks one URL what the newest build is and installs it on request. update.url in config.json is either a Gitea/Forgejo repo API base (…/api/v1/repos/<owner>/<repo>, whose newest release names the version and carries the binaries as assets) or any static directory holding VERSION, SHA256SUMS and hamlab-linux-<arch> — which is exactly what make release writes and what install.sh already downloads, so the demo site, a folder on a NAS and a Gitea release are all the same thing to it. "off" never looks. update.token is sent when the source is behind a login, so a private Gitea serves its own fleet without the instance being opened to the world. The binary is verified against SHA256SUMS before it goes anywhere near the running one, then takes the same path POST /api/self-update always did (runs version, runs check against this machine's config, atomic replace keeping hamlab.prev, re-exec in place) — that path is now one shared method, installBinary. Version ordering is install.sh's vnum() rule ported to Go and tested against it, so the daemon and the installer never disagree about which of two builds is newer; the commit count since the tag counts, the hash does not. Checked ~90 s after boot and daily (update.check: "never" switches that off), never installed unasked — an update restarts the daemon and drops open terminals, and the dialog says so. Settings → Updates: installed, published, when it last looked, the source, the release notes, Check for updates, Install; the rail badges Settings when a newer build is out (update_available rides on /api/me). Publishing is make gitea-release → scripts/gitea-release.sh, which creates the release and uploads the five architectures, SHA256SUMS, install.sh and VERSION; it is deliberately not part of make deploy, because deploy pushes this build to the machines this hub owns while publishing offers it to every install anywhere, and one tag per deploy would be noise. Re-publishing a version needs --force. Public since 2026-09-23, at Mitchell's instruction and after the trade was put to him twice: Gitea's instance-wide REQUIRE_SIGNIN_VIEW is off and mitchell/hamlab is marked public, so anyone can browse it, clone it and download a release with no account. The other five repos there are each marked private and are invisible — an anonymous request for one is a 404, not a login prompt — registration is still disabled, and new repos still default to private. No token is needed by anything any more (the read-only one was revoked; only hamlab-publish, which creates releases, remains). Gitea resolves releases/download/latest/<asset>, so install.sh and HAMLAB_URL now point straight at the newest release and never need changing. What is public is the source, the history, BUGS.md and this file — which describes the lab and names the console's hostname.

  • Auth hardening — WebAuthn passkeys (phone Face ID), TOTP fallback, optional NPM access list restricting to WireGuard/LAN.

  • Files tab — shipped 2026-09-18 (internal/files, Files.tsx): browse from / with breadcrumbs, an editable path box, sort, hidden-file toggle and a filter; rows show type icon, size, modified, perms and owner, link targets and broken links. Text files (≤ 2 MiB, sniffed) open in an editor with Ctrl+S / Save; images preview; everything else downloads. Upload via drag-and-drop or picker with per-file progress (streamed, no size limit). New folder / file, rename, move, chmod, copy path, delete (folders need the name typed back), and "open in terminal", which cds the active shell (or a fresh one) to the folder. Sudo toggle: root access re-executes the same binary as hamlab fs <op> under sudo -n, streaming bodies over stdin/stdout, so both modes share one implementation and one set of rules (absolute paths only; /, /etc, /usr, … are never deleted or moved whole). Every write is logged with the caller's IP.

  • Docker read-only view — docker ps/logs; dockhand stays the source of truth.

  • Notifications — push on service failure / SMART warning / UPS event.

  • Terminal extras — search, copy-mode, font/theme, split panes, upload via drag-drop into cwd.

  • Settings + themes — shipped 2026-09-18 (Settings.tsx, themes.ts): Settings page with Appearance (Dark, Light, Cyberpunk, Retro — CSS variables on <html> plus a matching xterm palette, applied before first paint from localStorage and synced hub-side via GET/PUT /api/prefs/ui so the choice follows the login), Account (change password, POST /api/password, current password required), Access tokens (list/revoke the tokens hubs hold), About (version, host, arch, uptime, certificate).

Network tab (2026-09-27)

A Network tab, per machine like every other: the interfaces as the kernel has them (addresses, MAC, MTU, negotiated link speed, carrier, byte totals, live rates from the sampler's own sensors), the default route and resolvers, what the internet sees of this connection, and an internet speed test.

Decisions worth keeping:

  • Measured by the daemon, not the browser. The test runs on the machine in the switcher, so the figure is that machine's line — the router's, if you switch to gate — and it carries on if the phone locks or the tab closes. The page polls a run it is not driving.
  • No tool to install. It measures against Cloudflare's public speed endpoints (__down, __up, cdn-cgi/trace), which is what makes it work identically on Debian and on OpenWrt, where opkg install costs flash. That is the feature's one external dependency, and it is named in internal/network/speedtest.go.
  • Several streams, HTTP/2 off, and the ramp thrown away. One stream over one connection measures a congestion window; HTTP/2 would put every stream back on one connection and do the same. The first second of each phase is measured and discarded.
  • Latency is the round trip less the far end's own Server-Timing. A zero-byte response is mostly Cloudflare's worker running — 20 to 45 ms of it against a few ms of wire. Left in, it reported 31 ms for a 7 ms line. Jitter is the mean change between consecutive probes, not a standard deviation, so one slow probe doesn't run away with it.
  • A refused phase is not a number. The far end rate-limits by volume, and a test moves a few hundred megabytes; several in a row earn 429s, and the block observed while building this lasted over twenty minutes. So: a 20-second cooldown counted from the last attempt (a refused test still used the line), a stream that retries twice before giving up, the rate computed over the window in which bytes were actually still arriving, and a plain sentence when it was cut short. Coasting to the end of the clock with nothing arriving reported a third of the truth.
  • A second download source, for when the first refuses. A plain 100 MB file on Linode's Seattle mirror, which has no opinion about volume. It is only reached after a refusal, and every result names the source it used — that mirror is in one place, so on a machine far from Seattle it would read low, and a number that quietly came from somewhere else would be a lie. Upload stays on Cloudflare, which has never been the one to refuse.
  • The last result is published as sensors (net.internet.*), so the line is a widget, a graph, an alert or a Prometheus series like anything else. Each carries the time the test ran, not the time it was asked for.
  • Not scheduled. Nothing takes a speed test on its own. A nightly one recorded to the history is the obvious next step and deliberately not built yet — the cron shape for it already exists twice over (ZFS snapshots, Docker image updates).

VPN tab (2026-09-27)

A VPN tab beside Network, per machine: every WireGuard tunnel on the machine in the switcher, and the devices on it — name, address, last handshake, where it connects from, and what it has moved.

  • Per machine is the point, not a limitation. The tunnel people dial into and a tunnel a machine dials out through are on different boxes here (gate and hamtv), and a combined view would invite exactly the question "which box is this on".
  • The kernel is the truth, the config only supplies names. wg show all dump has the handshakes and counters and knows nothing but public keys; names come from the # comment above a peer in a wg-quick file, or a UCI section's description on OpenWrt. Both are read only to label what the kernel already said.
  • Server or client is worked out, not declared. A peer allowed 0.0.0.0/0 that has an endpoint is a way out, so a tunnel holding one is this machine dialling out; a tunnel with a listen port and peers is one being dialled. Anything else says neither rather than guessing.
  • "Online" is a handshake in the last three minutes, and the tab says so. WireGuard is silent when idle; there is no connection to be in, and a tab claiming otherwise would be lying twice a day.
  • No secrets are read. Private keys are in the second column of the dump and are read past; pre-shared keys are noted as present and never read.
  • Devices can be created, renamed and removed (added 2026-09-27, asked for the same day), on a tunnel the machine configures — today OpenWrt's UCI, which is where the server here lives. A wg-quick host is reported unmanaged and shows no buttons: half-supporting a second configuration format is worse than an honest refusal.
    • Nothing restarts the network. uci commit writes the file and applies nothing; wg set changes one peer on the live interface. Proven on gate by the existing peers' transfer counters surviving a create and a delete — a reload would have zeroed them.
    • The two halves are ordered so they cannot disagree quietly. Adding writes the configuration first and rolls it back if the live tunnel refuses, because a device that works only after the next reboot is the worst of the three outcomes. Removing does the live tunnel first, since that is the moment access ends, and says so plainly if the configuration half then fails.
    • The keypair is generated on the machine and the profile, with its QR, is shown once. The private key is stored nowhere — the same bargain as a guest login's password.
    • Removing asks for the name to be typed back, at both ends. It cannot be undone and it sits one tap from a list of devices that all look alike.

To do

  • Grow the Files tab until copyparty can go — asked for 2026-09-23 as two items (Files polish, and deprecating copyparty), merged 2026-09-24: they are the same tab and the same work. copyparty on grid (copyparty.service, :3923, /etc/copyparty.conf, serving all of /tank as user core with one account behind NPM) overlaps Files almost entirely. The aim is to stop the service and keep only Ham Lab, with general ease-of-use polish along the way. What Files must gain first, in the order that matters:

    • Multi-select and bulk actions. copyparty selects many rows and moves, deletes or downloads them together; Files acts on one row at a time.
    • Download a folder, and a multi-file download, as a streamed zip or tar — the thing copyparty is reached for most.
    • Resumable uploads. copyparty's up2k chunks a file, survives a dropped connection and skips what is already there; Files streams one PUT, so a phone upload that drops at 90% starts over.
    • A gallery. Thumbnails for a folder of images and video, and play audio/video in place — /tank/media and /tank/immich are most of what is browsed.
    • Search below the current folder by name, since find in a terminal is the fallback today.
    • Decide when built: whether share links (copyparty's public per-path links) are wanted at all — they are the one feature that hands out access without a login, and Ham Lab has no notion of that; if not, say so in the docs rather than leaving a gap unexplained.
    • After: systemctl disable --now copyparty, remove its NPM host, keep /etc/copyparty.conf until a rollback is clearly not needed, then update hamlet's docs/hosts.md bare-metal list, docs/gotchas.md (the "docker ps is not the inventory" note names it) and the grid row in CLAUDE.md.

  • Grow the Docker tab until Dockhand can go — asked for 2026-09-23 and widened 2026-09-24 to take in Docker tab polish and feature work, on the same reasoning as the Files item: it is one tab, and the gaps that block retiring the old tool and the things that would simply make it better are the same body of work. The homelab half — the cutover itself, then moving every stack off Dockhand's data dir — is hamlet's docs/pending.md → "Retire Dockhand; Ham Lab becomes the orchestrator", agreed 2026-09-20. This entry is the app half.

    Done 2026-09-24 — the eight Mitchell picked from the inventory, all in the Docker tab: per-container Stats, Health (with the last few probe outputs), Events (Ham Lab keeps its own 4000-entry ring, since docker events forgets after about a minute), Shell (docker exec, handed to the Terminal tab) and Limits (docker update for memory, CPU and restart policy); an updates tab giving the whole machine at once with a per-container indicator in the list; and schedules, which write /etc/cron.d/hamlab-docker and call hamlab docker-update <id> — the shape the ZFS schedules already use. Also done: Ham Lab keeps working when dockhand is removed. It remembers where a container's bind mounts pointed (mounts in state.json) and resolves a stack through that record when no running container explains its path, running compose on the host instead; repoint does the same for a schedule at run time. Without this, removing the container would have made 17 stacks on core and 5 on grid unfindable at once.

    • Left from the Dockhand inventory, both to decide rather than build.
      • Git-sourced stacks. Dockhand's stack_sources table points a stack at a repo and pulls it. Its git_repositories and git_stacks tables are empty, so nothing here uses it — dropping it is the honest answer unless Mitchell says otherwise.
      • Webhooks, OIDC, LDAP, vulnerability scans. All four tables empty. Nothing to move.

    Done 2026-09-25 — the two gaps in the tab itself.

    • Logs are followed over a held connection, the way the Services tab follows a journal (internal/api/docker_logs.go, /api/docker/containers/{id}/logs/tail): one docker logs -f per viewer, stdout and stderr both, stderr coloured as an error. A reconnect resumes from the last line's timestamp, so a container that restarts or a phone that slept doesn't replay what is already on screen; the client dedupes by line rather than by "later than", because --since is inclusive and two lines can share a timestamp. Docker replays the opening backlog down two pipes in no particular order, so the first 300 ms is held and sorted before it is sent. The polled ?since= endpoint stays — it is what the resume is built on. Reconnects back off to 15 s, since a container that has stopped ends the stream immediately.

    • Networks can be removed and pruned — POST /api/docker/networks/{remove,prune}, the shape images and volumes already had. The overview says who is attached to each one, so the ✕ is only offered on a network nothing uses and never on Docker's own three. Prune is deliberately untested against this host: core has several networks belonging to stacks that are merely down (pihole_default, transmission_default, gluetun_default among them), and pruning would take them and hand them new subnets on the next deploy.

    • The *:2375 question, which this is the moment to settle. A stack deployed to grid or the Mac still goes over tcp://<node>:2375 from the hub. Decide whether the hub should instead ask the node's own Ham Lab to deploy locally, which would end the unauthenticated Docker socket exposure — its own item in docs/pending.md, currently filed under "decided, no action", and this is the change that would make closing it free.

  • A Virtualization tab — asked for 2026-09-23; scoped 2026-10-09 in VIRTUALIZATION.md, not started. Incus 6.0 LTS as the engine; the tab detects per machine whether Incus is running, installable, or unavailable, and whether KVM is there, and acts on the machine in the switcher — never a default host. Containers and VMs in one list, console/shell through the terminal machinery, snapshots, exports, and the old Proxmox estate on tank as a later import. That file has the probe of every host, the per-machine layout Ham Lab computes (pool driver, a bridge subnet that avoids the LAN's /8 and Docker's pools, the Docker FORWARD fix) and the phased build.

Open questions

  • Idle-session reaping policy (never / after N days) — leaning never; it's a single-user box.
  • Whether sudo -i-style root sessions get their own leaf colour. Cosmetic.
Reference in New Issue View Git Blame Copy Permalink