Container events are Ham Lab's own record, not Docker's short memory

docker events is a live stream with a small in-memory backlog: ask for thirty days and you
get a few minutes, and a daemon restart loses the lot. That is why Dockhand kept its own
copy. A watcher now follows the stream and appends lifecycle events to a ring beside the
config, flushed every thirty seconds, so the question of what happened to a container last
week has an answer. Health-check exec events are excluded — three per probe, thousands a
day, and the health panel says it better.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

PLAN.md

@@ -276,6 +276,83 @@ something long in one, switch away and back, refresh — it's all still there.
   a dashboard could silently reset the sampling rate for the whole machine, overriding the
   fleet refresh rate set in Settings. The sampler now has one source: `refreshMs` in the UI
   prefs, which the hub fans out, remembered per machine as `sampler_ms`.
+- **ZFS tab, with scheduled snapshots** — **shipped 2026-09-24** (`internal/storage/snapshots.go`,
+  `internal/api/zfs.go`, `ZFS.tsx`). Pools, datasets and snapshots moved out of Disks —
+  Disks is the hardware, ZFS is the filesystem on top of it — and the tab gained what it was
+  split out for: **snapshot schedules**. Hourly, daily, weekly or monthly, per dataset,
+  optionally recursive, with a **retention period** and a floor that keeps the newest few
+  whatever their age. A missed window (the machine was off) runs once on return rather than
+  being skipped.
+  **The naming is zfSnap's, deliberately.** `grid` has run zfSnap v1.11.1 from root cron at
+  02:30 with a 6-week TTL since before this repo existed, and it works. So rather than
+  invent a second convention for two tools to disagree over, this writes and reads exactly
+  what zfSnap does — `tank@2026-09-24_02.30.01--6w`, **retention encoded in the name**. Three
+  consequences: Ham Lab shows and expires the snapshots zfSnap already took; the two can run
+  side by side without fighting, since both only remove what has passed the expiry written
+  into its own name; and changing a schedule never re-dates old snapshots, because each
+  carries the rule it was taken under. The page **detects the existing cron job**
+  (zfSnap, zfs-auto-snapshot, sanoid, or a bare `zfs snapshot`) and says so, warning if a Ham
+  Lab schedule is running as well.
+  **Nothing is destroyed unseen.** Only names that parse as a stamp and a TTL are ever
+  candidates for expiry, so a snapshot taken by hand is invisible to a schedule; Run now
+  says that expired snapshots go with it; and a snapshot deleted by hand is confirmed first
+  and needs its full name echoed back by the caller. (A "what would it delete?" preview
+  shipped and was removed the same day as redundant — Mitchell's call.)
+  The datasets table is a **snapshot explorer**: each dataset shows how many snapshots it has
+  and what they cost, and opens into the snapshots themselves — when each was taken, **when
+  it expires** (read from the TTL in its own name, so no server call), its size, and Delete.
+  Deleting takes the same-named snapshot on descendant datasets too, which is how a recursive
+  schedule made it.
+  **A schedule is a real cron job** (2026-09-24, Mitchell's call after the first cut ran it
+  from a ticker inside the daemon). Ham Lab writes `/etc/cron.d/hamlab-zfs`, owns it, and
+  rewrites it whole on every change; each line calls `hamlab zfs-snapshot <id>` as the
+  account Ham Lab runs as, reaching zfs through sudo as the daemon does. Snapshots are
+  backup-shaped and must not stop because a console is down, upgrading, or uninstalled — and
+  a cron line is greppable by anyone looking at the machine the normal way. **A cron
+  expression is the single description of when a schedule fires**: the four common shapes
+  render to one, `custom` states one outright, and the next run is read back with the same
+  evaluator the Jobs tab uses on everyone else's lines, so the page, the file and the
+  prediction cannot disagree. A run's outcome is written to `zfs-runs/<id>.json` beside the
+  config, never to `state.json`, because the cron job is a separate process and the daemon
+  would clobber it. An existing external job can be **adopted** — its schedule, retention
+  and dataset read out of the cron line into a new Ham Lab schedule.
+  **zfSnap is retired on `grid`** as of that day: Ham Lab now takes the nightly `tank`
+  snapshot, the package is purged and the old cron line is gone, with the 515 snapshots it
+  had taken left untouched and fully readable here — which is what adopting its naming
+  bought. See `hamlet/docs/hosts.md`.
+- **KVM tab** — **shipped 2026-09-24** (`internal/api/kvm.go`, `KVM.tsx`, `config.KVM`).
+  Paste a device's address; its own web page appears in the tab. **Nothing in it knows
+  anything about any particular KVM** — the first device here is a Sipeed NanoKVM on `grid`
+  at `10.0.0.217`, but none of that is in the code.
+  **Why it proxies rather than framing the device directly.** Ham Lab is served over TLS and
+  a KVM usually is not, so a bare `http://…` iframe is blocked as mixed content before it
+  loads. Serving the device from Ham Lab's own origin (`/kvm/<id>/…`) fixes that, puts it
+  behind **Ham Lab's** login rather than a second one, and makes the iframe same-origin.
+  **The general problem, and the answer.** A device UI asks for its assets by absolute path
+  — `/assets/app.js`, `/api/whatever` — which no path prefix can rewrite away, and rewriting
+  bodies per device would not be general. So the prefix is only the entry point: any request
+  Ham Lab does not serve itself is routed to whichever KVM the asking page belongs to, read
+  from the **Referer**, falling back to a **cookie** when there is none (a WebSocket upgrade).
+  Ham Lab's own routes always match first, so a device can never shadow the console, and the
+  fallback is gated on a live session. The device never receives Ham Lab's session cookie,
+  and its `X-Frame-Options`/CSP/HSTS are dropped on the way out since Ham Lab is the origin
+  now. Redirects to the device's root are rewritten back through the prefix.
+  **One catch found in testing:** Ham Lab's own `frame-ancestors 'none'` forbids framing even
+  from the same origin, so `/kvm/` paths send `'self'` instead. Without that the iframe was
+  refused before it loaded.
+  **The keyboard** needs less than it sounds: a page cannot forward keys into an iframe and
+  does not need to, since one with focus receives them directly. Fullscreen plus
+  `navigator.keyboard.lock()` is what takes Escape, Tab and Ctrl-W back from the browser;
+  that is Chromium-only, and the tab says which version you are getting rather than
+  pretending.
+  **One KVM per machine, and the tab follows the machine switcher** (corrected the same day:
+  the first cut was a hub-side list you pointed at machines, which had it backwards). Switch
+  to `grid` and you get grid's; adding a second for a machine replaces the first. The
+  addresses are still kept **on the hub** rather than on each machine, because a KVM earns
+  its keep exactly when the machine is down and its own Ham Lab cannot answer — the hub
+  proxies the device either way — and a KVM need not belong to a machine Ham Lab manages at
+  all. Not done: holding the device's own login on the hub, so its page does not ask for a
+  second password.
 - **Skins** — 13 since 2026-09-19 (`themes.ts`: a palette is one `mk(…)` line): Dark,
   Light, Cyberpunk, Retro, **HUD** (hairlines of cyan on near-black, corner ticks, thin
   type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light;
@@ -465,11 +542,12 @@ something long in one, switch away and back, refresh — it's all still there.
 
 ## To do
 
-- **Deprecate copyparty** — asked for 2026-09-23, not started. `copyparty` on `grid`
+- **Grow the Files tab until copyparty can go** — asked for 2026-09-23 as two items (Files
-  (`copyparty.service`, `:3923`, `/etc/copyparty.conf`, serving all of `/tank` as user
+  polish, and deprecating copyparty), merged 2026-09-24: they are the same tab and the same
-  `core` with one account behind NPM) overlaps the **Files** tab almost entirely. The aim
+  work. `copyparty` on `grid` (`copyparty.service`, `:3923`, `/etc/copyparty.conf`, serving
-  is to stop the service and keep only Ham Lab. What Files must gain first, in the order
+  all of `/tank` as user `core` with one account behind NPM) overlaps **Files** almost
-  that matters:
+  entirely. The aim is to stop the service and keep only Ham Lab, with general ease-of-use
+  polish along the way. What Files must gain first, in the order that matters:
   - **Multi-select and bulk actions.** copyparty selects many rows and moves, deletes or
     downloads them together; Files acts on one row at a time.
   - **Download a folder**, and a multi-file download, as a streamed zip or tar — the thing
@@ -489,54 +567,46 @@ something long in one, switch away and back, refresh — it's all still there.
     `hamlet`'s `docs/hosts.md` bare-metal list, `docs/gotchas.md` (the "`docker ps` is not
     the inventory" note names it) and the `grid` row in `CLAUDE.md`.
 
-- **Deprecate Dockhand** — asked for 2026-09-23, not started; the *homelab* half of this
+- **Grow the Docker tab until Dockhand can go** — asked for 2026-09-23 and widened
-  (migrating every stack off Dockhand's data dir, then removing the container) is
+  2026-09-24 to take in Docker tab polish and feature work, on the same reasoning as the
-  `hamlet`'s `docs/pending.md` → "Retire Dockhand; Ham Lab becomes the orchestrator",
+  Files item: it is one tab, and the gaps that block retiring the old tool and the things
-  agreed 2026-09-20. This entry is the *app* half only: what Ham Lab must do before that
+  that would simply make it better are the same body of work. The *homelab* half —
-  migration can start. Its Stacks page already creates, edits, deploys, updates, restarts,
+  migrating every stack off Dockhand's data dir, then removing the container — is `hamlet`'s
-  stops, downs and deletes stacks on every machine, so the gaps are Dockhand's extras:
+  `docs/pending.md` → "Retire Dockhand; Ham Lab becomes the orchestrator", agreed
-  - **Git-sourced stacks.** Dockhand's `stack_sources` table points a stack at a repo and
+  2026-09-20. This entry is the *app* half.
-    pulls it. Ham Lab has no equivalent. Decide whether to build it or to drop it — most
+  Where it stands: Stacks already creates, edits, deploys, updates, restarts, stops, downs
-    stacks here have no row in that table anyway, so dropping it may be honest.
+  and deletes stacks on every machine, and Containers does start/stop/restart, logs that
-  - **Scheduled image updates.** Ham Lab checks for new images only when asked (a registry
+  follow, inspect, and the images/volumes/networks tables. So this is not a rewrite.
-    request Docker Hub counts) and never updates by itself. If anything relies on Dockhand
+  - **What Dockhand still does that Ham Lab does not.**
-    updating on a schedule, that must move here or be dropped on purpose.
+    - *Git-sourced stacks.* Dockhand's `stack_sources` table points a stack at a repo and
-  - **Webhooks / per-host environments**, if either is in use — inventory before deciding.
+      pulls it. Ham Lab has no equivalent. Decide whether to build it or drop it — most
-  - **Deploying from a host path without an orchestrator container.** Ham Lab already runs
+      stacks here have no row in that table anyway, so dropping it may be the honest answer.
-    compose inside `dockhand` because that is how the stacks were deployed; once they move
+    - *Scheduled image updates.* Ham Lab checks for new images only when asked, since each
-    to a neutral home the "via" path is unused for them, but a stack deployed to `grid` or
+      check is a registry request Docker Hub counts, and never updates by itself. If
-    the Mac still goes over `tcp://<node>:2375` from the hub. Decide whether Ham Lab should
+      anything relies on Dockhand updating on a schedule, that must move here or be dropped
-    instead ask the node's own Ham Lab to deploy locally, which would end the `*:2375`
+      on purpose. Note the Jobs tab and the ZFS schedules already show the shape this would
-    exposure (its own item in `docs/pending.md`).
+      take: a cron line Ham Lab owns, not a ticker in the daemon.
-  - *Not app work, but the trigger:* nothing here blocks starting the migration except the
+    - *Webhooks and per-host environments*, if either is in use. Inventory before deciding.
+  - **Known gaps in the tab itself**, worth closing whether or not Dockhand goes:
+    - *Removing networks* — volumes and images can be pruned and removed, networks cannot.
+    - *Container detail is thin next to the stack view.* No per-container resource limits,
+      restart-policy editing, or exec-into-container from the tab (the Terminal tab is the
+      workaround). Decide which of these belong here at all.
+    - *Logs are polled with `--since` rather than followed over a held connection.* That was
+      deliberate and works, but a long tail is choppier than the Services tab's live tail,
+      which does hold a connection. Worth reconciling the two.
+    - *Image update checks are remembered six hours and asked for per stack*, so there is no
+      one view of "what is out of date across the fleet".
+  - **The `*:2375` question, which this is the moment to settle.** Ham Lab runs compose
+    inside `dockhand` because that is how the stacks were deployed; once they move to a
+    neutral home that path is unused for them, but a stack deployed to `grid` or the Mac
+    still goes over `tcp://<node>:2375` from the hub. Decide whether the hub should instead
+    ask the node's own Ham Lab to deploy locally, which would end the unauthenticated
+    Docker socket exposure — its own item in `docs/pending.md`, currently filed under
+    "decided, no action", and this is the change that would make closing it free.
+  - *Not app work, but the trigger:* nothing above blocks starting the migration except the
     inventory. Do that first.
 
-- **Rename the Space tab** — asked for 2026-09-23, not started. "Space" is vague; the tab
-  is WizTree for a disk or pool. Candidates: **Usage**, **Disk Usage**, **Storage**. Note
-  `Disks` already exists and `Storage` would muddle with it, so Usage is probably the one.
-  The rename touches the rail label, the `View` id, the hash route (`#/<machine>/space`,
-  which people may have bookmarked — decide whether to keep `space` as the id and change
-  only the label), the footer's page name and `Space.tsx`'s own strings.
-
-- **ZFS gets its own tab, under Disks** — asked for 2026-09-23, not started. Today ZFS is a
-  section inside `Disks.tsx`: pools with health, capacity, fragmentation, scrub controls, a
-  dataset tree and per-dataset snapshot lists with "Snapshot now". Moving it out makes room
-  for what is actually wanted: **scheduled snapshots with a retention policy**, configured
-  from the console rather than by hand. What exists to build on — `grid` runs `zfSnap`
-  nightly at 02:30 from root cron with 6-week expiry (`zfsnap-cron-6w-daily.sh`, predates
-  this repo), and Ham Lab's Cron tab can already read, write and run root crontabs, so the
-  job can be managed rather than reinvented. Decide: does Ham Lab drive `zfSnap`/`zfs-auto-
-  snapshot` if present, or write and own its own timer? Per-dataset policies or one for the
-  pool? And it must show what a retention rule *will* delete before it runs.
-
-- **Rename Cron to something generic** — asked for 2026-09-23, not started. Mitchell's
-  reasoning: the Services tab is systemd but "Services" is the right generic name for it,
-  so the scheduling tab should be named for the job it does, not the tool. **Scheduled** or
-  **Jobs**. The tab already covers more than cron — `/etc/crontab`, `/etc/cron.d`, user
-  crontabs, `run-parts` directories *and* systemd timers — so the generic name is the
-  honest one. Same rename surface as the Space tab: rail label, `View` id, hash route,
-  footer name.
-
 - **A Virtualization tab** — asked for 2026-09-23, not started, and **large**. Run and
   manage **LXC containers and virtual machines** from the console, the way Proxmox does —
   which is the comparison Ham Lab has invited since its first line. Nothing exists yet.
@@ -548,44 +618,6 @@ something long in one, switch away and back, refresh — it's all still there.
   `internal/term`); snapshots and clones; and what it does on a host with no virtualisation
   at all, which is every host here today. Worth scoping as its own plan before starting.
 
-- **Files explorer: cleanup and ease of use** — asked for 2026-09-23, not started.
-  General polish and ease of use. The **Deprecate copyparty**
-  item names the heavier pieces (multi-select and bulk actions, folder download, resumable
-  uploads, a gallery, search) — worth doing the two together, since they are the same tab.
-
-- **A KVM tab** — asked for 2026-09-24, not started. **Dead simple is the brief**: paste one
-  HTTP link, the tab shows whatever UI that KVM serves, and the keyboard goes to it. **This
-  is KVM the keyboard-video-mouse switch, not KVM the hypervisor** — the Virtualization item
-  above means the other one.
-  - **The device**: a **Sipeed NanoKVM** attached to `grid` at **`10.0.0.217`** (MAC
-    `48:da:35:6f:d0:31`), web UI on port 80. The only one so far, so store a list, not one URL.
-  - **Framing works.** Probed 2026-09-24: it sends **no `X-Frame-Options` and no
-    `frame-ancestors`**, so it does not refuse to be embedded. No fallback needed.
-  - **Proxy it through the daemon — that is what makes the simple version possible.** The
-    KVM is **HTTP only**, and Ham Lab is served over TLS, so a bare `http://…` iframe is
-    blocked as mixed content and the "just paste a link" idea dies there. Serving it from Ham
-    Lab's own origin (`/api/kvm/<id>/…`, on the `proxyNode` machinery, which already carries
-    WebSockets — exactly what the video and input streams are) fixes three things at once:
-    no mixed content, the KVM sits behind **Ham Lab's** login rather than a second one
-    (`/api/vm/info` answers 401, so it has its own, which the hub can hold once), and the
-    iframe becomes **same-origin**, which matters for the keyboard below.
-  - **The keyboard needs less than it sounds.** A parent page *cannot* capture keys and
-    forward them into a **cross-origin** iframe — the browser forbids synthetic events across
-    origins — but it does not need to: an iframe with focus receives keystrokes directly, and
-    the NanoKVM's own page already handles them. So "capture and give it to the iframe" is
-    really: click-to-focus, keep focus there, and stop the console's own shortcuts from
-    eating keys while it is focused. What is genuinely worth adding is **fullscreen** plus
-    `navigator.keyboard.lock()`, which is what stops the *browser* swallowing Escape, Tab and
-    Ctrl-W so they reach the guest — note it is Chromium-only, so Firefox and iOS Safari get
-    the plain version. Proxying makes the iframe same-origin anyway, so real forwarding stays
-    available if some KVM ever needs it.
-  - **Where the list lives is the one real design decision.** A KVM earns its keep when the
-    machine is *down* — no OS, a bad kernel, a BIOS screen — which is when that host's Ham Lab
-    cannot answer. So it is **hub-side** and must work for machines running no Ham Lab at all.
-    A `kvm_url` in each node's own `config.json` would fail in exactly the case it is for.
-  - **Do not give it an NPM host** without deciding to: `*.bigham.dev` is a wildcard to the
-    public IP, so that would put BIOS-level control of `grid` on the internet behind only the
-    KVM's own login. Behind Ham Lab's proxy it inherits Ham Lab's auth instead.
 
 ## Open questions
 

cmd/hamlab/main.go

@@ -4,6 +4,7 @@ package main
 import (
 	"bufio"
 	"context"
+	"encoding/json"
 	"errors"
 	"flag"
 	"fmt"
@@ -32,6 +33,7 @@ import (
 	"hamlab/internal/metrics"
 	"hamlab/internal/nodes"
 	"hamlab/internal/services"
+	"hamlab/internal/priv"
 	"hamlab/internal/setup"
 	"hamlab/internal/space"
 	"hamlab/internal/storage"
@@ -59,6 +61,7 @@ func main() {
   serve                run the console in the foreground (what the service runs)
   passwd [user]        set a user's password (makes the account if it is new; reads stdin when piped)
   users                list the accounts
+  zfs-snapshot <id>    run one ZFS snapshot schedule (what its cron line calls)
   user-rm <user>       remove an account and sign it out everywhere
   check                verify the config and state load, then exit
   version
@@ -129,6 +132,10 @@ func main() {
 		if err := passwd(cfg, flag.Arg(1)); err != nil {
 			log.Fatal(err)
 		}
+	case "zfs-snapshot":
+		if err := zfsSnapshot(cfg, flag.Arg(1)); err != nil {
+			log.Fatal(err)
+		}
 	case "users":
 		migrateUsers(cfg)
 		if len(cfg.Users) == 0 {
@@ -178,6 +185,58 @@ func main() {
 
 // migrateUsers gives a pre-users config its first account, named after this OS user, so
 // the CLI sees the same accounts the daemon will.
+// zfsSnapshot runs one snapshot schedule: take it, then expire what has aged out. This is
+// what the cron line Ham Lab writes actually calls, so a schedule keeps running whether or
+// not the console is up — and keeps running if Ham Lab is never opened again.
+//
+// It deliberately does not write state.json. The daemon owns that file and would clobber
+// anything written from out here; the outcome goes beside it instead, in zfs-runs/.
+func zfsSnapshot(cfg *config.Config, id string) error {
+	if strings.TrimSpace(id) == "" {
+		return errors.New("usage: hamlab zfs-snapshot <schedule-id>")
+	}
+	st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
+	if err != nil {
+		return err
+	}
+	var raw json.RawMessage
+	st.View(func(s *store.State) { raw = s.ZFS })
+	var z struct {
+		Policies []storage.Policy `json:"policies"`
+	}
+	if len(raw) > 0 {
+		if err := json.Unmarshal(raw, &z); err != nil {
+			return fmt.Errorf("reading the schedules: %w", err)
+		}
+	}
+	var p *storage.Policy
+	for i := range z.Policies {
+		if z.Policies[i].ID == id {
+			p = &z.Policies[i]
+		}
+	}
+	if p == nil {
+		return fmt.Errorf("no snapshot schedule %q — it may have been removed; this cron line is stale", id)
+	}
+	res, runErr := storage.New(priv.Available()).Run(context.Background(), p)
+	rec := storage.RunRecord{At: time.Now(), Taken: res.Taken, Pruned: res.Pruned}
+	if runErr != nil {
+		rec.Error = runErr.Error()
+	}
+	if err := storage.WriteRun(cfg.DataDir(), id, rec); err != nil {
+		fmt.Fprintln(os.Stderr, "recording the run:", err)
+	}
+	if runErr != nil {
+		return runErr
+	}
+	fmt.Printf("%s: took %s", p.Name, res.Taken)
+	if len(res.Pruned) > 0 {
+		fmt.Printf(", expired %d", len(res.Pruned))
+	}
+	fmt.Println()
+	return nil
+}
+
 func migrateUsers(cfg *config.Config) {
 	name := os.Getenv("USER")
 	if u, err := user.Current(); err == nil && u.Username != "" {

internal/api/docker.go

@@ -394,3 +394,161 @@ func (s *Server) volumePrune(w http.ResponseWriter, r *http.Request) {
 	log.Printf("docker: volume prune (from %s)", clientIP(r))
 	jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
 }
+
+// ---- a container's health, its recent life, its console, and its limits ----
+
+// dockerHealth is the health check's verdict and the last few probe results. Docker keeps
+// the last five itself, which is all anyone wants: is it failing, and what did it say.
+func (s *Server) dockerHealth(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	if !dockerRef.MatchString(id) {
+		jsonErr(w, 400, "bad container")
+		return
+	}
+	ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
+	defer cancel()
+	out, err := dockerRun(ctx, "inspect", "--format", "{{json .State}}", id)
+	if err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	var st struct {
+		Status string `json:"Status"`
+		Health *struct {
+			Status        string `json:"Status"`
+			FailingStreak int    `json:"FailingStreak"`
+			Log           []struct {
+				Start    time.Time `json:"Start"`
+				End      time.Time `json:"End"`
+				ExitCode int       `json:"ExitCode"`
+				Output   string    `json:"Output"`
+			} `json:"Log"`
+		} `json:"Health"`
+	}
+	if json.Unmarshal(out, &st) != nil {
+		jsonErr(w, 500, "could not read the container's state")
+		return
+	}
+	if st.Health == nil {
+		jsonOK(w, 200, map[string]any{"configured": false, "state": st.Status})
+		return
+	}
+	jsonOK(w, 200, map[string]any{
+		"configured": true, "state": st.Status, "status": st.Health.Status,
+		"failing_streak": st.Health.FailingStreak, "log": st.Health.Log,
+	})
+}
+
+// dockerExec opens a shell inside a container, as a terminal session like any other — so
+// it reattaches, resizes and scrolls back the same way, and works on a managed machine
+// through the same proxy.
+func (s *Server) dockerExec(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	if !dockerRef.MatchString(id) {
+		jsonErr(w, 400, "bad container")
+		return
+	}
+	tool := dockerTool()
+	if tool == "" {
+		jsonErr(w, 500, "neither docker nor podman is installed here")
+		return
+	}
+	var body struct {
+		Shell string `json:"shell"`
+	}
+	_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
+	shell := strings.TrimSpace(body.Shell)
+	if shell == "" {
+		// most images have sh; bash is nicer where it exists, so try it first and fall back
+		shell = "if command -v bash >/dev/null 2>&1; then exec bash; else exec sh; fi"
+	} else if strings.ContainsAny(shell, "\n\r") {
+		jsonErr(w, 400, "bad shell")
+		return
+	}
+	argv := []string{tool, "exec", "-it", id, "sh", "-c", shell}
+	if priv.Available() && !dockerReachable() {
+		argv = priv.Wrap(argv...)
+	}
+	sess, err := s.mgr.CreateCmd("", "sh "+shortName(id), argv)
+	if err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	log.Printf("docker exec into %s as terminal %s by %s", id, sess.ID, clientIP(r))
+	jsonOK(w, 200, map[string]string{"id": sess.ID, "name": "sh " + shortName(id)})
+}
+
+func shortName(id string) string {
+	if len(id) > 20 {
+		return id[:20]
+	}
+	return id
+}
+
+// dockerReachable reports whether docker answers without help.
+func dockerReachable() bool {
+	tool := dockerTool()
+	if tool == "" {
+		return false
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	return exec.CommandContext(ctx, tool, "version", "--format", "{{.Server.Version}}").Run() == nil
+}
+
+// dockerUpdate changes a running container's limits and restart policy. `docker update`
+// does this live, without recreating it and without touching the compose file — which is
+// also the catch, and the caller is told: a deploy from compose puts back whatever compose
+// says.
+func (s *Server) dockerUpdate(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	if !dockerRef.MatchString(id) {
+		jsonErr(w, 400, "bad container")
+		return
+	}
+	var body struct {
+		Memory  string `json:"memory"`  // "512m", "2g", or "" to leave alone; "0" removes the limit
+		CPUs    string `json:"cpus"`    // "1.5", or "0" for no limit
+		Restart string `json:"restart"` // no | on-failure | unless-stopped | always
+	}
+	if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
+		jsonErr(w, 400, "bad request")
+		return
+	}
+	args := []string{"update"}
+	if m := strings.TrimSpace(body.Memory); m != "" {
+		if !regexp.MustCompile(`^[0-9]+([bkmgBKMG])?$`).MatchString(m) {
+			jsonErr(w, 400, "memory: a number, optionally with b, k, m or g — 0 removes the limit")
+			return
+		}
+		args = append(args, "--memory", m, "--memory-swap", m)
+	}
+	if c := strings.TrimSpace(body.CPUs); c != "" {
+		if !regexp.MustCompile(`^[0-9]+(\.[0-9]+)?$`).MatchString(c) {
+			jsonErr(w, 400, "cpus: a number like 1.5 — 0 removes the limit")
+			return
+		}
+		args = append(args, "--cpus", c)
+	}
+	if p := strings.TrimSpace(body.Restart); p != "" {
+		switch p {
+		case "no", "on-failure", "unless-stopped", "always":
+		default:
+			jsonErr(w, 400, "restart: no, on-failure, unless-stopped or always")
+			return
+		}
+		args = append(args, "--restart", p)
+	}
+	if len(args) == 1 {
+		jsonErr(w, 400, "nothing to change")
+		return
+	}
+	ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
+	defer cancel()
+	if _, err := dockerRun(ctx, append(args, id)...); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	log.Printf("docker update %s %v by %s", id, args[1:], clientIP(r))
+	w.WriteHeader(204)
+}

internal/api/docker_events.go

@@ -0,0 +1,229 @@
+package api
+
+// A container's history.
+//
+// `docker events` is a live stream with only a short in-memory backlog — ask it for the
+// last thirty days and you get the last few minutes, because the daemon keeps events in a
+// small ring and loses them all when it restarts. That is why Dockhand kept its own copy,
+// and why this does too: a watcher follows the stream and appends the lifecycle events to a
+// ring kept beside the config, so "what happened to this container last Tuesday" has an
+// answer.
+//
+// Only lifecycle events are kept. A health check produces three exec events every probe,
+// which is thousands a day and tells you nothing that the health panel does not.
+
+import (
+	"bufio"
+	"context"
+	"encoding/json"
+	"log"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"sort"
+	"strings"
+	"sync"
+	"time"
+
+	"hamlab/internal/priv"
+)
+
+const (
+	eventsKept     = 4000
+	eventsSaveEeem = 30 * time.Second
+)
+
+// ContainerEvent is one thing that happened to one container.
+type ContainerEvent struct {
+	T        int64  `json:"t"` // unix ms
+	Name     string `json:"name"`
+	Action   string `json:"action"`
+	Image    string `json:"image,omitempty"`
+	ExitCode string `json:"exit_code,omitempty"`
+	Signal   string `json:"signal,omitempty"`
+	Health   string `json:"health,omitempty"`
+}
+
+type eventLog struct {
+	mu    sync.Mutex
+	ring  []ContainerEvent
+	dirty bool
+	path  string
+}
+
+func (e *eventLog) load(path string) {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+	e.path = path
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return
+	}
+	var list []ContainerEvent
+	if json.Unmarshal(b, &list) == nil {
+		e.ring = list
+	}
+}
+
+func (e *eventLog) add(ev ContainerEvent) {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+	e.ring = append(e.ring, ev)
+	if len(e.ring) > eventsKept {
+		e.ring = append([]ContainerEvent(nil), e.ring[len(e.ring)-eventsKept:]...)
+	}
+	e.dirty = true
+}
+
+// forName is one container's events, newest first.
+func (e *eventLog) forName(name string, since time.Time, limit int) []ContainerEvent {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+	out := []ContainerEvent{}
+	for _, ev := range e.ring {
+		if ev.Name == name && ev.T >= since.UnixMilli() {
+			out = append(out, ev)
+		}
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].T > out[j].T })
+	if len(out) > limit {
+		out = out[:limit]
+	}
+	return out
+}
+
+func (e *eventLog) save() {
+	e.mu.Lock()
+	if !e.dirty || e.path == "" {
+		e.mu.Unlock()
+		return
+	}
+	b, err := json.Marshal(e.ring)
+	e.dirty = false
+	path := e.path
+	e.mu.Unlock()
+	if err != nil {
+		return
+	}
+	_ = os.MkdirAll(filepath.Dir(path), 0o700)
+	tmp := path + ".tmp"
+	if os.WriteFile(tmp, b, 0o600) == nil {
+		_ = os.Rename(tmp, path)
+	}
+}
+
+// watchDockerEvents follows the stream, restarting if Docker goes away. It is the only
+// thing that writes the log, so nothing contends with it.
+func (s *Server) watchDockerEvents(ctx context.Context) {
+	s.events.load(filepath.Join(s.cfg.DataDir(), "docker-events.json"))
+	go func() { // a periodic flush, so a crash loses at most half a minute of history
+		t := time.NewTicker(eventsSaveEeem)
+		defer t.Stop()
+		for {
+			select {
+			case <-ctx.Done():
+				s.events.save()
+				return
+			case <-t.C:
+				s.events.save()
+			}
+		}
+	}()
+	for {
+		if ctx.Err() != nil {
+			return
+		}
+		if dockerTool() == "" {
+			time.Sleep(time.Minute) // no Docker here; look again later in case it is installed
+			continue
+		}
+		s.followEvents(ctx)
+		select {
+		case <-ctx.Done():
+			return
+		case <-time.After(10 * time.Second): // the daemon restarted, or the pipe broke
+		}
+	}
+}
+
+func (s *Server) followEvents(ctx context.Context) {
+	args := []string{"events", "--format", "{{json .}}", "--filter", "type=container"}
+	for _, e := range []string{"create", "start", "restart", "stop", "kill", "die", "destroy", "oom", "pause", "unpause", "rename", "update", "health_status"} {
+		args = append(args, "--filter", "event="+e)
+	}
+	tool := dockerTool()
+	full := append([]string{tool}, args...)
+	if !dockerReachable() && priv.Available() {
+		full = priv.Wrap(full...)
+	}
+	cmd := exec.CommandContext(ctx, full[0], full[1:]...)
+	out, err := cmd.StdoutPipe()
+	if err != nil {
+		return
+	}
+	if err := cmd.Start(); err != nil {
+		return
+	}
+	sc := bufio.NewScanner(out)
+	sc.Buffer(make([]byte, 64<<10), 1<<20)
+	n := 0
+	for sc.Scan() {
+		var e struct {
+			Action   string `json:"Action"`
+			Time     int64  `json:"time"`
+			TimeNano int64  `json:"timeNano"`
+			Actor    struct {
+				Attributes map[string]string `json:"Attributes"`
+			} `json:"Actor"`
+		}
+		if json.Unmarshal(sc.Bytes(), &e) != nil || e.Action == "" {
+			continue
+		}
+		at := e.Actor.Attributes
+		ev := ContainerEvent{
+			T: e.Time * 1000, Name: at["name"], Image: at["image"],
+			Action: e.Action, ExitCode: at["exitCode"], Signal: at["signal"],
+		}
+		if e.TimeNano > 0 {
+			ev.T = e.TimeNano / 1e6
+		}
+		// health_status comes through as "health_status: healthy"
+		if a, h, ok := strings.Cut(e.Action, ": "); ok {
+			ev.Action, ev.Health = a, h
+		}
+		if ev.Name == "" {
+			continue
+		}
+		s.events.add(ev)
+		n++
+	}
+	_ = cmd.Wait()
+	if n > 0 {
+		log.Printf("docker events: stream ended after %d events; reconnecting", n)
+	}
+}
+
+// dockerEvents answers from Ham Lab's own history, not Docker's short memory.
+func (s *Server) dockerEvents(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	if !dockerRef.MatchString(id) {
+		jsonErr(w, 400, "bad container")
+		return
+	}
+	since := r.URL.Query().Get("since")
+	d := 7 * 24 * time.Hour
+	switch since {
+	case "24h":
+		d = 24 * time.Hour
+	case "30d":
+		d = 30 * 24 * time.Hour
+	default:
+		since = "7d"
+	}
+	ev := s.events.forName(id, time.Now().Add(-d), 500)
+	s.events.mu.Lock()
+	total := len(s.events.ring)
+	s.events.mu.Unlock()
+	jsonOK(w, 200, map[string]any{"events": ev, "since": since, "kept": total, "capacity": eventsKept})
+}

internal/api/kvm.go

@@ -0,0 +1,310 @@
+package api
+
+// IP-KVMs: a device's own web page, shown inside Ham Lab.
+//
+// The whole difficulty is that Ham Lab is served over TLS and a KVM usually is not, so a
+// browser refuses to put the device's page in an iframe — mixed content, blocked before
+// anything loads. The fix is to serve the device from Ham Lab's own origin, which also
+// means it inherits Ham Lab's login instead of needing a second one, and makes the iframe
+// same-origin.
+//
+// Proxying an arbitrary web app under a path prefix is the hard part, and nothing here
+// knows anything about any particular device. A device UI asks for its assets by absolute
+// path — "/assets/app.js", "/api/whatever" — which a prefix cannot rewrite away. So the
+// prefix is only the entry point: any request Ham Lab does not serve itself is routed to
+// whichever KVM the asking page belongs to, worked out from the Referer, and from a cookie
+// when there is no Referer (a WebSocket upgrade, say). Ham Lab's own routes always win, so
+// a device can never shadow the console.
+
+import (
+	"crypto/rand"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"io"
+	"log"
+	"net/http"
+	"net/http/httputil"
+	"net/url"
+	"strings"
+	"sync"
+	"time"
+
+	"hamlab/internal/config"
+)
+
+const kvmPrefix = "/kvm/"
+const kvmCookie = "hamlab_kvm"
+
+type kvmProxies struct {
+	mu sync.Mutex
+	m  map[string]*httputil.ReverseProxy
+}
+
+func (s *Server) kvm(id string) (config.KVM, bool) {
+	var found config.KVM
+	ok := false
+	s.cfg.View(func(c *config.Config) {
+		for _, k := range c.KVMs {
+			if k.ID == id {
+				found, ok = k, true
+			}
+		}
+	})
+	return found, ok
+}
+
+// kvmProxy is the reverse proxy for one device, made once and kept.
+func (s *Server) kvmProxy(k config.KVM) (*httputil.ReverseProxy, error) {
+	s.kvms.mu.Lock()
+	defer s.kvms.mu.Unlock()
+	if s.kvms.m == nil {
+		s.kvms.m = map[string]*httputil.ReverseProxy{}
+	}
+	if p, ok := s.kvms.m[k.ID]; ok {
+		return p, nil
+	}
+	target, err := url.Parse(k.URL)
+	if err != nil || target.Host == "" {
+		return nil, errors.New("this KVM's address is not a URL")
+	}
+	prefix := kvmPrefix + k.ID
+	p := &httputil.ReverseProxy{
+		Director: func(r *http.Request) {
+			r.URL.Scheme, r.URL.Host = target.Scheme, target.Host
+			// Entry through the prefix keeps the path after it; anything else arrived by its
+			// own absolute path and is passed through untouched.
+			if rest, ok := strings.CutPrefix(r.URL.Path, prefix); ok {
+				if rest == "" {
+					rest = "/"
+				}
+				r.URL.Path = rest
+			}
+			r.URL.RawPath = ""
+			r.Host = target.Host
+			// The device gets its own host and none of Ham Lab's session.
+			r.Header.Del("Origin")
+			stripCookie(r, cookieName, kvmCookie)
+		},
+		FlushInterval: -1, // a video stream must not be buffered
+		ModifyResponse: func(resp *http.Response) error {
+			// The device's own framing and transport policy would stop it being embedded, or
+			// send a browser off to plain HTTP. Ham Lab is the origin now, and says what it
+			// serves; its own headers are added on the way out.
+			for _, h := range []string{"X-Frame-Options", "Content-Security-Policy", "Content-Security-Policy-Report-Only", "Strict-Transport-Security", "Referrer-Policy"} {
+				resp.Header.Del(h)
+			}
+			// A redirect to the device's own root has to come back through the prefix.
+			if loc := resp.Header.Get("Location"); loc != "" {
+				if u, err := url.Parse(loc); err == nil && (u.Host == "" || u.Host == target.Host) && strings.HasPrefix(u.Path, "/") && !strings.HasPrefix(u.Path, prefix) {
+					u.Scheme, u.Host = "", ""
+					u.Path = prefix + u.Path
+					resp.Header.Set("Location", u.String())
+				}
+			}
+			return nil
+		},
+		ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
+			w.Header().Set("Content-Type", "application/json")
+			w.WriteHeader(http.StatusBadGateway)
+			_ = json.NewEncoder(w).Encode(map[string]string{"error": k.Name + " unreachable: " + err.Error()})
+		},
+	}
+	s.kvms.m[k.ID] = p
+	return p, nil
+}
+
+// stripCookie removes named cookies from a request, so Ham Lab's session is never handed to
+// a device and the device never sees which KVM the browser is looking at.
+func stripCookie(r *http.Request, names ...string) {
+	cs := r.Cookies()
+	kept := cs[:0]
+	for _, c := range cs {
+		drop := false
+		for _, n := range names {
+			if c.Name == n {
+				drop = true
+			}
+		}
+		if !drop {
+			kept = append(kept, c)
+		}
+	}
+	r.Header.Del("Cookie")
+	for _, c := range kept {
+		r.AddCookie(c)
+	}
+}
+
+func (s *Server) forgetKVMProxy(id string) {
+	s.kvms.mu.Lock()
+	delete(s.kvms.m, id)
+	s.kvms.mu.Unlock()
+}
+
+// serveKVM is the entry point: /kvm/<id>/… . It also drops a cookie naming the device, so
+// that requests the page makes later by absolute path can be traced back to it.
+func (s *Server) serveKVM(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	k, ok := s.kvm(id)
+	if !ok {
+		jsonErr(w, 404, "no such KVM")
+		return
+	}
+	p, err := s.kvmProxy(k)
+	if err != nil {
+		jsonErr(w, 400, err.Error())
+		return
+	}
+	http.SetCookie(w, &http.Cookie{Name: kvmCookie, Value: id, Path: "/", HttpOnly: true, Secure: secure(r), SameSite: http.SameSiteLaxMode})
+	p.ServeHTTP(w, r)
+}
+
+// kvmFor works out which device a request belongs to when it arrived by absolute path
+// rather than through the prefix: the Referer names the page that asked, and a cookie
+// covers what has no Referer at all. Nothing device-specific: it never has to know which
+// paths a given KVM uses.
+func (s *Server) kvmFor(r *http.Request) (config.KVM, bool) {
+	if ref := r.Header.Get("Referer"); ref != "" {
+		if u, err := url.Parse(ref); err == nil {
+			if rest, ok := strings.CutPrefix(u.Path, kvmPrefix); ok {
+				if id, _, _ := strings.Cut(rest, "/"); id != "" {
+					if k, ok := s.kvm(id); ok {
+						return k, true
+					}
+				}
+			}
+		}
+		// asked for by a page that is not a KVM: not ours to answer
+		return config.KVM{}, false
+	}
+	if c, err := r.Cookie(kvmCookie); err == nil && c.Value != "" {
+		return s.kvm(c.Value)
+	}
+	return config.KVM{}, false
+}
+
+// kvmFallback answers a request Ham Lab does not serve itself, when it came from inside a
+// KVM's page. It returns false when the request has nothing to do with a KVM, and the
+// caller carries on with its own 404.
+func (s *Server) kvmFallback(w http.ResponseWriter, r *http.Request) bool {
+	k, ok := s.kvmFor(r)
+	if !ok {
+		return false
+	}
+	p, err := s.kvmProxy(k)
+	if err != nil {
+		return false
+	}
+	p.ServeHTTP(w, r)
+	return true
+}
+
+func machineLabel(m string) string {
+	if m == "" {
+		return "this host"
+	}
+	return m
+}
+
+// ---- the list ----
+
+func (s *Server) listKVMs(w http.ResponseWriter, _ *http.Request) {
+	out := []config.KVM{}
+	s.cfg.View(func(c *config.Config) { out = append(out, c.KVMs...) })
+	jsonOK(w, 200, out)
+}
+
+// kvmForMachine is the KVM attached to a machine, "" being the hub itself.
+func (s *Server) kvmForMachine(machine string) (config.KVM, bool) {
+	var found config.KVM
+	ok := false
+	s.cfg.View(func(c *config.Config) {
+		for _, k := range c.KVMs {
+			if k.Machine == machine {
+				found, ok = k, true
+			}
+		}
+	})
+	return found, ok
+}
+
+func (s *Server) addKVM(w http.ResponseWriter, r *http.Request) {
+	var body struct{ Name, URL, Machine string }
+	if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
+		jsonErr(w, 400, "bad request")
+		return
+	}
+	raw := strings.TrimSpace(body.URL)
+	if raw != "" && !strings.Contains(raw, "://") {
+		raw = "http://" + raw // what you would type: an address, not a scheme
+	}
+	u, err := url.Parse(raw)
+	if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+		jsonErr(w, 400, "address: http(s)://host, or just the host")
+		return
+	}
+	name := strings.TrimSpace(body.Name)
+	if name == "" {
+		name = u.Host
+	}
+	if len(name) > maxNameLen {
+		jsonErr(w, 400, "that name is too long")
+		return
+	}
+	b := make([]byte, 6)
+	if _, err := rand.Read(b); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	machine := strings.TrimSpace(body.Machine)
+	k := config.KVM{ID: hex.EncodeToString(b), Name: name, URL: u.Scheme + "://" + u.Host + strings.TrimSuffix(u.Path, "/"), Machine: machine, Added: time.Now()}
+	// One per machine: a second one for the same host replaces the first rather than sitting
+	// beside it, since the tab shows the machine you are looking at.
+	var replaced string
+	if err := s.cfg.Update(func(c *config.Config) {
+		kept := c.KVMs[:0]
+		for _, old := range c.KVMs {
+			if old.Machine == machine {
+				replaced = old.ID
+				continue
+			}
+			kept = append(kept, old)
+		}
+		c.KVMs = append(kept, k)
+	}); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	if replaced != "" {
+		s.forgetKVMProxy(replaced)
+	}
+	log.Printf("kvm %q added for %s (%s) by %s", k.Name, machineLabel(machine), k.URL, clientIP(r))
+	jsonOK(w, 201, k)
+}
+
+func (s *Server) removeKVM(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	found := false
+	if err := s.cfg.Update(func(c *config.Config) {
+		kept := c.KVMs[:0]
+		for _, k := range c.KVMs {
+			if k.ID == id {
+				found = true
+				continue
+			}
+			kept = append(kept, k)
+		}
+		c.KVMs = kept
+	}); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	if !found {
+		jsonErr(w, 404, "no such KVM")
+		return
+	}
+	s.forgetKVMProxy(id)
+	log.Printf("kvm %s removed by %s", id, clientIP(r))
+	w.WriteHeader(204)
+}

internal/api/server.go

@@ -69,6 +69,8 @@ type Server struct {
 	storage     *storage.Manager
 	space       *space.Manager
 	update      updateState
+	events      eventLog
+	kvms        kvmProxies
 	fsLocal     files.FS
 	fsSudo      files.FS // nil without passwordless sudo
 	nodes       *nodes.Manager
@@ -137,7 +139,7 @@ func New(cfg *config.Config, sessions *auth.Sessions, mgr *term.Manager, col *me
 		go s.keepFleetInStep()
 	}
 	go s.watchForUpdates(context.Background())
-	go s.runSchedules(context.Background())
+	go s.watchDockerEvents(context.Background())
 
 	m := http.NewServeMux()
 	m.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(204) })
@@ -206,6 +208,10 @@ func New(cfg *config.Config, sessions *auth.Sessions, mgr *term.Manager, col *me
 	m.HandleFunc("POST /api/docker/containers/{id}/{action}", s.auth(s.dockerAction))
 	m.HandleFunc("GET /api/docker/containers/{id}/logs", s.auth(s.dockerLogs))
 	m.HandleFunc("GET /api/docker/containers/{id}/inspect", s.auth(s.dockerInspect))
+	m.HandleFunc("GET /api/docker/containers/{id}/health", s.auth(s.dockerHealth))
+	m.HandleFunc("GET /api/docker/containers/{id}/events", s.auth(s.dockerEvents))
+	m.HandleFunc("POST /api/docker/containers/{id}/exec", s.auth(s.dockerExec))
+	m.HandleFunc("POST /api/docker/containers/{id}/limits", s.auth(s.dockerUpdate))
 	m.HandleFunc("GET /api/alerts", s.auth(s.alertsGet))
 	m.HandleFunc("PUT /api/alerts", s.auth(s.alertsPut))
 	m.HandleFunc("POST /api/alerts/test", s.auth(s.alertsTest))
@@ -255,7 +261,6 @@ func New(cfg *config.Config, sessions *auth.Sessions, mgr *term.Manager, col *me
 	m.HandleFunc("GET /api/zfs", s.auth(s.zfsOverview))
 	m.HandleFunc("PUT /api/zfs/schedules", s.auth(s.zfsPolicyPut))
 	m.HandleFunc("DELETE /api/zfs/schedules/{id}", s.auth(s.zfsPolicyDelete))
-	m.HandleFunc("GET /api/zfs/schedules/{id}/expired", s.auth(s.zfsPreview))
 	m.HandleFunc("POST /api/zfs/schedules/{id}/run", s.auth(s.zfsRunNow))
 	m.HandleFunc("POST /api/zfs/snapshots/delete", s.auth(s.zfsSnapshotDelete))
 	m.HandleFunc("GET /api/space", s.auth(s.spaceOverview))
@@ -286,7 +291,20 @@ func New(cfg *config.Config, sessions *auth.Sessions, mgr *term.Manager, col *me
 	m.HandleFunc("PUT /api/prefs/sampler", s.auth(s.putSampler))
 	m.HandleFunc("GET /api/prefs/history", s.auth(s.getHistoryWindow))
 	m.HandleFunc("PUT /api/prefs/history", s.auth(s.putHistoryWindow))
-	m.HandleFunc("/api/", func(w http.ResponseWriter, _ *http.Request) { jsonErr(w, 404, "not found") })
+	m.HandleFunc("GET /api/kvms", s.auth(s.listKVMs))
+	m.HandleFunc("POST /api/kvms", s.auth(s.addKVM))
+	m.HandleFunc("DELETE /api/kvms/{id}", s.auth(s.removeKVM))
+	m.HandleFunc("/kvm/{id}/{rest...}", s.auth(s.serveKVM))
+	m.HandleFunc("/kvm/{id}", s.auth(s.serveKVM))
+	// A path Ham Lab does not serve may still belong to a KVM's page, which asks for its
+	// own assets by absolute path. Ham Lab's own routes matched first, so this can only
+	// catch what would otherwise have been a 404.
+	m.HandleFunc("/api/", s.auth(func(w http.ResponseWriter, r *http.Request) {
+		if s.kvmFallback(w, r) {
+			return
+		}
+		jsonErr(w, 404, "not found")
+	}))
 	m.HandleFunc("/", s.static)
 	s.mux = m
 	return s
@@ -309,9 +327,16 @@ func sensorPath(p string) bool {
 func (s *Server) Handler() http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		h := w.Header()
-		// The console must never be framed. An embedded widget exists to be: it is a page
+		// The console must never be framed. Two exceptions: an embedded widget exists to be
-		// that can only read sensors, and only with a read-only token in its URL.
+		// (a page that can only read sensors, and only with a read-only token in its URL),
-		if r.URL.Path != "/embed/widget" {
+		// and a proxied KVM is framed by the console itself — 'none' forbids that even from
+		// the same origin, so those paths say 'self' instead of opening up.
+		switch {
+		case r.URL.Path == "/embed/widget":
+		case strings.HasPrefix(r.URL.Path, kvmPrefix):
+			h.Set("X-Frame-Options", "SAMEORIGIN")
+			h.Set("Content-Security-Policy", "frame-ancestors 'self'")
+		default:
 			h.Set("X-Frame-Options", "DENY")
 			h.Set("Content-Security-Policy", "frame-ancestors 'none'")
 		}
@@ -357,6 +382,17 @@ func (s *Server) setCookie(w http.ResponseWriter, r *http.Request, tok string, m
 	})
 }
 
+// signedIn reports whether a request carries a live browser session, without the
+// side effects of the auth middleware.
+func (s *Server) signedIn(r *http.Request) bool {
+	c, err := r.Cookie(cookieName)
+	if err != nil {
+		return false
+	}
+	_, ok := s.sessions.Lookup(c.Value, clientKey(r))
+	return ok
+}
+
 // auth accepts the browser session cookie or a bearer API token issued to a hub.
 func (s *Server) auth(next http.HandlerFunc) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
@@ -2447,6 +2483,12 @@ func (s *Server) static(w http.ResponseWriter, r *http.Request) {
 			return
 		}
 	}
+	// A path with no file of ours behind it may belong to a KVM's page, which asks for its
+	// own assets by absolute path. Checked here rather than in the router because this
+	// handler is the catch-all — and gated on a session, since this handler is public.
+	if s.signedIn(r) && s.kvmFallback(w, r) {
+		return
+	}
 	// SPA fallback
 	w.Header().Set("Cache-Control", "no-cache")
 	http.ServeFileFS(w, r, s.ui, "index.html")

internal/api/zfs.go

@@ -16,8 +16,9 @@ import (
 	"io"
 	"log"
 	"net/http"
+	"os"
+	"path/filepath"
 	"strings"
-	"sync"
 	"time"
 
 	"hamlab/internal/storage"
@@ -28,11 +29,6 @@ type zfsState struct {
 	Policies []storage.Policy `json:"policies"`
 }
 
-type zfsEngine struct {
-	s  *Server
-	mu sync.Mutex
-}
-
 func (s *Server) zfsLoad() zfsState {
 	var raw json.RawMessage
 	s.store.View(func(st *store.State) { raw = st.ZFS })
@@ -99,45 +95,22 @@ func randomID() (string, error) {
 	return hex.EncodeToString(b), nil
 }
 
-// ---- the schedule runs itself ----
+// ---- the cron file ----
+//
+// A schedule is a cron line, written to /etc/cron.d/hamlab-zfs, which Ham Lab owns and
+// rewrites whole whenever the schedules change. Nothing in the daemon runs them: snapshots
+// are backup-shaped and should not stop because a web console is down, being upgraded, or
+// uninstalled. The line calls `hamlab zfs-snapshot <id>` as the account Ham Lab runs as,
+// which reaches zfs through sudo exactly as the daemon does.
 
-// runSchedules takes the snapshots that are due. Every 30 s, which is fine for a schedule
+// syncCron puts the current schedules on the machine.
-// whose finest grain is hourly, and cheap when nothing is due.
+func (s *Server) syncCron(ctx context.Context, z zfsState) error {
-func (s *Server) runSchedules(ctx context.Context) {
+	exe, err := os.Executable()
-	t := time.NewTicker(30 * time.Second)
-	defer t.Stop()
-	for {
-		select {
-		case <-ctx.Done():
-			return
-		case <-t.C:
-		}
-		z := s.zfsLoad()
-		now := time.Now()
-		changed := false
-		for i := range z.Policies {
-			p := &z.Policies[i]
-			if !p.Due(now) {
-				continue
-			}
-			res, err := s.storage.Run(ctx, p)
-			p.LastRun = now
-			p.LastTaken, p.Pruned = res.Taken, len(res.Pruned)
 	if err != nil {
-				p.LastError = err.Error()
+		return err
-				log.Printf("zfs schedule %q: %v", p.Name, err)
-			} else {
-				p.LastError = ""
-				log.Printf("zfs schedule %q: took %s, expired %d", p.Name, res.Taken, len(res.Pruned))
-			}
-			changed = true
-		}
-		if changed {
-			if err := s.zfsSave(z); err != nil {
-				log.Printf("zfs schedule: saving: %v", err)
-			}
-		}
 	}
+	exe, _ = filepath.EvalSymlinks(exe)
+	return s.storage.WriteCronFile(ctx, z.Policies, s.user, exe, s.cfg.Path())
 }
 
 // ---- endpoints ----
@@ -145,12 +118,16 @@ func (s *Server) runSchedules(ctx context.Context) {
 func (s *Server) zfsOverview(w http.ResponseWriter, r *http.Request) {
 	z := s.zfsLoad()
 	now := time.Now()
+	dir := s.cfg.DataDir()
 	out := make([]map[string]any, 0, len(z.Policies))
 	for i := range z.Policies {
 		p := z.Policies[i]
-		m := map[string]any{"policy": p}
+		expr, _ := p.CronExpr()
+		m := map[string]any{"policy": p, "cron": expr, "last": storage.ReadRun(dir, p.ID)}
 		if p.Enabled {
-			m["next"] = p.NextRun(now)
+			if n := p.NextRun(now); !n.IsZero() {
+				m["next"] = n
+			}
 		}
 		out = append(out, m)
 	}
@@ -173,7 +150,6 @@ func (s *Server) zfsPolicyPut(w http.ResponseWriter, r *http.Request) {
 	found := false
 	for i := range z.Policies {
 		if z.Policies[i].ID == p.ID && p.ID != "" {
-			p.LastRun, p.LastError, p.LastTaken, p.Pruned = z.Policies[i].LastRun, z.Policies[i].LastError, z.Policies[i].LastTaken, z.Policies[i].Pruned
 			z.Policies[i] = p
 			found = true
 		}
@@ -193,7 +169,12 @@ func (s *Server) zfsPolicyPut(w http.ResponseWriter, r *http.Request) {
 		jsonErr(w, 500, err.Error())
 		return
 	}
-	log.Printf("zfs schedule %q saved by %s (%s, %s, keep %s)", p.Name, clientIP(r), p.Dataset, p.Every, p.KeepFor)
+	if err := s.syncCron(r.Context(), z); err != nil {
+		jsonErr(w, 500, "saved, but writing "+storage.CronFile+" failed: "+err.Error())
+		return
+	}
+	expr, _ := p.CronExpr()
+	log.Printf("zfs schedule %q saved by %s (%s, %q, keep %s)", p.Name, clientIP(r), p.Dataset, expr, p.KeepFor)
 	jsonOK(w, 200, p)
 }
 
@@ -218,6 +199,11 @@ func (s *Server) zfsPolicyDelete(w http.ResponseWriter, r *http.Request) {
 		jsonErr(w, 500, err.Error())
 		return
 	}
+	if err := s.syncCron(r.Context(), z); err != nil {
+		jsonErr(w, 500, "removed, but writing "+storage.CronFile+" failed: "+err.Error())
+		return
+	}
+	storage.ForgetRun(s.cfg.DataDir(), id)
 	log.Printf("zfs schedule %s removed by %s (its snapshots are left alone)", id, clientIP(r))
 	w.WriteHeader(204)
 }
@@ -231,24 +217,6 @@ func (s *Server) zfsPolicy(id string) (storage.Policy, bool) {
 	return storage.Policy{}, false
 }
 
-// zfsPreview is what this schedule would destroy if it ran now. Nothing is deleted.
-func (s *Server) zfsPreview(w http.ResponseWriter, r *http.Request) {
-	p, ok := s.zfsPolicy(r.PathValue("id"))
-	if !ok {
-		jsonErr(w, 404, "no such schedule")
-		return
-	}
-	doomed, err := s.storage.Doomed(r.Context(), &p)
-	if err != nil {
-		jsonErr(w, 500, err.Error())
-		return
-	}
-	if doomed == nil {
-		doomed = []storage.Snapshot{}
-	}
-	jsonOK(w, 200, map[string]any{"expired": doomed})
-}
-
 func (s *Server) zfsRunNow(w http.ResponseWriter, r *http.Request) {
 	id := r.PathValue("id")
 	p, ok := s.zfsPolicy(id)
@@ -257,18 +225,11 @@ func (s *Server) zfsRunNow(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	res, err := s.storage.Run(r.Context(), &p)
-	now := time.Now()
+	rec := storage.RunRecord{At: time.Now(), Taken: res.Taken, Pruned: res.Pruned}
-	z := s.zfsLoad()
-	for i := range z.Policies {
-		if z.Policies[i].ID == id {
-			z.Policies[i].LastRun, z.Policies[i].LastTaken, z.Policies[i].Pruned = now, res.Taken, len(res.Pruned)
-			z.Policies[i].LastError = ""
 	if err != nil {
-				z.Policies[i].LastError = err.Error()
+		rec.Error = err.Error()
 	}
-		}
+	_ = storage.WriteRun(s.cfg.DataDir(), id, rec)
-	}
-	_ = s.zfsSave(z)
 	if err != nil {
 		jsonErr(w, 500, err.Error())
 		return

internal/config/config.go

@@ -61,6 +61,23 @@ type UpdateSource struct {
 	Check string `json:"check,omitempty"`
 }
 
+// KVM is an IP-KVM: a device with a web page that gives screen, keyboard and mouse access
+// to a machine, including when that machine has no OS running. Ham Lab holds the list on
+// the hub and proxies each one, because a KVM matters exactly when the machine it is
+// attached to cannot answer for itself.
+type KVM struct {
+	ID   string `json:"id"`
+	Name string `json:"name"`
+	// URL is the device's own address, as you would type it into a browser on the LAN.
+	URL string `json:"url"`
+	// Machine is the host this KVM is attached to, and is the key: one KVM per machine, shown
+	// on that machine's KVM tab. Empty means the hub itself. The list lives on the hub even
+	// for a node's KVM, because a KVM earns its keep when the machine is down and cannot
+	// answer for itself.
+	Machine string    `json:"machine"`
+	Added   time.Time `json:"added"`
+}
+
 type Config struct {
 	Listen string `json:"listen"`
 	// PasswordHash is the single login of installs before users existed. On load it becomes
@@ -70,6 +87,7 @@ type Config struct {
 	TLS          string  `json:"tls,omitempty"` // "auto" → self-signed cert in the data dir; "" → plain HTTP
 	Tokens       []Token `json:"tokens,omitempty"`
 	Nodes        []Node  `json:"nodes,omitempty"`
+	KVMs         []KVM   `json:"kvms,omitempty"`
 	// Updates: where a newer build comes from. See UpdateSource.
 	Updates UpdateSource `json:"update,omitempty"`
 

internal/cron/schedule.go

@@ -132,6 +132,17 @@ func (s *schedule) dayMatches(t time.Time) bool {
 	}
 }
 
+// NextRun is the first time strictly after `from` that a cron expression fires. Exported so
+// anything that writes a cron line — the ZFS snapshot schedules — can say when it runs next
+// using the same evaluator that reads everyone else's.
+func NextRun(expr string, from time.Time) (time.Time, bool) { return nextRun(expr, from) }
+
+// ValidSchedule reports whether a cron expression parses.
+func ValidSchedule(expr string) error {
+	_, err := parseSchedule(expr)
+	return err
+}
+
 // nextRun finds the first time strictly after `from` that the schedule fires.
 func nextRun(expr string, from time.Time) (time.Time, bool) {
 	s, err := parseSchedule(expr)

internal/storage/snapshots.go

@@ -18,13 +18,21 @@ package storage
 // to it. And every policy can be asked what it would delete, before it deletes anything.
 
 import (
+	"bytes"
 	"context"
+	"encoding/json"
 	"errors"
 	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
 	"sort"
 	"strconv"
 	"strings"
 	"time"
+
+	"hamlab/internal/cron"
+	"hamlab/internal/priv"
 )
 
 // stampLayout is zfSnap's timestamp: 2026-09-24_02.30.01.
@@ -41,12 +49,15 @@ type Policy struct {
 	Dataset string `json:"dataset"`
 	// Recursive takes one atomic snapshot of the dataset and everything under it (zfs -r).
 	Recursive bool `json:"recursive"`
-	// Every is hourly | daily | weekly | monthly. At is "HH:MM" (the minute alone is used
+	// Every is hourly | daily | weekly | monthly | custom. At is "HH:MM" (the minute alone
-	// for hourly). Weekday is 0=Sunday, Day is 1–28.
+	// is used for hourly). Weekday is 0=Sunday, Day is 1–28. With "custom", Cron is a plain
+	// cron expression and the rest is ignored — the escape hatch for anything the four
+	// common shapes cannot say.
 	Every   string `json:"every"`
 	At      string `json:"at"`
 	Weekday int    `json:"weekday"`
 	Day     int    `json:"day"`
+	Cron    string `json:"cron,omitempty"`
 	// KeepFor is how long a snapshot lives: "6w", "30d", "12h". It is written into the
 	// snapshot's own name, zfSnap-style, so each one carries the rule it was taken under.
 	// KeepMin is a floor — that many newest snapshots survive however old they are, so a
@@ -56,10 +67,6 @@ type Policy struct {
 	Prefix  string `json:"prefix"`
 	Enabled bool   `json:"enabled"`
 
-	LastRun   time.Time `json:"last_run,omitempty"`
-	LastError string    `json:"last_error,omitempty"`
-	LastTaken string    `json:"last_taken,omitempty"` // the snapshot the last run made
-	Pruned    int       `json:"pruned,omitempty"`     // how many the last run destroyed
 }
 
 func (p *Policy) prefix() string { return strings.TrimSpace(p.Prefix) }
@@ -74,8 +81,15 @@ func (p *Policy) Validate() error {
 	}
 	switch p.Every {
 	case "hourly", "daily", "weekly", "monthly":
+	case "custom":
+		if strings.TrimSpace(p.Cron) == "" {
+			return errors.New("a custom schedule needs a cron expression")
+		}
 	default:
-		return errors.New("every: hourly, daily, weekly or monthly")
+		return errors.New("every: hourly, daily, weekly, monthly or custom")
+	}
+	if _, err := p.CronExpr(); err != nil {
+		return err
 	}
 	if _, _, err := p.clock(); err != nil {
 		return err
@@ -144,67 +158,50 @@ func ParseAge(s string) (time.Duration, error) {
 	return 0, fmt.Errorf("%q: the unit is m, h, d or w", s)
 }
 
-// lastDue is the most recent moment this policy was supposed to run, at or before now.
+// CronExpr renders the schedule as a cron expression — which is what actually runs it. The
-// A policy whose LastRun is older than that is due — so a machine that was off through
+// four common shapes are sugar over this; a custom policy states it outright. One expression
-// its window takes one snapshot when it comes back, rather than skipping silently.
+// is the only description of when a schedule fires, so the page, the cron file and the next
-func (p *Policy) lastDue(now time.Time) time.Time {
+// run can never disagree.
-	hour, min, _ := p.clock()
+func (p *Policy) CronExpr() (string, error) {
-	y, mo, d := now.Date()
+	if p.Every == "custom" {
-	loc := now.Location()
+		expr := strings.Join(strings.Fields(p.Cron), " ")
+		if err := cron.ValidSchedule(expr); err != nil {
+			return "", fmt.Errorf("cron expression: %w", err)
+		}
+		if strings.HasPrefix(expr, "@reboot") {
+			return "", errors.New("@reboot is not a schedule for snapshots")
+		}
+		return expr, nil
+	}
+	hour, min, err := p.clock()
+	if err != nil {
+		return "", err
+	}
 	switch p.Every {
 	case "hourly":
-		t := time.Date(y, mo, d, now.Hour(), min, 0, 0, loc)
+		return fmt.Sprintf("%d * * * *", min), nil
-		if t.After(now) {
-			t = t.Add(-time.Hour)
-		}
-		return t
 	case "daily":
-		t := time.Date(y, mo, d, hour, min, 0, 0, loc)
+		return fmt.Sprintf("%d %d * * *", min, hour), nil
-		if t.After(now) {
-			t = t.AddDate(0, 0, -1)
-		}
-		return t
 	case "weekly":
-		t := time.Date(y, mo, d, hour, min, 0, 0, loc)
+		return fmt.Sprintf("%d %d * * %d", min, hour, p.Weekday), nil
-		back := (int(t.Weekday()) - p.Weekday + 7) % 7
-		t = t.AddDate(0, 0, -back)
-		if t.After(now) {
-			t = t.AddDate(0, 0, -7)
-		}
-		return t
 	case "monthly":
-		t := time.Date(y, mo, p.Day, hour, min, 0, 0, loc)
+		return fmt.Sprintf("%d %d %d * *", min, hour, p.Day), nil
-		if t.After(now) {
-			t = t.AddDate(0, -1, 0)
 	}
-		return t
+	return "", errors.New("unknown schedule")
-	}
-	return time.Time{}
 }
 
-// NextRun is when this policy runs next, for the UI to show.
+// NextRun is when cron will next fire this schedule, read with the same evaluator the Jobs
+// tab uses on everyone else's lines.
 func (p *Policy) NextRun(now time.Time) time.Time {
-	last := p.lastDue(now)
+	expr, err := p.CronExpr()
-	switch p.Every {
+	if err != nil {
-	case "hourly":
-		return last.Add(time.Hour)
-	case "daily":
-		return last.AddDate(0, 0, 1)
-	case "weekly":
-		return last.AddDate(0, 0, 7)
-	case "monthly":
-		return last.AddDate(0, 1, 0)
-	}
 		return time.Time{}
-}
-
-// Due reports whether the policy should run now.
-func (p *Policy) Due(now time.Time) bool {
-	if !p.Enabled {
-		return false
 	}
-	d := p.lastDue(now)
+	t, ok := cron.NextRun(expr, now)
-	return !d.IsZero() && p.LastRun.Before(d)
+	if !ok {
+		return time.Time{}
+	}
+	return t
 }
 
 // SnapTime reads a snapshot's name in zfSnap's convention and returns when it was taken,
@@ -349,3 +346,131 @@ func (m *Manager) Run(ctx context.Context, p *Policy) (RunResult, error) {
 	}
 	return res, nil
 }
+
+// ---- what happened when it ran ----
+//
+// The policies are a definition the daemon owns; what a run did is written by whatever ran
+// it, which is a cron job in its own process. Keeping the two apart means the CLI never has
+// to write state.json behind the daemon's back — the last writer would win and one of them
+// would lose its work.
+
+// RunRecord is the outcome of one run of one schedule.
+type RunRecord struct {
+	At     time.Time `json:"at"`
+	Taken  string    `json:"taken,omitempty"`
+	Pruned []string  `json:"pruned,omitempty"`
+	Error  string    `json:"error,omitempty"`
+}
+
+func runPath(dir, id string) string { return filepath.Join(dir, "zfs-runs", id+".json") }
+
+// WriteRun records what a run did, beside the config.
+func WriteRun(dir, id string, r RunRecord) error {
+	if err := os.MkdirAll(filepath.Join(dir, "zfs-runs"), 0o700); err != nil {
+		return err
+	}
+	b, err := json.Marshal(r)
+	if err != nil {
+		return err
+	}
+	p := runPath(dir, id)
+	tmp := p + ".tmp"
+	if err := os.WriteFile(tmp, b, 0o600); err != nil {
+		return err
+	}
+	return os.Rename(tmp, p)
+}
+
+// ReadRun is the last thing a schedule did, or the zero value if it has never run.
+func ReadRun(dir, id string) RunRecord {
+	var r RunRecord
+	b, err := os.ReadFile(runPath(dir, id))
+	if err == nil {
+		_ = json.Unmarshal(b, &r)
+	}
+	return r
+}
+
+// ForgetRun drops a schedule's record, for when the schedule itself goes.
+func ForgetRun(dir, id string) { _ = os.Remove(runPath(dir, id)) }
+
+// ---- the cron file ----
+
+// CronFile is where Ham Lab writes the snapshot schedules. One file it owns entirely and
+// rewrites whole, so there is never a half-updated set of lines, and so a schedule removed
+// in the UI is removed from the machine.
+const CronFile = "/etc/cron.d/hamlab-zfs"
+
+// CronFileText renders every schedule as cron lines. user is who they run as — the account
+// Ham Lab itself runs as, which reaches zfs through sudo exactly as the daemon does, so the
+// files it writes stay owned by that account rather than by root.
+func CronFileText(policies []Policy, user, exe, config string) (string, error) {
+	var b strings.Builder
+	b.WriteString("# Ham Lab snapshot schedules — written by the ZFS tab, rewritten whole on every change.\n")
+	b.WriteString("# Edit them there, not here: anything added by hand is lost the next time a schedule changes.\n")
+	b.WriteString("SHELL=/bin/sh\n")
+	b.WriteString("PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n\n")
+	for i := range policies {
+		p := &policies[i]
+		if !p.Enabled {
+			b.WriteString("# " + p.Name + " — paused\n")
+			continue
+		}
+		expr, err := p.CronExpr()
+		if err != nil {
+			return "", fmt.Errorf("%s: %w", p.Name, err)
+		}
+		name := strings.ReplaceAll(p.Name, "\n", " ")
+		fmt.Fprintf(&b, "# %s — %s, keep %s\n", name, p.Dataset, p.KeepFor)
+		fmt.Fprintf(&b, "%s %s %s -config %s zfs-snapshot %s\n\n", expr, user, exe, config, p.ID)
+	}
+	return b.String(), nil
+}
+
+// WriteCronFile puts the schedules on the machine, or removes the file when there are none.
+func (m *Manager) WriteCronFile(ctx context.Context, policies []Policy, user, exe, config string) error {
+	live := false
+	for i := range policies {
+		if policies[i].Enabled {
+			live = true
+		}
+	}
+	if !live {
+		out, code, err := m.run(ctx, "rm", "-f", CronFile)
+		if err != nil {
+			return err
+		}
+		if code != 0 {
+			return errors.New(strings.TrimSpace(string(out)))
+		}
+		return nil
+	}
+	text, err := CronFileText(policies, user, exe, config)
+	if err != nil {
+		return err
+	}
+	return m.writeRoot(ctx, CronFile, text)
+}
+
+// writeRoot puts text at a root-owned path, through the same privilege escalation every
+// other write here uses.
+func (m *Manager) writeRoot(ctx context.Context, path, text string) error {
+	args := []string{"tee", path}
+	if m.sudo {
+		args = priv.Wrap(args...)
+	}
+	ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
+	defer cancel()
+	cmd := exec.CommandContext(ctx, args[0], args[1:]...)
+	cmd.Stdin = strings.NewReader(text)
+	var errb bytes.Buffer
+	cmd.Stdout, cmd.Stderr = nil, &errb
+	if err := cmd.Run(); err != nil {
+		msg := strings.TrimSpace(errb.String())
+		if msg == "" {
+			msg = err.Error()
+		}
+		return errors.New(msg)
+	}
+	return nil
+}

internal/storage/snapshots_test.go

@@ -1,8 +1,11 @@
 package storage
 
 import (
+	"strings"
 	"testing"
 	"time"
+
+	"hamlab/internal/cron"
 )
 
 func TestSnapTimeReadsZfSnapNames(t *testing.T) {
@@ -33,39 +36,62 @@ func TestSnapTimeReadsZfSnapNames(t *testing.T) {
 	}
 }
 
-func TestScheduleIsDueAndNext(t *testing.T) {
+func TestCronExprAndNextRun(t *testing.T) {
 	now := time.Date(2026, 9, 24, 9, 15, 0, 0, time.Local)
-	daily := &Policy{Every: "daily", At: "02:30", Enabled: true}
+	for _, c := range []struct {
-	if d := daily.lastDue(now); d.Hour() != 2 || d.Day() != 24 {
+		p    Policy
-		t.Fatalf("daily lastDue = %v", d)
+		want string
+	}{
+		{Policy{Every: "daily", At: "02:30"}, "30 2 * * *"},
+		{Policy{Every: "hourly", At: "00:30"}, "30 * * * *"},
+		{Policy{Every: "weekly", At: "04:00", Weekday: 0}, "0 4 * * 0"},
+		{Policy{Every: "monthly", At: "05:15", Day: 12}, "15 5 12 * *"},
+		{Policy{Every: "custom", Cron: "*/15  9-17 * * 1-5"}, "*/15 9-17 * * 1-5"},
+	} {
+		got, err := c.p.CronExpr()
+		if err != nil || got != c.want {
+			t.Errorf("CronExpr(%+v) = %q, %v; want %q", c.p.Every, got, err, c.want)
 		}
-	if !daily.Due(now) {
-		t.Fatal("never run: must be due")
 	}
-	daily.LastRun = now.Add(-time.Hour) // ran at 08:15 today, after 02:30
+	// the next run is read with the same evaluator the Jobs tab uses
-	if daily.Due(now) {
+	daily := Policy{Every: "daily", At: "02:30"}
-		t.Fatal("already ran since the window: not due")
+	if n := daily.NextRun(now); n.Hour() != 2 || n.Minute() != 30 || n.Day() != 25 {
+		t.Fatalf("NextRun = %v, want 25th 02:30", n)
 	}
-	if n := daily.NextRun(now); n.Day() != 25 || n.Hour() != 2 {
+	// a custom expression that does not parse is refused, not silently ignored
-		t.Fatalf("daily NextRun = %v", n)
+	if _, err := (&Policy{Every: "custom", Cron: "not a schedule"}).CronExpr(); err == nil {
+		t.Fatal("a bad cron expression must be refused")
 	}
-	// a machine that was off through its window runs once when it comes back, not never
+	if _, err := (&Policy{Every: "custom", Cron: "@reboot"}).CronExpr(); err == nil {
-	daily.LastRun = now.AddDate(0, 0, -2)
+		t.Fatal("@reboot is not a snapshot schedule")
-	if !daily.Due(now) {
-		t.Fatal("a missed window must still be due")
 	}
-	// disabled never runs
+}
-	daily.Enabled = false
+
-	if daily.Due(now) {
+func TestCronFileText(t *testing.T) {
-		t.Fatal("disabled must never be due")
+	ps := []Policy{
+		{ID: "abc", Name: "tank nightly", Dataset: "tank", Every: "daily", At: "02:30", KeepFor: "6w", Enabled: true},
+		{ID: "def", Name: "paused one", Dataset: "tank/apps", Every: "hourly", At: "00:05", KeepFor: "2d"},
 	}
-	weekly := &Policy{Every: "weekly", At: "04:00", Weekday: int(time.Sunday), Enabled: true}
+	text, err := CronFileText(ps, "core", "/usr/local/bin/hamlab", "/home/core/apps/hamlab/config.json")
-	if d := weekly.lastDue(now); d.Weekday() != time.Sunday || d.After(now) {
+	if err != nil {
-		t.Fatalf("weekly lastDue = %v (%v)", d, d.Weekday())
+		t.Fatal(err)
+	}
+	if !strings.Contains(text, "30 2 * * * core /usr/local/bin/hamlab -config /home/core/apps/hamlab/config.json zfs-snapshot abc") {
+		t.Fatalf("the enabled schedule is missing or malformed:\n%s", text)
+	}
+	if strings.Contains(text, "zfs-snapshot def") {
+		t.Fatalf("a paused schedule must not be scheduled:\n%s", text)
+	}
+	// every line cron will read must parse as a schedule
+	for _, line := range strings.Split(text, "\n") {
+		line = strings.TrimSpace(line)
+		if line == "" || strings.HasPrefix(line, "#") || strings.Contains(line, "=") {
+			continue
+		}
+		f := strings.Fields(line)
+		if err := cron.ValidSchedule(strings.Join(f[:5], " ")); err != nil {
+			t.Errorf("line %q: %v", line, err)
 		}
-	hourly := &Policy{Every: "hourly", At: "00:30", Enabled: true}
-	if d := hourly.lastDue(now); d.Minute() != 30 || d.Hour() != 8 {
-		t.Fatalf("hourly lastDue at 09:15 with minute 30 = %v, want 08:30", d)
 	}
 }
 
@@ -77,6 +103,8 @@ func TestValidateRefusesNonsense(t *testing.T) {
 	bad := []Policy{
 		{Dataset: "", Every: "daily", At: "02:30", KeepFor: "6w"},
 		{Dataset: "tank", Every: "yearly", At: "02:30", KeepFor: "6w"},
+		{Dataset: "tank", Every: "custom", KeepFor: "6w"},
+		{Dataset: "tank", Every: "custom", Cron: "not a schedule", KeepFor: "6w"},
 		{Dataset: "tank", Every: "daily", At: "25:00", KeepFor: "6w"},
 		{Dataset: "tank", Every: "daily", At: "02:30"},                          // no retention
 		{Dataset: "tank", Every: "daily", At: "02:30", KeepFor: "6 weeks"},      // unparseable

internal/term/manager.go

@@ -84,10 +84,21 @@ func newID() string {
 
 // Create starts a new login shell. id may be empty.
 func (m *Manager) Create(id, name string) (*Session, error) {
+	return m.CreateCmd(id, name, nil)
+}
+
+// CreateCmd is Create with something other than a login shell: argv nil is the shell, and
+// anything else runs that instead — `docker exec -it …` for a container's console. Every
+// session is the same thing afterwards, so attach, resize, the scrollback ring and the tab
+// strip work without knowing which it is.
+func (m *Manager) CreateCmd(id, name string, argv []string) (*Session, error) {
 	if id == "" {
 		id = newID()
 	}
 	cmd := exec.Command(m.shell, "-l")
+	if len(argv) > 0 {
+		cmd = exec.Command(argv[0], argv[1:]...)
+	}
 	cmd.Dir = m.home
 	cmd.Env = append(os.Environ(), "TERM=xterm-256color", "COLORTERM=truecolor", "HAMLAB=1")
 	ptmx, err := pty.StartWithSize(cmd, &pty.Winsize{Cols: defaultCol, Rows: defaultRow})

web/src/Docker.tsx

@@ -1,14 +1,16 @@
 // Docker: containers grouped by the stack that deployed them, with start / stop / restart,
 // logs that follow, and inspect — then images, volumes and networks. Live CPU and memory
 // come from the same container sensors everything else uses.
-import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
+import { FormEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react'
-import { api, DockerContainer, DockerInfo, ImageCheck, Stack, StackDiff } from './api'
+import { api, DockerContainer, DockerEvent, DockerHealth, DockerInfo, ImageCheck, Stack, StackDiff } from './api'
 import TaskOutput from './TaskOutput'
 import StackNew, { DupFrom } from './StackNew'
-import { Sensor, fmtValue, useSensorStream } from './sensors'
+import { Sensor, fetchHistories, fmtValue, useSensorStream } from './sensors'
 import { useHashSel } from './hashsel'
-import { useRefreshMs } from './prefs'
+import { fmtWindowSec, useHistorySec, useRefreshMs } from './prefs'
-import { askText, askYes, Btn, CloseBtn, Loading, setUnsaved, toast } from './ui'
+import Sparkline from './Sparkline'
+import { fmtBytes } from './fmt'
+import { askText, askYes, Btn, CloseBtn, Loading, setUnsaved, toast, useEscape } from './ui'
 
 type Tab = 'containers' | 'stacks' | 'images' | 'volumes' | 'networks'
 const dot = (c: DockerContainer) => (c.state !== 'running' ? (c.state === 'exited' || c.state === 'dead' ? 'bad' : 'warn') : c.health === 'unhealthy' ? 'bad' : c.health === 'starting' ? 'warn' : 'ok')
@@ -50,18 +52,26 @@ function LogView({ id }: { id: string }) {
   )
 }
 
+// What a container is doing, and what can be done to it: its numbers over time, its health
+// checks, what has happened to it lately, a shell inside it, and its limits.
 function Detail({ c, live, onBack, onChanged }: { c: DockerContainer; live: Record<string, Sensor>; onBack: () => void; onChanged: () => void }) {
-  const [tab, setTab] = useState<'logs' | 'inspect'>('logs')
+  const [tab, setTab] = useState<'logs' | 'stats' | 'health' | 'events' | 'inspect'>('logs')
   const [info, setInfo] = useState<Record<string, unknown> | null>(null)
-  useEffect(() => { setInfo(null); if (tab === 'inspect') api.dockerInspect(c.name).then(setInfo).catch((e) => toast(e instanceof Error ? e.message : 'failed', 'err')) }, [tab, c.name])
+  const [limits, setLimits] = useState(false)
+  useEffect(() => { setInfo(null); if (tab === 'inspect' || limits) api.dockerInspect(c.name).then(setInfo).catch((e) => toast(e instanceof Error ? e.message : 'failed', 'err')) }, [tab, limits, c.name])
   const act = (a: 'start' | 'stop' | 'restart' | 'remove') => async () => { try { await api.dockerAction(c.name, a); toast(`${c.name}: ${a} ok`); onChanged() } catch (e) { toast(`${c.name}: ${e instanceof Error ? e.message : 'failed'}`, 'err') } }
   const running = c.state === 'running'
-  const s = (k: string) => live[`container.${c.name.toLowerCase().replace(/[^a-z0-9-]+/g, '_').replace(/^_|_$/g, '')}.${k}`]
+  const seg = c.name.toLowerCase().replace(/[^a-z0-9-]+/g, '_').replace(/^_|_$/g, '')
+  const s = (k: string) => live[`container.${seg}.${k}`]
   const mounts = (info?.mounts as { Source: string; Destination: string; RW: boolean }[] | undefined) ?? []
+  const shell = async () => {
+    try { const t = await api.dockerExec(c.name); toast(`shell open as “${t.name}” on the Terminal tab`); window.dispatchEvent(new CustomEvent('hamlab:terminal', { detail: t.id })) }
+    catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
+  }
   return (
     <div className="svc-detail">
       <CloseBtn onClick={onBack} />
-      <div className="svc-head"><button className="btn back" onClick={onBack}>‹ Back</button><span className={'sdot ' + dot(c)} /><span className="dname">{c.name}</span></div>
+      <div className="svc-head"><button className="btn back" onClick={onBack}>‹ Back</button><span className={'sdot ' + dot(c)} /><span className="dname">{c.name}</span>{c.health && c.health !== 'none' && <span className={'badge' + (c.health === 'healthy' ? '' : ' bad')}>{c.health}</span>}</div>
       <div className="props">
         <div><span className="k">Image</span><span className="mono small">{c.image}</span></div>
         <div><span className="k">Status</span><span>{c.status}</span></div>
@@ -72,11 +82,22 @@ function Detail({ c, live, onBack, onChanged }: { c: DockerContainer; live: Reco
       <div className="svc-actions">
         {!running && <Btn className="btn on" onClick={act('start')}>Start</Btn>}
         {!running && <Btn className="btn danger" confirm={`Remove the container ${c.name}? Its image, volumes and bind-mounted data stay; anything written inside the container itself is lost.${c.stack ? ` Deploying the ${c.stack} stack makes it again.` : ''}`} onClick={async () => { await act('remove')(); onBack() }}>Remove</Btn>}
+        {running && <Btn className="btn" onClick={shell}>Shell</Btn>}
         {running && <Btn className="btn" confirm={`Restart ${c.name}?`} onClick={act('restart')}>Restart</Btn>}
+        <Btn className="btn" onClick={() => setLimits(true)}>Limits</Btn>
         {running && <Btn className="btn danger" confirm={`Stop ${c.name}? It stays stopped until something starts it.`} onClick={act('stop')}>Stop</Btn>}
       </div>
-      <div className="tabs"><button className={tab === 'logs' ? 'on' : ''} onClick={() => setTab('logs')}>Logs</button><button className={tab === 'inspect' ? 'on' : ''} onClick={() => setTab('inspect')}>Inspect</button></div>
+      <div className="tabs">
+        <button className={tab === 'logs' ? 'on' : ''} onClick={() => setTab('logs')}>Logs</button>
+        <button className={tab === 'stats' ? 'on' : ''} onClick={() => setTab('stats')}>Stats</button>
+        <button className={tab === 'health' ? 'on' : ''} onClick={() => setTab('health')}>Health</button>
+        <button className={tab === 'events' ? 'on' : ''} onClick={() => setTab('events')}>Events</button>
+        <button className={tab === 'inspect' ? 'on' : ''} onClick={() => setTab('inspect')}>Inspect</button>
+      </div>
       {tab === 'logs' && <LogView id={c.name} />}
+      {tab === 'stats' && <Stats seg={seg} live={live} />}
+      {tab === 'health' && <Health id={c.name} />}
+      {tab === 'events' && <Events id={c.name} />}
       {tab === 'inspect' && (!info ? <Loading label="Inspecting…" /> : (
         <div className="props">
           <div><span className="k">Restarts</span><span>{String(info.restart || 'no')} policy · restarted {String(info.restarts ?? 0)}×</span></div>
@@ -88,11 +109,154 @@ function Detail({ c, live, onBack, onChanged }: { c: DockerContainer; live: Reco
           <div><span className="k">Env</span><span className="mono small muted">{((info.env_names as string[]) ?? []).join('  ')} <span className="muted">(names only)</span></span></div>
         </div>
       ))}
+      {limits && <LimitsDialog c={c} info={info} onClose={() => setLimits(false)} onSaved={() => { setLimits(false); onChanged() }} />}
+    </div>
+  )
+}
+
+// CPU and memory over time, from the container sensors the sampler already keeps.
+function Stats({ seg, live }: { seg: string; live: Record<string, Sensor> }) {
+  const [hist, setHist] = useState<{ id: string; t: number[]; v: number[] }[] | null>(null)
+  const historySec = useHistorySec()
+  useEffect(() => {
+    let alive = true
+    fetchHistories(`container.${seg}.cpu,container.${seg}.mem`)
+      .then((h) => { if (alive) setHist(h) })
+      .catch(() => { if (alive) setHist([]) })
+    return () => { alive = false }
+  }, [seg])
+  const css = (n: string) => getComputedStyle(document.documentElement).getPropertyValue(n).trim() || '#3987e5'
+  const of = (k: string) => hist?.find((h) => h.id === `container.${seg}.${k}`)?.v ?? []
+  const cpu = of('cpu'), mem = of('mem')
+  if (hist === null) return <Loading label="Reading the history…" />
+  if (cpu.length < 2 && mem.length < 2) return <div className="notice">No history yet for this container — it is sampled while it runs, so give it a moment.</div>
+  return (
+    <div className="cards">
+      <section className="card">
+        <div className="sub-head"><span className="dname">CPU</span><span className="spacer" /><span className="num">{fmtValue(live[`container.${seg}.cpu`]?.value, '%')} of a core</span></div>
+        <Sparkline series={[{ name: 'cpu', values: cpu, color: css('--s1') }]} format={(v) => `${v.toFixed(1)}%`} height={64} />
+        <div className="muted small">the last {fmtWindowSec(historySec)}</div>
+      </section>
+      <section className="card">
+        <div className="sub-head"><span className="dname">Memory</span><span className="spacer" /><span className="num">{fmtValue(live[`container.${seg}.mem`]?.value, 'B')}</span></div>
+        <Sparkline series={[{ name: 'mem', values: mem, color: css('--s2') }]} format={(v) => fmtBytes(v)} height={64} />
+        <div className="muted small">the last {fmtWindowSec(historySec)}</div>
+      </section>
+    </div>
+  )
+}
+
+// The health check's verdict, and what its last few probes actually said.
+function Health({ id }: { id: string }) {
+  const [h, setH] = useState<DockerHealth | null>(null)
+  const [err, setErr] = useState('')
+  useEffect(() => { setH(null); api.dockerHealth(id).then(setH).catch((e) => setErr(e instanceof Error ? e.message : 'failed')) }, [id])
+  if (err) return <div className="error">{err}</div>
+  if (!h) return <Loading label="Asking…" />
+  if (!h.configured) return <div className="notice">This image defines no health check, so Docker only knows whether the process is running — which it is ({h.state}). A health check is a <span className="mono">healthcheck:</span> block in the compose file.</div>
+  return (
+    <>
+      <div className="props">
+        <div><span className="k">Health</span><span className={h.status === 'healthy' ? '' : 'warn-text'}>{h.status}</span></div>
+        <div><span className="k">State</span><span>{h.state}</span></div>
+        {!!h.failing_streak && <div><span className="k">Failing</span><span className="warn-text">{h.failing_streak} probe{h.failing_streak === 1 ? '' : 's'} in a row</span></div>}
+      </div>
+      <div className="muted small" style={{ marginTop: 8 }}>The last {h.log?.length ?? 0} probes — Docker keeps five.</div>
+      {(h.log ?? []).slice().reverse().map((p, i) => (
+        <section className="card" key={i} style={{ marginTop: 6 }}>
+          <div className="sub-head">
+            <span className={'sdot ' + (p.ExitCode === 0 ? 'ok' : 'bad')} />
+            <span>{new Date(p.Start).toLocaleString()}</span>
+            <span className="spacer" />
+            <span className="muted small">exit {p.ExitCode} · {Math.max(0, new Date(p.End).getTime() - new Date(p.Start).getTime())} ms</span>
+          </div>
+          {p.Output.trim() && <pre className="run-out" style={{ maxHeight: 140 }}>{p.Output.trim()}</pre>}
+        </section>
+      ))}
+    </>
+  )
+}
+
+// What has happened to this container lately, asked of Docker rather than kept here.
+function Events({ id }: { id: string }) {
+  const [ev, setEv] = useState<DockerEvent[] | null>(null)
+  const [kept, setKept] = useState(0)
+  const [since, setSince] = useState('7d')
+  const [err, setErr] = useState('')
+  useEffect(() => { setEv(null); setErr(''); api.dockerEvents(id, since).then((r) => { setEv(r.events); setKept(r.kept) }).catch((e) => setErr(e instanceof Error ? e.message : 'failed')) }, [id, since])
+  const bad = (a: string, h?: string) => /die|kill|oom|destroy/.test(a) || (h !== undefined && h !== 'healthy')
+  return (
+    <>
+      <div className="chips" style={{ marginBottom: 8 }}>
+        {['24h', '7d', '30d'].map((x) => <button key={x} className={'chip' + (since === x ? ' on' : '')} onClick={() => setSince(x)}>{x}</button>)}
+      </div>
+      {err && <div className="error">{err}</div>}
+      {!ev && !err && <Loading label="Asking Docker…" />}
+      {ev && ev.length === 0 && <div className="notice">Nothing for this container in the last {since}. Ham Lab keeps its own record — Docker's own only reaches back a few minutes — so history starts from when this version was installed. {kept} events kept across every container.</div>}
+      {ev && ev.length > 0 && (
+        <div className="snap-list">
+          {ev.map((e, i) => (
+            <div className="snap" key={i}>
+              <span className={bad(e.action, e.health) ? 'warn-text' : ''}>{e.action}{e.health ? `: ${e.health}` : ''}</span>
+              <span className="muted">{new Date(e.t).toLocaleString()}</span>
+              <span className="muted small">{e.exit_code ? `exit ${e.exit_code}` : ''}{e.signal ? ` signal ${e.signal}` : ''}</span>
+              <span />
+              <span />
+            </div>
+          ))}
+        </div>
+      )}
+    </>
+  )
+}
+
+// Memory, CPU and restart policy, changed on the running container. Docker does this live,
+// which is the point — and also the catch, said plainly below.
+function LimitsDialog({ c, info, onClose, onSaved }: { c: DockerContainer; info: Record<string, unknown> | null; onClose: () => void; onSaved: () => void }) {
+  const curMem = Number(info?.memory_limit ?? 0)
+  const [mem, setMem] = useState(curMem > 0 ? String(Math.round(curMem / (1 << 20))) + 'm' : '')
+  const [cpus, setCpus] = useState('')
+  const [restart, setRestart] = useState(String(info?.restart ?? ''))
+  const [err, setErr] = useState('')
+  const [busy, setBusy] = useState(false)
+  useEscape(onClose)
+  const save = async (e: FormEvent) => {
+    e.preventDefault(); setErr(''); setBusy(true)
+    try { await api.dockerLimits(c.name, { memory: mem, cpus, restart }); toast(`${c.name} updated`); onSaved() }
+    catch (e) { setErr(e instanceof Error ? e.message : 'failed'); setBusy(false) }
+  }
+  return (
+    <div className="modal-back" onClick={onClose}>
+      <form className="modal" onClick={(e) => e.stopPropagation()} onSubmit={save}>
+        <h3>Limits for {c.name}</h3>
+        <label className="muted">Memory — 512m, 2g, or 0 for no limit{curMem > 0 ? ` (now ${fmtBytes(curMem)})` : ' (no limit now)'}
+          <input className="mono" value={mem} onChange={(e) => setMem(e.target.value)} placeholder="leave blank to keep" spellCheck={false} />
+        </label>
+        <label className="muted">CPUs — 1.5 means one and a half cores, 0 for no limit
+          <input className="mono" value={cpus} onChange={(e) => setCpus(e.target.value)} placeholder="leave blank to keep" spellCheck={false} />
+        </label>
+        <label className="muted">Restart policy
+          <select value={restart} onChange={(e) => setRestart(e.target.value)}>
+            <option value="">leave as it is</option>
+            {['no', 'on-failure', 'unless-stopped', 'always'].map((p) => <option key={p} value={p}>{p}</option>)}
+          </select>
+        </label>
+        <div className="notice attn">
+          This changes the running container now, without recreating it. It does <b>not</b> edit the
+          compose file{c.stack ? <> — so deploying <b>{c.stack}</b> puts back whatever the file says</> : ''}.
+          Edit the stack to make it stick.
+        </div>
+        {err && <div className="error">{err}</div>}
+        <div className="svc-actions">
+          <button type="submit" className="btn on" disabled={busy || (!mem && !cpus && !restart)}>Apply</button>
+          <span className="spacer" />
+          <button type="button" className="btn" onClick={onClose}>Cancel</button>
+        </div>
+      </form>
     </div>
   )
 }
 
-// A plain line diff (longest common subsequence): compose files are short, so the simple way is fine.
 function lineDiff(a: string, b: string): { s: ' ' | '+' | '-'; t: string }[] {
   const x = a.split('\n'), y = b.split('\n'); const n = x.length, m = y.length
   const L = Array.from({ length: n + 1 }, () => new Uint16Array(m + 1))
@@ -120,6 +284,7 @@ const maskEnv = (text: string) => text.split('\n').map((l) => { const i = l.inde
 
 // One stack: its compose file to edit, and the four things to do with it. Saving validates
 // with compose itself and keeps the previous file beside it; deploying is a separate, asked-for act.
+
 function StackDetail({ name, startTask, onBack, onChanged, onDuplicate }: { name: string; startTask?: string; onBack: () => void; onChanged: () => void; onDuplicate: (from: DupFrom) => void }) {
   const [st, setSt] = useState<Stack | null>(null)
   const [text, setText] = useState(''); const [orig, setOrig] = useState('')

web/src/KVM.tsx

@@ -0,0 +1,187 @@
+// The KVM of the machine you are looking at: the device's own web page, shown in the tab.
+//
+// One per machine, and it follows the machine switcher — switch to `grid` and you get
+// grid's KVM. The address is held on the **hub**, not on the machine, because a KVM earns
+// its keep exactly when that machine is down and cannot answer for itself; the hub proxies
+// the device either way.
+//
+// Nothing here knows anything about any particular KVM. It is an iframe pointed at Ham
+// Lab's own proxy, so whatever web UI the device serves is what you get.
+//
+// On the keyboard: a page cannot capture keys and forward them into an iframe, and does not
+// need to. An iframe with focus receives them directly. What the browser itself swallows —
+// Escape, Tab, Ctrl-W, the function keys — is what `navigator.keyboard.lock()` releases,
+// and that needs fullscreen and only exists in Chromium. Everywhere else you get the plain
+// version, which is most keys, and the device's own on-screen buttons for the rest.
+import { FormEvent, useCallback, useEffect, useRef, useState } from 'react'
+import { KVM as Device, api } from './api'
+import { Btn, Loading, askYes, toast, useEscape } from './ui'
+
+// lock() releases the keys the browser would otherwise keep for itself. Chromium only; a
+// browser without it simply does not get the extra keys.
+type KeyboardLock = { lock?: (keys?: string[]) => Promise<void>; unlock?: () => void }
+const keyboardAPI = (): KeyboardLock | undefined => (navigator as unknown as { keyboard?: KeyboardLock }).keyboard
+
+// Fullscreen, across the three ways browsers spell it. Safari on a Mac wants the webkit
+// prefix; Safari on iOS has no fullscreen for an ordinary element at all — only for a
+// <video> — so there the page expands inside the viewport instead, which is the same thing
+// minus the browser's own chrome.
+type FSElement = HTMLElement & { webkitRequestFullscreen?: () => Promise<void> | void }
+type FSDocument = Document & { webkitFullscreenElement?: Element | null; webkitExitFullscreen?: () => Promise<void> | void }
+const nativeFullscreen = () => {
+  if (typeof document === 'undefined') return false
+  const el = document.documentElement as FSElement
+  return typeof el.requestFullscreen === 'function' || typeof el.webkitRequestFullscreen === 'function'
+}
+const fullscreenElement = () => (document as FSDocument).fullscreenElement ?? (document as FSDocument).webkitFullscreenElement ?? null
+async function enterFullscreen(el: FSElement) {
+  if (typeof el.requestFullscreen === 'function') return el.requestFullscreen()
+  if (typeof el.webkitRequestFullscreen === 'function') return el.webkitRequestFullscreen()
+  throw new Error('this browser has no fullscreen for a page element')
+}
+function exitFullscreen() {
+  const d = document as FSDocument
+  if (typeof d.exitFullscreen === 'function') return d.exitFullscreen()
+  if (typeof d.webkitExitFullscreen === 'function') return d.webkitExitFullscreen()
+}
+
+export default function KVM({ machine, title }: { machine: string | null; title: string }) {
+  const [kvm, setKvm] = useState<Device | null | undefined>(undefined)
+  const [adding, setAdding] = useState(false)
+  const [err, setErr] = useState('')
+  const [full, setFull] = useState(false)   // native fullscreen is on
+  const [expanded, setExpanded] = useState(false) // the fallback: filling the viewport in-page
+  const box = useRef<HTMLDivElement>(null)
+  const frame = useRef<HTMLIFrameElement>(null)
+  const key = machine ?? '' // the hub is a machine too, and its key is the empty one
+
+  const load = useCallback(async () => {
+    try { setKvm((await api.kvms()).find((k) => k.machine === key) ?? null) }
+    catch (e) { setErr(e instanceof Error ? e.message : 'failed') }
+  }, [key])
+  useEffect(() => { load() }, [load])
+
+  // Fullscreen plus a keyboard lock is what makes Escape and Tab reach the machine rather
+  // than the browser. Both are undone together when fullscreen ends, however it ends.
+  useEffect(() => {
+    const onChange = () => {
+      const on = !!fullscreenElement()
+      setFull(on)
+      if (!on) keyboardAPI()?.unlock?.()
+    }
+    document.addEventListener('fullscreenchange', onChange)
+    document.addEventListener('webkitfullscreenchange', onChange)
+    return () => {
+      document.removeEventListener('fullscreenchange', onChange)
+      document.removeEventListener('webkitfullscreenchange', onChange)
+      keyboardAPI()?.unlock?.()
+    }
+  }, [])
+  // Expanding is in-page, so Escape is ours to handle.
+  useEscape(() => setExpanded(false), expanded)
+
+  const goBig = async () => {
+    if (!nativeFullscreen()) { setExpanded(true); frame.current?.focus(); return }
+    try {
+      if (box.current) await enterFullscreen(box.current)
+      await keyboardAPI()?.lock?.()
+      frame.current?.focus()
+    } catch {
+      setExpanded(true) // refused (an iframe policy, a permissions policy): fill the page instead
+      frame.current?.focus()
+    }
+  }
+  const goSmall = () => { setExpanded(false); if (fullscreenElement()) exitFullscreen() }
+
+  const remove = async () => {
+    if (!kvm || !await askYes(`Forget ${title}'s KVM? The device itself is not touched — this only removes its address.`, 'Remove')) return
+    try { await api.removeKVM(kvm.id); toast('removed'); await load() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
+  }
+
+  if (err && kvm === undefined) return <div className="dash"><div className="error">{err}</div></div>
+  if (kvm === undefined) return <div className="dash"><Loading label="Looking…" /></div>
+
+  const locks = !!keyboardAPI()?.lock
+  const big = full || expanded
+
+  return (
+    <div className="kvm">
+      <div className="kvm-bar">
+        <span className="dname">{title}</span>
+        {kvm
+          ? <span className="muted small mono kvm-url" title={kvm.url}>{kvm.name !== kvm.url ? `${kvm.name} · ` : ''}{kvm.url}</span>
+          : <span className="muted small">no KVM</span>}
+        <span className="spacer" />
+        {kvm && <>
+          <Btn className="btn" onClick={goBig} title={!nativeFullscreen() ? 'fill the screen — this browser has no true fullscreen for a page element' : locks ? 'fullscreen, and the browser stops keeping Escape, Tab and Ctrl-W for itself' : 'fullscreen — this browser will still keep some keys for itself'}>{nativeFullscreen() ? 'Fullscreen' : 'Expand'}</Btn>
+          <a className="btn" href={kvm.url} target="_blank" rel="noreferrer" title="the device directly, outside Ham Lab — only works on the LAN">Open ↗</a>
+          <Btn className="btn" onClick={() => setAdding(true)}>Change</Btn>
+          <Btn className="btn danger" onClick={remove}>Remove</Btn>
+        </>}
+        {!kvm && <Btn className="btn on" onClick={() => setAdding(true)}>+ Add a KVM for {title}</Btn>}
+      </div>
+
+      {kvm ? (
+        <div className={'kvm-frame' + (full ? ' full' : '') + (expanded ? ' expanded' : '')} ref={box}>
+          <iframe ref={frame} key={kvm.id} src={`/kvm/${kvm.id}/`} title={kvm.name}
+            allow="fullscreen; clipboard-read; clipboard-write" />
+          {expanded && <button className="btn kvm-shrink" onClick={goSmall} title="back to the console">Done</button>}
+        </div>
+      ) : (
+        <div className="svc-empty muted kvm-none">
+          <div>
+            <p>No KVM for <b>{title}</b>.</p>
+            <p className="small">Add one with its address and its page appears here. The address is kept on the hub,
+              so it still works when {title} does not — which is when you will want it.</p>
+          </div>
+        </div>
+      )}
+      {kvm && !big && (
+        <div className="kvm-hint muted small">
+          Click the picture, then type — the keys go to {title}.
+          {locks ? ' Fullscreen also takes Escape, Tab and Ctrl-W back from the browser.' : ' This browser keeps Escape, Tab and Ctrl-W for itself; use the device’s own on-screen keys for those, or a Chromium browser.'}
+        </div>
+      )}
+      {adding && <AddDialog machine={key} title={title} existing={kvm} onClose={() => setAdding(false)} onAdded={() => { setAdding(false); load() }} />}
+    </div>
+  )
+}
+
+function AddDialog({ machine, title, existing, onClose, onAdded }: { machine: string; title: string; existing: Device | null; onClose: () => void; onAdded: () => void }) {
+  const [url, setUrl] = useState(existing?.url ?? '')
+  const [name, setName] = useState(existing?.name ?? '')
+  const [err, setErr] = useState('')
+  const [busy, setBusy] = useState(false)
+  useEscape(onClose)
+  const save = async (e: FormEvent) => {
+    e.preventDefault(); setErr(''); setBusy(true)
+    try { await api.addKVM(name.trim(), url.trim(), machine); onAdded() }
+    catch (e) { setErr(e instanceof Error ? e.message : 'failed'); setBusy(false) }
+  }
+  return (
+    <div className="modal-back" onClick={onClose}>
+      <form className="modal" onClick={(e) => e.stopPropagation()} onSubmit={save}>
+        <h3>{existing ? `Change ${title}'s KVM` : `A KVM for ${title}`}</h3>
+        <label className="muted">Address — whatever you would type into a browser
+          <input autoFocus className="mono" value={url} onChange={(e) => setUrl(e.target.value)} placeholder="10.0.0.217" spellCheck={false} autoCapitalize="off" required />
+        </label>
+        <label className="muted">Name — optional; the address is used otherwise
+          <input value={name} onChange={(e) => setName(e.target.value)} placeholder={`${title} KVM`} maxLength={64} />
+        </label>
+        <div className="muted small">
+          Ham Lab serves the device from its own address, so a KVM that speaks only plain HTTP
+          still works here, and it is behind this login rather than needing a second one. The
+          address is kept on the hub, so it still reaches the device when {title} is down. The
+          device itself is not changed and is never reachable from outside.
+          {existing && <> Saving replaces the one there now — a machine has one KVM.</>}
+        </div>
+        {err && <div className="error">{err}</div>}
+        <div className="svc-actions">
+          <button type="submit" className="btn on" disabled={busy || !url.trim()}>{existing ? 'Save' : 'Add'}</button>
+          <span className="spacer" />
+          <button type="button" className="btn" onClick={onClose}>Cancel</button>
+        </div>
+      </form>
+    </div>
+  )
+}

web/src/Shell.tsx

@@ -12,6 +12,7 @@ import Jobs from './Jobs'
 import Updates from './Updates'
 import Disks from './Disks'
 import ZFS from './ZFS'
+import KVM from './KVM'
 import Files from './Files'
 import Top from './Top'
 import Machines from './Machines'
@@ -54,22 +55,23 @@ const GROUPS: Group[] = [
   { id: 'procs', label: 'Processes', icon: '▚' },
   { id: 'terminal', label: 'Terminal', icon: '>_' },
   { id: 'services', label: 'Services', icon: '◈' },
-  { id: 'docker', label: 'Docker', icon: DockerIcon },
   { id: 'jobs', label: 'Jobs', icon: ClockIcon },
-  { id: 'updates', label: 'Updates', icon: '⇪' },
   { id: 'disks', label: 'Disks', icon: '▤' },
-  { id: 'zfs', label: 'ZFS', icon: '⛁' },
-  { id: 'usage', label: 'Usage', icon: '▧' },
   { id: 'files', label: 'Files', icon: FolderIcon },
+  { id: 'usage', label: 'Usage', icon: '▧' },
+  { id: 'zfs', label: 'ZFS', icon: '⛁' },
+  { id: 'docker', label: 'Docker', icon: DockerIcon },
+  { id: 'kvm', label: 'KVM', icon: '⌨' },
+  { id: 'updates', label: 'Updates', icon: '⇪' },
   { id: 'alerts', label: 'Alerts', icon: BellIcon },
   { id: 'machines', label: 'Machines', icon: '⌂' },
-  { id: 'users', label: 'Users', icon: '⚇' },
   { id: 'widgets', label: 'Widgets', icon: '▦' },
+  { id: 'users', label: 'Users', icon: '⚇' },
   { id: 'settings', label: 'Settings', icon: '⚙' },
 ]
 
-type View = 'terminal' | 'widgets' | 'docker' | 'alerts' | 'services' | 'jobs' | 'updates' | 'disks' | 'zfs' | 'usage' | 'files' | 'procs' | 'machines' | 'users' | 'settings'
+type View = 'terminal' | 'widgets' | 'docker' | 'alerts' | 'services' | 'jobs' | 'updates' | 'disks' | 'zfs' | 'usage' | 'files' | 'procs' | 'machines' | 'kvm' | 'users' | 'settings'
-const VIEWS: View[] = ['terminal', 'widgets', 'docker', 'alerts', 'services', 'jobs', 'updates', 'disks', 'zfs', 'usage', 'files', 'procs', 'machines', 'users', 'settings']
+const VIEWS: View[] = ['terminal', 'widgets', 'docker', 'alerts', 'services', 'jobs', 'updates', 'disks', 'zfs', 'usage', 'files', 'procs', 'machines', 'kvm', 'users', 'settings']
 const HUB_ONLY: View[] = ['machines', 'users', 'settings', 'alerts']
 
 // URL hash is the source of truth for where you are: #/<machine|hub>/<view>. Back and
@@ -123,6 +125,20 @@ export default function Shell({ me: hubMe, onLoggedOut }: { me: Me; onLoggedOut:
 
   useEffect(() => { reload() }, [reload])
 
+  // A shell opened inside a container from the Docker tab is a terminal like any other:
+  // pick it up, show the Terminal tab, and make it the active one.
+  useEffect(() => {
+    const h = async (e: Event) => {
+      const id = (e as CustomEvent<string>).detail
+      if (!id) return
+      await reload()
+      setActive(id)
+      setView('terminal')
+    }
+    window.addEventListener('hamlab:terminal', h)
+    return () => window.removeEventListener('hamlab:terminal', h)
+  }, [reload])
+
   // Machine list for the switcher (hub-local), refreshed while the console is open.
   const loadMachines = useCallback(async () => { try { setMachines((await api.machines()).nodes) } catch { /* not a hub or offline */ } }, [])
   useEffect(() => { loadMachines(); const t = setInterval(loadMachines, listRate); return () => clearInterval(t) }, [loadMachines, listRate])
@@ -269,6 +285,7 @@ export default function Shell({ me: hubMe, onLoggedOut }: { me: Me; onLoggedOut:
         <main className="main" key={machine ?? 'hub'}>
           {view === 'settings' && <Settings me={hubMe} />}
           {view === 'users' && <Users me={hubMe} />}
+          {view === 'kvm' && <KVM key={machine ?? 'hub'} machine={machine} title={title} />}
           {view === 'usage' && <Usage key={machine ?? ''} />}
           {view === 'machines' && <Machines hub={hubMe} onChanged={loadMachines} onOpen={(n) => { switchMachine(n); setView('procs') }} />}
           {view === 'widgets' && <Gallery />}
@@ -306,7 +323,7 @@ export default function Shell({ me: hubMe, onLoggedOut }: { me: Me; onLoggedOut:
         <span>{machine ? `${title} · ` : ''}{me.version.startsWith('v') ? me.version : 'v' + me.version}</span>
         <span>up {fmtUptime(me.uptime)}</span>
         <span className="spacer" />
-        <span className="muted">{view === 'widgets' ? 'Widgets' : view === 'docker' ? 'Docker' : view === 'alerts' ? 'Alerts' : view === 'services' ? 'Services' : view === 'jobs' ? 'Jobs' : view === 'updates' ? 'Updates' : view === 'disks' ? 'Disks' : view === 'zfs' ? 'ZFS' : view === 'usage' ? 'Usage' : view === 'files' ? 'Files' : view === 'procs' ? 'Processes' : view === 'machines' ? 'Machines' : view === 'users' ? 'Users' : view === 'settings' ? 'Settings' : activeTerm ? activeTerm.name : ''}</span>
+        <span className="muted">{view === 'widgets' ? 'Widgets' : view === 'docker' ? 'Docker' : view === 'alerts' ? 'Alerts' : view === 'services' ? 'Services' : view === 'jobs' ? 'Jobs' : view === 'updates' ? 'Updates' : view === 'disks' ? 'Disks' : view === 'zfs' ? 'ZFS' : view === 'usage' ? 'Usage' : view === 'files' ? 'Files' : view === 'procs' ? 'Processes' : view === 'machines' ? 'Machines' : view === 'kvm' ? 'KVM' : view === 'users' ? 'Users' : view === 'settings' ? 'Settings' : activeTerm ? activeTerm.name : ''}</span>
       </footer>
     </div>
   )

web/src/ZFS.tsx

@@ -9,7 +9,7 @@
 // disagree over. An existing cron job is detected and shown, so the page tells the truth
 // about the machine whether Ham Lab or something else is doing the work.
 import { FormEvent, useCallback, useEffect, useState } from 'react'
-import { api, ZFSExternal, ZFSInfo, ZFSOverview, ZFSPolicy, ZFSSnapshot } from './api'
+import { api, ZFSExternal, ZFSInfo, ZFSOverview, ZFSPolicy, ZFSRun, ZFSSnapshot } from './api'
 import { fmtBytes } from './fmt'
 import { useRefreshMs } from './prefs'
 import { askText, askYes, Btn, Loading, Meter, Spinner, toast, useEscape } from './ui'
@@ -26,7 +26,8 @@ const soon = (t?: string) => {
 }
 const WEEKDAYS = ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday']
 const describe = (p: ZFSPolicy) => {
-  const when = p.every === 'hourly' ? `every hour at :${(p.at.split(':')[1] ?? '00').padStart(2, '0')}`
+  const when = p.every === 'custom' ? `on ${p.cron}`
+    : p.every === 'hourly' ? `every hour at :${(p.at.split(':')[1] ?? '00').padStart(2, '0')}`
     : p.every === 'daily' ? `every day at ${p.at}`
       : p.every === 'weekly' ? `every ${WEEKDAYS[p.weekday] ?? 'Sunday'} at ${p.at}`
         : `on day ${p.day} of the month at ${p.at}`
@@ -67,18 +68,43 @@ function PoolCard({ p, onChanged }: { p: ZFSInfo['pools'][number]; onChanged: ()
   )
 }
 
+// A snapshot's name carries when it was taken and how long it lives — 2026-09-24_02.30.01--6w
+// — so the explorer can say when each one goes without asking the server anything.
+function snapMeta(full: string): { taken?: Date; expires?: Date } {
+  const snap = full.split('@')[1] ?? ''
+  const [body, ttl] = snap.split('--')
+  const m = /(\d{4})-(\d{2})-(\d{2})_(\d{2})\.(\d{2})\.(\d{2})$/.exec(body)
+  if (!m) return {}
+  const taken = new Date(+m[1], +m[2] - 1, +m[3], +m[4], +m[5], +m[6])
+  const a = /^(\d+)([mhdw])$/.exec(ttl ?? '')
+  if (!a) return { taken }
+  const ms = { m: 60e3, h: 3600e3, d: 86400e3, w: 7 * 86400e3 }[a[2]]!
+  return { taken, expires: new Date(taken.getTime() + +a[1] * ms) }
+}
+
+// One dataset in the snapshot explorer: how many it has and what they cost, opening into
+// the snapshots themselves, each of which can be taken now or deleted.
 function DatasetRow({ d, onChanged }: { d: ZFSInfo['datasets'][number]; onChanged: () => void }) {
   const [snaps, setSnaps] = useState<ZFSSnapshot[] | null>(null)
   const [open, setOpen] = useState(false)
+  const reload = useCallback(async () => {
+    try { setSnaps(await api.zfsSnapshots(d.name)) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err'); setSnaps([]) }
+  }, [d.name])
   async function toggle() {
     setOpen(!open)
-    if (!open && !snaps) { try { setSnaps(await api.zfsSnapshots(d.name)) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err'); setSnaps([]) } }
+    if (!open && !snaps) await reload()
   }
   async function snapshot() {
-    const def = 'hamlab-' + new Date().toISOString().slice(0, 16).replace(/[-:T]/g, '').replace(/(\d{8})(\d{4})/, '$1-$2')
+    const def = new Date().toISOString().slice(0, 19).replace('T', '_').replace(/:/g, '.')
     const name = await askText(`Snapshot name for ${d.name}@`, def)
     if (!name) return
-    try { await api.zfsSnapshot(d.name, name.trim()); toast(`${d.name}@${name.trim()} created`); setSnaps(null); onChanged() }
+    try { await api.zfsSnapshot(d.name, name.trim()); toast(`${d.name}@${name.trim()} created`); await reload(); onChanged() }
+    catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
+  }
+  async function destroy(s: ZFSSnapshot) {
+    const short = s.name.split('@')[1]
+    if (!await askYes(`Delete ${s.name}? It is removed from this dataset and from every dataset beneath it that has a snapshot of the same name. This cannot be undone.`, 'Delete')) return
+    try { await api.deleteZfsSnapshot(s.name, true); toast(`${short} deleted`); await reload(); onChanged() }
     catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
   }
   const depth = d.name.split('/').length - 1
@@ -86,18 +112,29 @@ function DatasetRow({ d, onChanged }: { d: ZFSInfo['datasets'][number]; onChange
     <>
       <div className="ds-row" style={{ paddingLeft: 12 + depth * 16 }}>
         <span className="ds-name" onClick={toggle}><span className="chev">{open ? '▾' : '▸'}</span>{d.name.split('/').pop()}{d.type === 'volume' && <span className="badge">zvol</span>}</span>
-        <span className="num">{fmtBytes(d.used)}</span>
+        <span className="num">{d.snapshots}</span>
-        <span className="num muted">{fmtBytes(d.refer)}</span>
         <span className="num muted">{d.used_by_snapshots ? fmtBytes(d.used_by_snapshots) : '—'}</span>
-        <span className="num muted">{d.snapshots}</span>
+        <span className="num muted">{fmtBytes(d.used)}</span>
-        <span className="num muted">{d.compressratio}</span>
+        <span className="num muted">{fmtBytes(d.refer)}</span>
         <span className="ds-mount muted">{d.mount}</span>
       </div>
       {open && (
         <div className="ds-snaps">
-          <div className="svc-actions"><Btn className="btn" onClick={snapshot}>Snapshot now</Btn><span className="muted">avail {fmtBytes(d.avail)}</span></div>
+          <div className="svc-actions"><Btn className="btn" onClick={snapshot}>Snapshot now</Btn><span className="muted small">{d.snapshots} snapshot{d.snapshots === 1 ? '' : 's'} · {fmtBytes(d.avail)} free on this dataset</span></div>
           {snaps === null ? <div className="muted"><Spinner /> listing…</div> : snaps.length === 0 ? <div className="muted">No snapshots.</div> : (
-            <div className="snap-list">{snaps.slice().reverse().map((s) => <div key={s.name} className="snap"><span className="mono">{s.name.split('@')[1]}</span><span className="num muted">{new Date(s.created).toLocaleString()} · {fmtBytes(s.used)}</span></div>)}</div>
+            <div className="snap-list">{snaps.slice().reverse().map((s) => {
+              const { taken, expires } = snapMeta(s.name)
+              const gone = expires && expires.getTime() < Date.now()
+              return (
+                <div key={s.name} className="snap">
+                  <span className="mono snap-name" title={s.name}>{s.name.split('@')[1]}</span>
+                  <span className="muted snap-when">{taken ? taken.toLocaleString() : new Date(s.created).toLocaleString()}</span>
+                  <span className={'muted snap-exp' + (gone ? ' warn' : '')}>{expires ? (gone ? 'expired' : `expires ${expires.toLocaleDateString()}`) : 'kept'}</span>
+                  <span className="num muted">{fmtBytes(s.used)}</span>
+                  <Btn className="btn danger snap-del" onClick={() => destroy(s)}>Delete</Btn>
+                </div>
+              )
+            })}</div>
           )}
         </div>
       )}
@@ -107,7 +144,7 @@ function DatasetRow({ d, onChanged }: { d: ZFSInfo['datasets'][number]; onChange
 
 // ---- schedules ----
 
-function ExternalNotice({ x, mine }: { x: ExternalJob; mine: number }) {
+function ExternalNotice({ x, mine, onAdopt }: { x: ExternalJob; mine: number; onAdopt: () => void }) {
   return (
     <div className={'notice' + (mine > 0 ? ' attn' : '')} style={{ marginBottom: 8 }}>
       <b>{x.tool}</b> is already taking snapshots on this machine, from cron as <b>{x.user}</b>:
@@ -115,25 +152,21 @@ function ExternalNotice({ x, mine }: { x: ExternalJob; mine: number }) {
       <span className="muted">in {x.source}{x.last ? ` · last ran ${ago(x.last)}` : ''}</span>
       {mine > 0
         ? <div className="warn" style={{ marginTop: 6 }}>A Ham Lab schedule is running as well. Both use the same names and both only remove snapshots that have passed the expiry written into them, so they will not fight — but they will take twice as many. Keep one.</div>
-        : <div className="muted small" style={{ marginTop: 6 }}>Ham Lab reads the snapshots it makes and shows when each expires. Add a schedule here only if you mean to replace it; edit this one on the Jobs tab.</div>}
+        : <div className="svc-actions" style={{ marginTop: 6 }}>
+            <Btn className="btn" onClick={onAdopt}>Adopt as a schedule</Btn>
+            <span className="muted small">Copies it into a Ham Lab schedule you can edit here. Turn the old cron job off on the Jobs tab once you are happy with it.</span>
+          </div>}
     </div>
   )
 }
 type ExternalJob = ZFSExternal
 
-function ScheduleCard({ p, next, onChanged, onEdit }: { p: ZFSPolicy; next?: string; onChanged: () => void; onEdit: () => void }) {
+function ScheduleCard({ p, cron, next, last, onChanged, onEdit }: { p: ZFSPolicy; cron: string; next?: string; last: ZFSRun; onChanged: () => void; onEdit: () => void }) {
-  const [expired, setExpired] = useState<ZFSSnapshot[] | null>(null)
   const [busy, setBusy] = useState('')
-  const preview = async () => {
-    setBusy('preview')
-    try { setExpired((await api.zfsExpired(p.id)).expired) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') } finally { setBusy('') }
-  }
   const run = async () => {
-    const n = expired?.length
+    if (!await askYes(`Take a snapshot of ${p.dataset} now? Snapshots past their retention are removed at the same time, as they would be on the schedule.`, 'Run now')) return
-    const warn = n ? ` It will also destroy ${n} expired snapshot${n === 1 ? '' : 's'}.` : ''
-    if (!await askYes(`Take a snapshot of ${p.dataset} now?${warn}`, 'Run now')) return
     setBusy('run')
-    try { const r = await api.runZfsSchedule(p.id); toast(`${r.taken}${r.pruned.length ? ` · ${r.pruned.length} expired removed` : ''}`); setExpired(null); onChanged() }
+    try { const r = await api.runZfsSchedule(p.id); toast(`${r.taken}${r.pruned.length ? ` · ${r.pruned.length} expired removed` : ''}`); onChanged() }
     catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') } finally { setBusy('') }
   }
   const remove = async () => {
@@ -146,7 +179,7 @@ function ScheduleCard({ p, next, onChanged, onEdit }: { p: ZFSPolicy; next?: str
   return (
     <section className="card">
       <div className="sub-head">
-        <span className={'dot ' + (p.last_error ? 'bad' : p.enabled ? 'ok' : '')} />
+        <span className={'dot ' + (last.error ? 'bad' : p.enabled ? 'ok' : '')} />
         <span className="dname">{p.name || p.dataset}</span>
         {!p.enabled && <span className="badge">paused</span>}
         <span className="spacer" />
@@ -155,25 +188,13 @@ function ScheduleCard({ p, next, onChanged, onEdit }: { p: ZFSPolicy; next?: str
       <div className="props" style={{ marginTop: 6 }}>
         <div><span className="k">Dataset</span><span className="mono">{p.dataset}</span></div>
         <div><span className="k">When</span><span>{describe(p)}</span></div>
-        <div><span className="k">Last run</span><span>{ago(p.last_run)}{p.pruned ? ` · ${p.pruned} expired removed` : ''}</span></div>
+        <div><span className="k">Cron</span><span className="mono small" title={`in ${'/etc/cron.d/hamlab-zfs'}`}>{cron}</span></div>
-        {p.last_taken && <div><span className="k">Last snapshot</span><span className="mono small">{p.last_taken.split('@')[1]}</span></div>}
+        <div><span className="k">Last run</span><span>{ago(last.at)}{last.pruned?.length ? ` · ${last.pruned.length} expired removed` : ''}</span></div>
+        {last.taken && <div><span className="k">Last snapshot</span><span className="mono small">{last.taken.split('@')[1]}</span></div>}
         {p.keep_min > 0 && <div><span className="k">Floor</span><span>always keep the newest {p.keep_min}</span></div>}
       </div>
-      {p.last_error && <div className="error" style={{ marginTop: 8 }}>{p.last_error}</div>}
+      {last.error && <div className="error" style={{ marginTop: 8 }}>{last.error}</div>}
-      {expired && (
-        <div className="notice" style={{ marginTop: 8 }}>
-          {expired.length === 0 ? 'Nothing has expired — a run now would only take a snapshot.' : (
-            <>
-              <b>{expired.length}</b> snapshot{expired.length === 1 ? '' : 's'} would be destroyed by a run now:
-              <div className="snap-list" style={{ marginTop: 6, maxHeight: 180, overflow: 'auto' }}>
-                {expired.map((s) => <div key={s.name} className="snap"><span className="mono">{s.name.split('@')[1]}</span><span className="num muted">{new Date(s.created).toLocaleDateString()} · {fmtBytes(s.used)}</span></div>)}
-              </div>
-            </>
-          )}
-        </div>
-      )}
       <div className="svc-actions">
-        <Btn className="btn" busy={busy === 'preview'} onClick={preview}>What would it delete?</Btn>
         <Btn className="btn on" busy={busy === 'run'} onClick={run}>Run now</Btn>
         <button className="btn" onClick={onEdit}>Edit</button>
         <Btn className="btn" onClick={toggle}>{p.enabled ? 'Pause' : 'Resume'}</Btn>
@@ -210,14 +231,20 @@ function ScheduleDialog({ p, datasets, onClose, onSaved }: { p: Partial<ZFSPolic
         </label>
         <label className="muted">How often
           <span className="chips">
-            {(['hourly', 'daily', 'weekly', 'monthly'] as const).map((k) => (
+            {(['hourly', 'daily', 'weekly', 'monthly', 'custom'] as const).map((k) => (
               <button type="button" key={k} className={'chip' + (f.every === k ? ' on' : '')} onClick={() => set('every', k)}>{k}</button>
             ))}
           </span>
         </label>
+        {f.every === 'custom' ? (
+          <label className="muted">Cron expression — minute hour day month weekday
+            <input className="mono" value={f.cron ?? ''} onChange={(e) => set('cron', e.target.value)} placeholder="0 */6 * * *" spellCheck={false} required />
+          </label>
+        ) : (
           <label className="muted">At
             <input type="time" value={f.at} onChange={(e) => set('at', e.target.value)} />
           </label>
+        )}
         {f.every === 'weekly' && (
           <label className="muted">On
             <select value={f.weekday} onChange={(e) => set('weekday', Number(e.target.value))}>
@@ -230,6 +257,9 @@ function ScheduleDialog({ p, datasets, onClose, onSaved }: { p: Partial<ZFSPolic
             <input type="number" min={1} max={28} value={f.day} onChange={(e) => set('day', Number(e.target.value))} />
           </label>
         )}
+        <label className="muted">Snapshot name prefix — optional; the timestamp and retention follow it
+          <input value={f.prefix ?? ''} onChange={(e) => set('prefix', e.target.value)} placeholder="none" spellCheck={false} />
+        </label>
         <label className="muted">Keep for — written into each snapshot's name, as zfSnap does it: 6w, 30d, 12h
           <input value={f.keep_for} onChange={(e) => set('keep_for', e.target.value)} placeholder="6w" spellCheck={false} required />
         </label>
@@ -255,6 +285,26 @@ function ScheduleDialog({ p, datasets, onClose, onSaved }: { p: Partial<ZFSPolic
   )
 }
 
+// adopt reads an existing cron job into the fields of a new schedule, so replacing it is a
+// matter of checking the numbers rather than retyping them. zfSnap's own flags are read
+// where they can be: -a <age> is the retention, -r is recursive.
+function adopt(x: ExternalJob, fallbackDataset: string): Partial<ZFSPolicy> {
+  const cmd = x.command
+  const keep = /-a\s+(\d+[mhdwMy])/.exec(cmd)?.[1] ?? /--\s*(\d+[mhdw])\b/.exec(cmd)?.[1] ?? '6w'
+  const f = x.schedule.trim().split(/\s+/)
+  const min = Number(f[0]), hour = Number(f[1])
+  const simple = f.length === 5 && Number.isInteger(min) && Number.isInteger(hour) && f[2] === '*' && f[3] === '*' && f[4] === '*'
+  const ds = /\b(tank[\w/-]*)\b/.exec(cmd)?.[1] ?? fallbackDataset
+  return {
+    ...BLANK, name: `${x.tool} (adopted)`, dataset: ds,
+    recursive: /\s-\w*r/.test(cmd) || true,
+    every: simple ? 'daily' : 'custom',
+    at: simple ? `${String(hour).padStart(2, '0')}:${String(min).padStart(2, '0')}` : '02:30',
+    cron: simple ? '' : x.schedule.trim(),
+    keep_for: keep, keep_min: 1, enabled: true,
+  }
+}
+
 // ---- page ----
 
 export default function ZFS() {
@@ -286,14 +336,20 @@ export default function ZFS() {
       <div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Snapshot schedules</h2><span className="spacer" />
         <Btn className="btn on" onClick={() => setEditing({ ...BLANK, dataset: zfs.pools[0]?.name ?? '' })}>+ New schedule</Btn>
       </div>
-      {ov.external && <ExternalNotice x={ov.external} mine={mine} />}
+      {ov.external && <ExternalNotice x={ov.external} mine={mine} onAdopt={() => setEditing(adopt(ov.external!, zfs.pools[0]?.name ?? ''))} />}
       {ov.policies.length === 0 && !ov.external && <div className="notice">Nothing is taking snapshots on a schedule. Snapshots are written the way zfSnap writes them, with the retention in the name, so anything already here is understood.</div>}
-      <div className="cards">{ov.policies.map(({ policy, next }) => (
+      <div className="cards">{ov.policies.map(({ policy, cron, next, last }) => (
-        <ScheduleCard key={policy.id} p={policy} next={next} onChanged={load} onEdit={() => setEditing(policy)} />
+        <ScheduleCard key={policy.id} p={policy} cron={cron} next={next} last={last ?? {}} onChanged={load} onEdit={() => setEditing(policy)} />
       ))}</div>
 
-      <div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Datasets</h2><span className="spacer" /><span className="muted small">used · referenced · in snapshots · count · compression · mount</span></div>
+      <div className="dash-bar" style={{ marginTop: 12 }}><h2 className="section">Snapshots</h2><span className="spacer" />
+        <span className="muted small">{zfs.datasets.reduce((a, d) => a + d.snapshots, 0)} across {zfs.datasets.length} datasets · open one to browse, take or delete</span>
+      </div>
       <section className="card ds-table">
+        <div className="ds-row head">
+          <span>Dataset</span><span className="num">Snapshots</span><span className="num">They use</span>
+          <span className="num">Dataset uses</span><span className="num">Referenced</span><span>Mount</span>
+        </div>
         {zfs.datasets.map((d) => <DatasetRow key={d.name} d={d} onChanged={load} />)}
       </section>
 

web/src/api.ts

@@ -109,6 +109,10 @@ export interface StorageDevice { name: string; kind: string; model?: string; ven
 export interface SmartAttr { id: number; name: string; value: number; worst: number; thresh: number; raw: string; failed?: string; warn: boolean }
 export interface SmartInfo { device: string; available: boolean; supported: boolean; standby: boolean; passed: boolean | null; temp: number; power_on_hours: number; power_cycles: number; model?: string; serial?: string; firmware?: string; attrs: SmartAttr[]; nvme?: { critical_warning: number; percentage_used: number; available_spare: number; spare_threshold: number; media_errors: number; error_log_entries: number; data_read_bytes: number; data_written_bytes: number; unsafe_shutdowns: number }; mmc?: { life_a: number; life_b: number; pre_eol: number }; self_test?: string; self_test_remaining?: number; self_test_supported: boolean; self_test_note?: string; warnings: string[]; error?: string; fetched_at: number }
 export interface DockerContainer { id: string; name: string; image: string; state: string; status: string; health: string; ports: string; stack: string; service: string; created: string; mem: number }
+export interface DockerHealthProbe { Start: string; End: string; ExitCode: number; Output: string }
+export interface DockerHealth { configured: boolean; state: string; status?: string; failing_streak?: number; log?: DockerHealthProbe[] }
+export interface DockerEvent { t: number; name: string; action: string; image?: string; exit_code?: string; signal?: string; health?: string }
+
 export interface DockerInfo { available: boolean; message?: string; tool?: string; containers: DockerContainer[]; images: Record<string, unknown>[]; volumes: Record<string, unknown>[]; networks: Record<string, unknown>[] }
 export interface Stack { name: string; containers: string[]; running: number; file: string; host_file: string; files: string[]; host_files: string[] | null; images: string[] | null; via: string; remote: string; held_by: string; editable: boolean; note?: string; has_env: boolean }
 export interface StackHome { dir: string; inner: string; via: string; remote: string; held_by: string; stacks: number; default: boolean }
@@ -133,9 +137,12 @@ export interface SpaceNode { name: string; dir: boolean; size: number; apparent:
 export interface SpaceTree extends SpaceNode { path: string; children?: SpaceTree[]; more_n?: number; more_size?: number; total: number }
 export interface SpaceFile { path: string; size: number; apparent: number; mtime: number }
 
-export interface ZFSPolicy { id: string; name: string; dataset: string; recursive: boolean; every: 'hourly' | 'daily' | 'weekly' | 'monthly'; at: string; weekday: number; day: number; keep_for: string; keep_min: number; prefix?: string; enabled: boolean; last_run?: string; last_error?: string; last_taken?: string; pruned?: number }
+export interface KVM { id: string; name: string; url: string; machine: string; added: string }
+
+export interface ZFSPolicy { id: string; name: string; dataset: string; recursive: boolean; every: 'hourly' | 'daily' | 'weekly' | 'monthly' | 'custom'; at: string; weekday: number; day: number; cron?: string; keep_for: string; keep_min: number; prefix?: string; enabled: boolean }
+export interface ZFSRun { at?: string; taken?: string; pruned?: string[]; error?: string }
 export interface ZFSExternal { tool: string; schedule: string; command: string; user: string; source: string; id: string; next?: number; last?: number }
-export interface ZFSOverview { zfs: ZFSInfo; policies: { policy: ZFSPolicy; next?: string }[]; external?: ZFSExternal | null }
+export interface ZFSOverview { zfs: ZFSInfo; policies: { policy: ZFSPolicy; cron: string; next?: string; last: ZFSRun }[]; external?: ZFSExternal | null }
 
 export interface FileEntry { name: string; type: 'dir' | 'file' | 'link' | 'other'; size: number; mode: string; perm: string; mtime: number; user: string; group: string; target?: string; broken?: boolean }
 export interface FileInfo extends FileEntry { path: string; text: boolean }
@@ -183,10 +190,13 @@ export const api = {
   storage: () => req<{ devices: StorageDevice[]; zfs: ZFSInfo; smartctl: boolean }>('GET', '/api/storage'),
   smart: (dev: string, force = false) => req<SmartInfo>('GET', `/api/storage/smart/${dev}${force ? '?force=1' : ''}`),
   smartTest: (dev: string, kind: 'short' | 'long') => req<void>('POST', `/api/storage/smart/${dev}/test/${kind}`),
+  // Always the hub's: it holds every machine's KVM, so the tab works when the machine does not.
+  kvms: () => req<KVM[]>('GET', '/api/kvms', undefined, true),
+  addKVM: (name: string, url: string, machine: string) => req<KVM>('POST', '/api/kvms', { name, url, machine }, true),
+  removeKVM: (id: string) => req<void>('DELETE', `/api/kvms/${encodeURIComponent(id)}`, undefined, true),
   zfsOverview: () => req<ZFSOverview>('GET', '/api/zfs'),
   saveZfsSchedule: (p: Partial<ZFSPolicy>) => req<ZFSPolicy>('PUT', '/api/zfs/schedules', p),
   deleteZfsSchedule: (id: string) => req<void>('DELETE', `/api/zfs/schedules/${encodeURIComponent(id)}`),
-  zfsExpired: (id: string) => req<{ expired: ZFSSnapshot[] }>('GET', `/api/zfs/schedules/${encodeURIComponent(id)}/expired`),
   runZfsSchedule: (id: string) => req<{ taken: string; pruned: string[] }>('POST', `/api/zfs/schedules/${encodeURIComponent(id)}/run`),
   deleteZfsSnapshot: (name: string, recursive: boolean) => req<void>('POST', '/api/zfs/snapshots/delete', { name, confirm: name, recursive }),
   zfsSnapshots: (dataset: string) => req<ZFSSnapshot[]>('GET', `/api/storage/zfs/snapshots?dataset=${encodeURIComponent(dataset)}`),
@@ -242,6 +252,10 @@ export const api = {
   dockerAction: (id: string, action: 'start' | 'stop' | 'restart' | 'remove') => req<void>('POST', `/api/docker/containers/${encodeURIComponent(id)}/${action}`),
   dockerLogs: (id: string, since = '') => req<{ lines: { t: string; m: string }[] }>('GET', `/api/docker/containers/${encodeURIComponent(id)}/logs${since ? `?since=${encodeURIComponent(since)}` : '?n=300'}`),
   dockerInspect: (id: string) => req<Record<string, unknown>>('GET', `/api/docker/containers/${encodeURIComponent(id)}/inspect`),
+  dockerHealth: (id: string) => req<DockerHealth>('GET', `/api/docker/containers/${encodeURIComponent(id)}/health`),
+  dockerEvents: (id: string, since = '7d') => req<{ events: DockerEvent[]; since: string; kept: number; capacity: number }>('GET', `/api/docker/containers/${encodeURIComponent(id)}/events?since=${since}`),
+  dockerExec: (id: string) => req<{ id: string; name: string }>('POST', `/api/docker/containers/${encodeURIComponent(id)}/exec`, {}),
+  dockerLimits: (id: string, p: { memory?: string; cpus?: string; restart?: string }) => req<void>('POST', `/api/docker/containers/${encodeURIComponent(id)}/limits`, p),
   stacks: () => req<{ stacks: Stack[] }>('GET', '/api/docker/stacks'),
   stackHomes: () => req<{ homes: StackHome[] }>('GET', '/api/docker/stacks/homes'),
   createStack: (p: { name: string; text: string; env: string; home: string; env_from?: string }) => req<{ stack: Stack }>('POST', '/api/docker/stacks', p),

web/src/styles.css

@@ -285,14 +285,21 @@ h2.section { margin: 0; font-size: 13px; font-weight: 600; color: var(--muted);
    the fallback above stays a fallback. */
 .disk-cards { grid-template-columns: repeat(auto-fill, minmax(min(420px, 100%), 1fr)); }
 .datasets { padding: 0; overflow-x: auto; }
-.ds-row { display: grid; grid-template-columns: minmax(160px, 1fr) 80px 80px 80px 50px 50px minmax(120px, 1fr); gap: 8px; align-items: center; padding: 6px 12px; border-bottom: 1px solid var(--border); font-size: 13px; min-width: 700px; } /* the columns and gaps add up to 692: at 640 the row ended before its own cells did */
+.ds-row { display: grid; grid-template-columns: minmax(160px, 1fr) 80px 90px 90px 90px minmax(120px, 1fr); gap: 8px; align-items: center; padding: 6px 12px; border-bottom: 1px solid var(--border); font-size: 13px; min-width: 700px; }
 .ds-row.head { color: var(--muted); font-size: 11px; text-transform: uppercase; letter-spacing: .5px; }
 .ds-row .num { text-align: right; }
 .ds-name { cursor: pointer; display: flex; align-items: center; gap: 6px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
 .ds-mount { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; font-family: ui-monospace, monospace; font-size: 12px; }
 .ds-snaps { padding: 8px 12px 10px 40px; border-bottom: 1px solid var(--border); background: var(--bg); display: flex; flex-direction: column; gap: 6px; }
 .snap-list { display: flex; flex-direction: column; gap: 2px; max-height: 40vh; overflow-y: auto; }
-.snap { display: flex; justify-content: space-between; gap: 12px; font-size: 12px; }
+.snap { display: grid; grid-template-columns: minmax(150px, 1fr) 150px 120px 70px auto; gap: 10px; align-items: center; font-size: 12px; padding: 2px 0; border-bottom: 1px solid var(--border); }
+.snap .snap-name { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+.snap .snap-when, .snap .snap-exp { white-space: nowrap; }
+.snap .snap-exp.warn { color: var(--s2); }
+.snap .num { text-align: right; }
+.snap .snap-del { padding: 0 6px; font-size: 11px; opacity: .55; }
+.snap:hover .snap-del { opacity: 1; }
+@media (max-width: 800px) { .snap { grid-template-columns: minmax(120px, 1fr) auto auto; } .snap .snap-when, .snap .snap-exp { display: none; } }
 
 /* files */
 .files { position: absolute; inset: 0; display: flex; flex-direction: column; background: var(--bg); }
@@ -733,3 +740,19 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
 /* ZFS: the schedule dialog's controls match the rest of the modal forms */
 .modal select, .modal input[type=time], .modal input[type=number] { padding: 9px 10px; background: var(--bg); border: 1px solid var(--border); border-radius: 6px; color: var(--fg); font-size: 16px; }
 .modal .chips { margin-top: 2px; }
+
+/* ---- KVM: a device's own web page, framed ---- */
+.kvm { position: absolute; inset: 0; display: flex; flex-direction: column; background: var(--bg); min-height: 0; }
+.kvm-bar { display: flex; align-items: center; gap: 8px; padding: 8px 12px; border-bottom: 1px solid var(--border); background: var(--bg2); flex-wrap: wrap; }
+.kvm-bar .kvm-url { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; max-width: 28ch; }
+.kvm-frame { flex: 1; min-height: 0; background: #000; position: relative; }
+.kvm-frame iframe { width: 100%; height: 100%; border: 0; display: block; background: #000; }
+.kvm-frame.full { background: #000; }
+.kvm-hint { padding: 6px 12px; border-top: 1px solid var(--border); }
+@media (max-width: 800px) { .kvm-bar .kvm-url { display: none; } }
+.kvm-none { text-align: center; } .kvm-none p { margin: 4px 0; }
+/* No true fullscreen for a page element (Safari on iOS): fill the viewport in-page instead. */
+.kvm-frame.expanded { position: fixed; inset: 0; z-index: 60; }
+.kvm-frame.expanded iframe { height: 100%; }
+.kvm-shrink { position: absolute; right: max(10px, env(safe-area-inset-right)); bottom: max(10px, env(safe-area-inset-bottom)); z-index: 61; background: var(--bg2); border: 1px solid var(--border); opacity: .75; }
+.kvm-shrink:hover, .kvm-shrink:active { opacity: 1; }