plan: the Docker tab's Dockhand gap is closed; the cutover is homelab work

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

PLAN.md

@@ -344,10 +344,15 @@ something long in one, switch away and back, refresh — it's all still there.
   does not need to, since one with focus receives them directly. Fullscreen plus
   `navigator.keyboard.lock()` is what takes Escape, Tab and Ctrl-W back from the browser;
   that is Chromium-only, and the tab says which version you are getting rather than
-  pretending. The list is **hub-side** on purpose — a KVM earns its keep when the machine is
+  pretending.
-  down, which is when that machine's own Ham Lab cannot answer — and a KVM need not belong to
+  **One KVM per machine, and the tab follows the machine switcher** (corrected the same day:
-  a machine Ham Lab manages at all. Not done: holding the device's own login on the hub, so
+  the first cut was a hub-side list you pointed at machines, which had it backwards). Switch
-  its page does not ask for a second password.
+  to `grid` and you get grid's; adding a second for a machine replaces the first. The
+  addresses are still kept **on the hub** rather than on each machine, because a KVM earns
+  its keep exactly when the machine is down and its own Ham Lab cannot answer — the hub
+  proxies the device either way — and a KVM need not belong to a machine Ham Lab manages at
+  all. Not done: holding the device's own login on the hub, so its page does not ask for a
+  second password.
 - **Skins** — 13 since 2026-09-19 (`themes.ts`: a palette is one `mk(…)` line): Dark,
   Light, Cyberpunk, Retro, **HUD** (hairlines of cyan on near-black, corner ticks, thin
   type), Nord, Gruvbox, Catppuccin + Latte, Dracula, Tokyo Night, Solarized Dark + Light;
@@ -566,41 +571,40 @@ something long in one, switch away and back, refresh — it's all still there.
   2026-09-24 to take in Docker tab polish and feature work, on the same reasoning as the
   Files item: it is one tab, and the gaps that block retiring the old tool and the things
   that would simply make it better are the same body of work. The *homelab* half —
-  migrating every stack off Dockhand's data dir, then removing the container — is `hamlet`'s
+  the cutover itself, then moving every stack off Dockhand's data dir — is `hamlet`'s
   `docs/pending.md` → "Retire Dockhand; Ham Lab becomes the orchestrator", agreed
   2026-09-20. This entry is the *app* half.
-  Where it stands: Stacks already creates, edits, deploys, updates, restarts, stops, downs
+
-  and deletes stacks on every machine, and Containers does start/stop/restart, logs that
+  **Done 2026-09-24 — the eight Mitchell picked from the inventory**, all in the Docker tab:
-  follow, inspect, and the images/volumes/networks tables. So this is not a rewrite.
+  per-container **Stats**, **Health** (with the last few probe outputs), **Events** (Ham Lab
-  - **What Dockhand still does that Ham Lab does not.**
+  keeps its own 4000-entry ring, since `docker events` forgets after about a minute),
+  **Shell** (`docker exec`, handed to the Terminal tab) and **Limits** (`docker update` for
+  memory, CPU and restart policy); an **updates** tab giving the whole machine at once with
+  a per-container indicator in the list; the same across every machine, asked of the hub;
+  and **schedules**, which write `/etc/cron.d/hamlab-docker` and call
+  `hamlab docker-update <id>` — the shape the ZFS schedules already use.
+  Also done: Ham Lab keeps working when `dockhand` is removed. It remembers where a
+  container's bind mounts pointed (`mounts` in `state.json`) and resolves a stack through
+  that record when no running container explains its path, running compose on the host
+  instead; `repoint` does the same for a schedule at run time. Without this, removing the
+  container would have made 17 stacks on `core` and 5 on `grid` unfindable at once.
+
+  - **Left from the Dockhand inventory, both to decide rather than build.**
     - *Git-sourced stacks.* Dockhand's `stack_sources` table points a stack at a repo and
-      pulls it. Ham Lab has no equivalent. Decide whether to build it or drop it — most
+      pulls it. Its `git_repositories` and `git_stacks` tables are **empty**, so nothing
-      stacks here have no row in that table anyway, so dropping it may be the honest answer.
+      here uses it — dropping it is the honest answer unless Mitchell says otherwise.
-    - *Scheduled image updates.* Ham Lab checks for new images only when asked, since each
+    - *Webhooks, OIDC, LDAP, vulnerability scans.* All four tables empty. Nothing to move.
-      check is a registry request Docker Hub counts, and never updates by itself. If
-      anything relies on Dockhand updating on a schedule, that must move here or be dropped
-      on purpose. Note the Jobs tab and the ZFS schedules already show the shape this would
-      take: a cron line Ham Lab owns, not a ticker in the daemon.
-    - *Webhooks and per-host environments*, if either is in use. Inventory before deciding.
   - **Known gaps in the tab itself**, worth closing whether or not Dockhand goes:
     - *Removing networks* — volumes and images can be pruned and removed, networks cannot.
-    - *Container detail is thin next to the stack view.* No per-container resource limits,
-      restart-policy editing, or exec-into-container from the tab (the Terminal tab is the
-      workaround). Decide which of these belong here at all.
     - *Logs are polled with `--since` rather than followed over a held connection.* That was
       deliberate and works, but a long tail is choppier than the Services tab's live tail,
       which does hold a connection. Worth reconciling the two.
-    - *Image update checks are remembered six hours and asked for per stack*, so there is no
+  - **The `*:2375` question, which this is the moment to settle.** A stack deployed to
-      one view of "what is out of date across the fleet".
+    `grid` or the Mac still goes over `tcp://<node>:2375` from the hub. Decide whether the
-  - **The `*:2375` question, which this is the moment to settle.** Ham Lab runs compose
+    hub should instead ask the node's own Ham Lab to deploy locally, which would end the
-    inside `dockhand` because that is how the stacks were deployed; once they move to a
+    unauthenticated Docker socket exposure — its own item in `docs/pending.md`, currently
-    neutral home that path is unused for them, but a stack deployed to `grid` or the Mac
+    filed under "decided, no action", and this is the change that would make closing it
-    still goes over `tcp://<node>:2375` from the hub. Decide whether the hub should instead
+    free.
-    ask the node's own Ham Lab to deploy locally, which would end the unauthenticated
-    Docker socket exposure — its own item in `docs/pending.md`, currently filed under
-    "decided, no action", and this is the change that would make closing it free.
-  - *Not app work, but the trigger:* nothing above blocks starting the migration except the
-    inventory. Do that first.
 
 - **A Virtualization tab** — asked for 2026-09-23, not started, and **large**. Run and
   manage **LXC containers and virtual machines** from the console, the way Proxmox does —

cmd/hamlab/main.go

@@ -32,8 +32,8 @@ import (
 	"hamlab/internal/files"
 	"hamlab/internal/metrics"
 	"hamlab/internal/nodes"
-	"hamlab/internal/services"
 	"hamlab/internal/priv"
+	"hamlab/internal/services"
 	"hamlab/internal/setup"
 	"hamlab/internal/space"
 	"hamlab/internal/storage"
@@ -62,6 +62,7 @@ func main() {
   passwd [user]        set a user's password (makes the account if it is new; reads stdin when piped)
   users                list the accounts
   zfs-snapshot <id>    run one ZFS snapshot schedule (what its cron line calls)
+  docker-update <id>   run one image-update schedule (what its cron line calls)
   user-rm <user>       remove an account and sign it out everywhere
   check                verify the config and state load, then exit
   version
@@ -132,6 +133,10 @@ func main() {
 		if err := passwd(cfg, flag.Arg(1)); err != nil {
 			log.Fatal(err)
 		}
+	case "docker-update":
+		if err := dockerUpdateJob(cfg, flag.Arg(1)); err != nil {
+			log.Fatal(err)
+		}
 	case "zfs-snapshot":
 		if err := zfsSnapshot(cfg, flag.Arg(1)); err != nil {
 			log.Fatal(err)
@@ -185,6 +190,48 @@ func main() {
 
 // migrateUsers gives a pre-users config its first account, named after this OS user, so
 // the CLI sees the same accounts the daemon will.
+// dockerUpdateJob pulls a stack's images, and deploys them when the schedule says to. This
+// is what the cron line Ham Lab writes actually calls, so image updates keep happening
+// whether or not the console is up.
+func dockerUpdateJob(cfg *config.Config, id string) error {
+	if strings.TrimSpace(id) == "" {
+		return errors.New("usage: hamlab docker-update <schedule-id>")
+	}
+	st, err := store.Open(filepath.Join(cfg.DataDir(), "state.json"))
+	if err != nil {
+		return err
+	}
+	var raw json.RawMessage
+	st.View(func(s *store.State) { raw = s.DockerJobs })
+	var d struct {
+		Jobs []api.UpdateJob `json:"jobs"`
+	}
+	if len(raw) > 0 {
+		if err := json.Unmarshal(raw, &d); err != nil {
+			return fmt.Errorf("reading the schedules: %w", err)
+		}
+	}
+	for _, j := range d.Jobs {
+		if j.ID != id {
+			continue
+		}
+		// A pull that never finishes must not leave a cron job resident until the next reboot.
+		ctx, cancel := context.WithTimeout(context.Background(), 2*time.Hour)
+		defer cancel()
+		out, runErr := api.RunUpdate(ctx, st, j, api.DockerTool())
+		rec := storage.RunRecord{At: time.Now(), Taken: j.Stack}
+		if runErr != nil {
+			rec.Error = runErr.Error()
+		}
+		if err := storage.WriteRun(cfg.DataDir(), "docker-"+id, rec); err != nil {
+			fmt.Fprintln(os.Stderr, "recording the run:", err)
+		}
+		fmt.Print(out)
+		return runErr
+	}
+	return fmt.Errorf("no image-update schedule %q — it may have been removed; this cron line is stale", id)
+}
+
 // zfsSnapshot runs one snapshot schedule: take it, then expire what has aged out. This is
 // what the cron line Ham Lab writes actually calls, so a schedule keeps running whether or
 // not the console is up — and keeps running if Ham Lab is never opened again.

internal/api/docker.go

@@ -49,6 +49,10 @@ func dockerTool() string {
 }
 
 // dockerRun runs the tool as this user, and again as root if the socket said no.
+// DockerTool is whichever of docker or podman this machine has, for callers outside the
+// package — the scheduled updates the CLI runs.
+func DockerTool() string { return dockerTool() }
+
 func dockerRun(ctx context.Context, args ...string) ([]byte, error) {
 	tool := dockerTool()
 	if tool == "" {
@@ -394,3 +398,161 @@ func (s *Server) volumePrune(w http.ResponseWriter, r *http.Request) {
 	log.Printf("docker: volume prune (from %s)", clientIP(r))
 	jsonOK(w, 200, map[string]string{"result": lines[len(lines)-1]})
 }
+
+// ---- a container's health, its recent life, its console, and its limits ----
+
+// dockerHealth is the health check's verdict and the last few probe results. Docker keeps
+// the last five itself, which is all anyone wants: is it failing, and what did it say.
+func (s *Server) dockerHealth(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	if !dockerRef.MatchString(id) {
+		jsonErr(w, 400, "bad container")
+		return
+	}
+	ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
+	defer cancel()
+	out, err := dockerRun(ctx, "inspect", "--format", "{{json .State}}", id)
+	if err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	var st struct {
+		Status string `json:"Status"`
+		Health *struct {
+			Status        string `json:"Status"`
+			FailingStreak int    `json:"FailingStreak"`
+			Log           []struct {
+				Start    time.Time `json:"Start"`
+				End      time.Time `json:"End"`
+				ExitCode int       `json:"ExitCode"`
+				Output   string    `json:"Output"`
+			} `json:"Log"`
+		} `json:"Health"`
+	}
+	if json.Unmarshal(out, &st) != nil {
+		jsonErr(w, 500, "could not read the container's state")
+		return
+	}
+	if st.Health == nil {
+		jsonOK(w, 200, map[string]any{"configured": false, "state": st.Status})
+		return
+	}
+	jsonOK(w, 200, map[string]any{
+		"configured": true, "state": st.Status, "status": st.Health.Status,
+		"failing_streak": st.Health.FailingStreak, "log": st.Health.Log,
+	})
+}
+
+// dockerExec opens a shell inside a container, as a terminal session like any other — so
+// it reattaches, resizes and scrolls back the same way, and works on a managed machine
+// through the same proxy.
+func (s *Server) dockerExec(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	if !dockerRef.MatchString(id) {
+		jsonErr(w, 400, "bad container")
+		return
+	}
+	tool := dockerTool()
+	if tool == "" {
+		jsonErr(w, 500, "neither docker nor podman is installed here")
+		return
+	}
+	var body struct {
+		Shell string `json:"shell"`
+	}
+	_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
+	shell := strings.TrimSpace(body.Shell)
+	if shell == "" {
+		// most images have sh; bash is nicer where it exists, so try it first and fall back
+		shell = "if command -v bash >/dev/null 2>&1; then exec bash; else exec sh; fi"
+	} else if strings.ContainsAny(shell, "\n\r") {
+		jsonErr(w, 400, "bad shell")
+		return
+	}
+	argv := []string{tool, "exec", "-it", id, "sh", "-c", shell}
+	if priv.Available() && !dockerReachable() {
+		argv = priv.Wrap(argv...)
+	}
+	sess, err := s.mgr.CreateCmd("", "sh "+shortName(id), argv)
+	if err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	log.Printf("docker exec into %s as terminal %s by %s", id, sess.ID, clientIP(r))
+	jsonOK(w, 200, map[string]string{"id": sess.ID, "name": "sh " + shortName(id)})
+}
+
+func shortName(id string) string {
+	if len(id) > 20 {
+		return id[:20]
+	}
+	return id
+}
+
+// dockerReachable reports whether docker answers without help.
+func dockerReachable() bool {
+	tool := dockerTool()
+	if tool == "" {
+		return false
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	return exec.CommandContext(ctx, tool, "version", "--format", "{{.Server.Version}}").Run() == nil
+}
+
+// dockerUpdate changes a running container's limits and restart policy. `docker update`
+// does this live, without recreating it and without touching the compose file — which is
+// also the catch, and the caller is told: a deploy from compose puts back whatever compose
+// says.
+func (s *Server) dockerUpdate(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	if !dockerRef.MatchString(id) {
+		jsonErr(w, 400, "bad container")
+		return
+	}
+	var body struct {
+		Memory  string `json:"memory"`  // "512m", "2g", or "" to leave alone; "0" removes the limit
+		CPUs    string `json:"cpus"`    // "1.5", or "0" for no limit
+		Restart string `json:"restart"` // no | on-failure | unless-stopped | always
+	}
+	if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body); err != nil {
+		jsonErr(w, 400, "bad request")
+		return
+	}
+	args := []string{"update"}
+	if m := strings.TrimSpace(body.Memory); m != "" {
+		if !regexp.MustCompile(`^[0-9]+([bkmgBKMG])?$`).MatchString(m) {
+			jsonErr(w, 400, "memory: a number, optionally with b, k, m or g — 0 removes the limit")
+			return
+		}
+		args = append(args, "--memory", m, "--memory-swap", m)
+	}
+	if c := strings.TrimSpace(body.CPUs); c != "" {
+		if !regexp.MustCompile(`^[0-9]+(\.[0-9]+)?$`).MatchString(c) {
+			jsonErr(w, 400, "cpus: a number like 1.5 — 0 removes the limit")
+			return
+		}
+		args = append(args, "--cpus", c)
+	}
+	if p := strings.TrimSpace(body.Restart); p != "" {
+		switch p {
+		case "no", "on-failure", "unless-stopped", "always":
+		default:
+			jsonErr(w, 400, "restart: no, on-failure, unless-stopped or always")
+			return
+		}
+		args = append(args, "--restart", p)
+	}
+	if len(args) == 1 {
+		jsonErr(w, 400, "nothing to change")
+		return
+	}
+	ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
+	defer cancel()
+	if _, err := dockerRun(ctx, append(args, id)...); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	log.Printf("docker update %s %v by %s", id, args[1:], clientIP(r))
+	w.WriteHeader(204)
+}

internal/api/docker_events.go

@@ -0,0 +1,229 @@
+package api
+
+// A container's history.
+//
+// `docker events` is a live stream with only a short in-memory backlog — ask it for the
+// last thirty days and you get the last few minutes, because the daemon keeps events in a
+// small ring and loses them all when it restarts. That is why Dockhand kept its own copy,
+// and why this does too: a watcher follows the stream and appends the lifecycle events to a
+// ring kept beside the config, so "what happened to this container last Tuesday" has an
+// answer.
+//
+// Only lifecycle events are kept. A health check produces three exec events every probe,
+// which is thousands a day and tells you nothing that the health panel does not.
+
+import (
+	"bufio"
+	"context"
+	"encoding/json"
+	"log"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"sort"
+	"strings"
+	"sync"
+	"time"
+
+	"hamlab/internal/priv"
+)
+
+const (
+	eventsKept     = 4000
+	eventsSaveEvery = 30 * time.Second
+)
+
+// ContainerEvent is one thing that happened to one container.
+type ContainerEvent struct {
+	T        int64  `json:"t"` // unix ms
+	Name     string `json:"name"`
+	Action   string `json:"action"`
+	Image    string `json:"image,omitempty"`
+	ExitCode string `json:"exit_code,omitempty"`
+	Signal   string `json:"signal,omitempty"`
+	Health   string `json:"health,omitempty"`
+}
+
+type eventLog struct {
+	mu    sync.Mutex
+	ring  []ContainerEvent
+	dirty bool
+	path  string
+}
+
+func (e *eventLog) load(path string) {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+	e.path = path
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return
+	}
+	var list []ContainerEvent
+	if json.Unmarshal(b, &list) == nil {
+		e.ring = list
+	}
+}
+
+func (e *eventLog) add(ev ContainerEvent) {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+	e.ring = append(e.ring, ev)
+	if len(e.ring) > eventsKept {
+		e.ring = append([]ContainerEvent(nil), e.ring[len(e.ring)-eventsKept:]...)
+	}
+	e.dirty = true
+}
+
+// forName is one container's events, newest first.
+func (e *eventLog) forName(name string, since time.Time, limit int) []ContainerEvent {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+	out := []ContainerEvent{}
+	for _, ev := range e.ring {
+		if ev.Name == name && ev.T >= since.UnixMilli() {
+			out = append(out, ev)
+		}
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].T > out[j].T })
+	if len(out) > limit {
+		out = out[:limit]
+	}
+	return out
+}
+
+func (e *eventLog) save() {
+	e.mu.Lock()
+	if !e.dirty || e.path == "" {
+		e.mu.Unlock()
+		return
+	}
+	b, err := json.Marshal(e.ring)
+	e.dirty = false
+	path := e.path
+	e.mu.Unlock()
+	if err != nil {
+		return
+	}
+	_ = os.MkdirAll(filepath.Dir(path), 0o700)
+	tmp := path + ".tmp"
+	if os.WriteFile(tmp, b, 0o600) == nil {
+		_ = os.Rename(tmp, path)
+	}
+}
+
+// watchDockerEvents follows the stream, restarting if Docker goes away. It is the only
+// thing that writes the log, so nothing contends with it.
+func (s *Server) watchDockerEvents(ctx context.Context) {
+	s.events.load(filepath.Join(s.cfg.DataDir(), "docker-events.json"))
+	go func() { // a periodic flush, so a crash loses at most half a minute of history
+		t := time.NewTicker(eventsSaveEvery)
+		defer t.Stop()
+		for {
+			select {
+			case <-ctx.Done():
+				s.events.save()
+				return
+			case <-t.C:
+				s.events.save()
+			}
+		}
+	}()
+	for {
+		if ctx.Err() != nil {
+			return
+		}
+		if dockerTool() == "" {
+			time.Sleep(time.Minute) // no Docker here; look again later in case it is installed
+			continue
+		}
+		s.followEvents(ctx)
+		select {
+		case <-ctx.Done():
+			return
+		case <-time.After(10 * time.Second): // the daemon restarted, or the pipe broke
+		}
+	}
+}
+
+func (s *Server) followEvents(ctx context.Context) {
+	args := []string{"events", "--format", "{{json .}}", "--filter", "type=container"}
+	for _, e := range []string{"create", "start", "restart", "stop", "kill", "die", "destroy", "oom", "pause", "unpause", "rename", "update", "health_status"} {
+		args = append(args, "--filter", "event="+e)
+	}
+	tool := dockerTool()
+	full := append([]string{tool}, args...)
+	if !dockerReachable() && priv.Available() {
+		full = priv.Wrap(full...)
+	}
+	cmd := exec.CommandContext(ctx, full[0], full[1:]...)
+	out, err := cmd.StdoutPipe()
+	if err != nil {
+		return
+	}
+	if err := cmd.Start(); err != nil {
+		return
+	}
+	sc := bufio.NewScanner(out)
+	sc.Buffer(make([]byte, 64<<10), 1<<20)
+	n := 0
+	for sc.Scan() {
+		var e struct {
+			Action   string `json:"Action"`
+			Time     int64  `json:"time"`
+			TimeNano int64  `json:"timeNano"`
+			Actor    struct {
+				Attributes map[string]string `json:"Attributes"`
+			} `json:"Actor"`
+		}
+		if json.Unmarshal(sc.Bytes(), &e) != nil || e.Action == "" {
+			continue
+		}
+		at := e.Actor.Attributes
+		ev := ContainerEvent{
+			T: e.Time * 1000, Name: at["name"], Image: at["image"],
+			Action: e.Action, ExitCode: at["exitCode"], Signal: at["signal"],
+		}
+		if e.TimeNano > 0 {
+			ev.T = e.TimeNano / 1e6
+		}
+		// health_status comes through as "health_status: healthy"
+		if a, h, ok := strings.Cut(e.Action, ": "); ok {
+			ev.Action, ev.Health = a, h
+		}
+		if ev.Name == "" {
+			continue
+		}
+		s.events.add(ev)
+		n++
+	}
+	_ = cmd.Wait()
+	if n > 0 {
+		log.Printf("docker events: stream ended after %d events; reconnecting", n)
+	}
+}
+
+// dockerEvents answers from Ham Lab's own history, not Docker's short memory.
+func (s *Server) dockerEvents(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	if !dockerRef.MatchString(id) {
+		jsonErr(w, 400, "bad container")
+		return
+	}
+	since := r.URL.Query().Get("since")
+	d := 7 * 24 * time.Hour
+	switch since {
+	case "24h":
+		d = 24 * time.Hour
+	case "30d":
+		d = 30 * 24 * time.Hour
+	default:
+		since = "7d"
+	}
+	ev := s.events.forName(id, time.Now().Add(-d), 500)
+	s.events.mu.Lock()
+	total := len(s.events.ring)
+	s.events.mu.Unlock()
+	jsonOK(w, 200, map[string]any{"events": ev, "since": since, "kept": total, "capacity": eventsKept})
+}

internal/api/dockersched.go

@@ -0,0 +1,351 @@
+package api
+
+// Scheduled image updates.
+//
+// Dockhand kept three containers up to date nightly — unbound, nginx and pihole, which is
+// DNS and ingress — and that is the one thing it does that Ham Lab did not. This replaces
+// it, and deliberately in the same shape as the ZFS snapshot schedules: a real line in a
+// cron file Ham Lab owns, calling the binary, rather than a ticker inside the daemon.
+// Updating an image is not something that should stop happening because a web console is
+// down, and a cron line is visible to anyone looking at the machine the normal way.
+//
+// A schedule stores everything the run needs — project, compose files, how compose has to
+// be invoked for that stack — so the cron job is self-contained and does not have to
+// rediscover the world before it can act.
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"io"
+	"log"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+
+	"hamlab/internal/compose"
+	"hamlab/internal/cron"
+	"hamlab/internal/storage"
+	"hamlab/internal/store"
+)
+
+// UpdateJob is a standing instruction to pull a stack's images, and optionally deploy them.
+type UpdateJob struct {
+	ID      string `json:"id"`
+	Name    string `json:"name"`
+	Stack   string `json:"stack"`
+	Machine string `json:"machine,omitempty"` // for display; the job runs where it was made
+	// Services narrows it to some of the stack's services; empty is all of them.
+	Services []string `json:"services,omitempty"`
+	Cron     string   `json:"cron"`
+	// Apply says whether a newer image is deployed or only fetched. Check-only still pulls,
+	// so the Docker tab shows what is behind without a registry round trip.
+	Apply   bool `json:"apply"`
+	Enabled bool `json:"enabled"`
+
+	// How to run compose for this stack, captured when the schedule was made.
+	Spec composeSpec `json:"spec"`
+}
+
+type composeSpec struct {
+	Project string   `json:"project"`
+	Files   []string `json:"files"`
+	Via     string   `json:"via,omitempty"`
+	Remote  string   `json:"remote,omitempty"`
+}
+
+func (c composeSpec) toSpec(tool string) compose.Spec {
+	return compose.Spec{Project: c.Project, Files: c.Files, Via: c.Via, Remote: c.Remote, Tool: tool}
+}
+
+const dockerCronFile = "/etc/cron.d/hamlab-docker"
+
+type dockerSchedState struct {
+	Jobs []UpdateJob `json:"jobs"`
+}
+
+func (s *Server) dsLoad() dockerSchedState {
+	var raw json.RawMessage
+	s.store.View(func(st *store.State) { raw = st.DockerJobs })
+	var d dockerSchedState
+	if len(raw) > 0 {
+		_ = json.Unmarshal(raw, &d)
+	}
+	if d.Jobs == nil {
+		d.Jobs = []UpdateJob{}
+	}
+	return d
+}
+
+func (s *Server) dsSave(d dockerSchedState) error {
+	b, err := json.Marshal(d)
+	if err != nil {
+		return err
+	}
+	return s.store.Update(func(st *store.State) { st.DockerJobs = b })
+}
+
+// dsCronText renders the schedules as cron lines. Same shape as the ZFS file: Ham Lab owns
+// it whole and rewrites it on every change.
+func dsCronText(jobs []UpdateJob, user, exe, config string) (string, error) {
+	var b strings.Builder
+	b.WriteString("# Ham Lab image-update schedules — written by the Docker tab, rewritten whole on every change.\n")
+	b.WriteString("# Edit them there, not here: anything added by hand is lost the next time a schedule changes.\n")
+	b.WriteString("SHELL=/bin/sh\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n\n")
+	for i := range jobs {
+		j := &jobs[i]
+		if !j.Enabled {
+			b.WriteString("# " + j.Name + " — paused\n")
+			continue
+		}
+		expr := strings.Join(strings.Fields(j.Cron), " ")
+		if err := cron.ValidSchedule(expr); err != nil {
+			return "", err
+		}
+		what := "pull only"
+		if j.Apply {
+			what = "pull and deploy"
+		}
+		b.WriteString("# " + j.Name + " — " + j.Stack + ", " + what + "\n")
+		b.WriteString(expr + " " + user + " " + exe + " -config " + config + " docker-update " + j.ID + "\n\n")
+	}
+	return b.String(), nil
+}
+
+func (s *Server) dsSyncCron(ctx context.Context, d dockerSchedState) error {
+	live := false
+	for i := range d.Jobs {
+		if d.Jobs[i].Enabled {
+			live = true
+		}
+	}
+	if !live {
+		return s.storage.RemoveRootFile(ctx, dockerCronFile)
+	}
+	exe, err := os.Executable()
+	if err != nil {
+		return err
+	}
+	exe, _ = filepath.EvalSymlinks(exe)
+	text, err := dsCronText(d.Jobs, s.user, exe, s.cfg.Path())
+	if err != nil {
+		return err
+	}
+	return s.storage.WriteRootFile(ctx, dockerCronFile, text)
+}
+
+// repoint keeps a schedule working when what deployed its stack has been removed.
+//
+// A schedule captures how to invoke compose at the moment it is made, so the cron job never
+// has to rediscover the world at 3am. But "run compose inside this container" stops being
+// true the day that container is deleted — which is exactly what happens when an
+// orchestrator is retired, and would otherwise break every schedule made before it went.
+// So: if the container named is not running, the files are looked up again, through live
+// mounts and remembered ones both, and compose runs on the host instead.
+func repoint(ctx context.Context, db *store.Store, spec compose.Spec) compose.Spec {
+	if spec.Via == "" || db == nil {
+		return spec
+	}
+	if out, err := dockerRun(ctx, "inspect", "-f", "{{.State.Running}}", spec.Via); err == nil && strings.TrimSpace(string(out)) == "true" {
+		return spec
+	}
+	st := stack{Name: spec.Project, File: spec.Files[0], Files: spec.Files}
+	st.resolve(mountsSeen(ctx, db)) // read-only: this also runs from cron, where the daemon owns state.json
+	if !st.Editable || st.Via != "" {
+		return spec // still only reachable through a container, or not reachable at all
+	}
+	log.Printf("docker schedule %s: %s is gone, running compose here against %s", spec.Project, spec.Via, st.HostFiles[0])
+	spec.Files, spec.Via = st.HostFiles, ""
+	return spec
+}
+
+// RunUpdate pulls, and deploys when the schedule says to. Shared with the CLI so the button
+// and the cron line cannot drift apart.
+func RunUpdate(ctx context.Context, db *store.Store, j UpdateJob, tool string) (string, error) {
+	if len(j.Spec.Files) == 0 {
+		return "", errors.New("this schedule has no compose file recorded — open it in the Docker tab and save it again")
+	}
+	spec := repoint(ctx, db, j.Spec.toSpec(tool))
+	var log strings.Builder
+	run := func(args ...string) error {
+		cmd := compose.Cmd(ctx, spec, args...)
+		out, err := cmd.CombinedOutput()
+		log.Write(out)
+		return err
+	}
+	args := append([]string{"pull"}, j.Services...)
+	if err := run(args...); err != nil {
+		return log.String(), err
+	}
+	if !j.Apply {
+		return log.String(), nil
+	}
+	up := append([]string{"up", "-d"}, j.Services...)
+	if err := run(up...); err != nil {
+		return log.String(), err
+	}
+	return log.String(), nil
+}
+
+// ---- endpoints ----
+
+func (s *Server) dsList(w http.ResponseWriter, _ *http.Request) {
+	d := s.dsLoad()
+	now := time.Now()
+	out := make([]map[string]any, 0, len(d.Jobs))
+	for _, j := range d.Jobs {
+		m := map[string]any{"job": j, "last": storage.ReadRun(s.cfg.DataDir(), "docker-"+j.ID)}
+		if j.Enabled {
+			if n, ok := cron.NextRun(strings.Join(strings.Fields(j.Cron), " "), now); ok {
+				m["next"] = n
+			}
+		}
+		out = append(out, m)
+	}
+	jsonOK(w, 200, map[string]any{"jobs": out, "file": dockerCronFile})
+}
+
+func (s *Server) dsPut(w http.ResponseWriter, r *http.Request) {
+	var j UpdateJob
+	if err := json.NewDecoder(io.LimitReader(r.Body, 32<<10)).Decode(&j); err != nil {
+		jsonErr(w, 400, "bad request")
+		return
+	}
+	j.Cron = strings.Join(strings.Fields(j.Cron), " ")
+	if err := cron.ValidSchedule(j.Cron); err != nil {
+		jsonErr(w, 400, "schedule: "+err.Error())
+		return
+	}
+	if strings.TrimSpace(j.Stack) == "" {
+		jsonErr(w, 400, "which stack?")
+		return
+	}
+	// resolve the stack now, so the cron job never has to
+	ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
+	defer cancel()
+	stacks, err := s.findStacks(ctx)
+	if err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	found := false
+	for _, st := range stacks {
+		if st.Name == j.Stack {
+			// seenFiles is the list as compose itself sees it: host paths normally, the
+			// orchestrator container's own paths when compose runs inside one. Getting this
+			// backwards gives a cron line that works from the console and fails at 3am.
+			j.Spec = composeSpec{Project: st.Name, Files: seenFiles(st, -1, ""), Via: st.Via, Remote: st.Remote}
+			if len(j.Spec.Files) == 0 {
+				jsonErr(w, 409, "that stack's compose file isn't anywhere this console can reach")
+				return
+			}
+			found = true
+		}
+	}
+	if !found {
+		jsonErr(w, 404, "no stack called "+j.Stack+" on this machine")
+		return
+	}
+	if strings.TrimSpace(j.Name) == "" {
+		j.Name = j.Stack
+	}
+	d := s.dsLoad()
+	replaced := false
+	for i := range d.Jobs {
+		if d.Jobs[i].ID == j.ID && j.ID != "" {
+			d.Jobs[i] = j
+			replaced = true
+		}
+	}
+	if !replaced {
+		b := make([]byte, 6)
+		if _, err := rand.Read(b); err != nil {
+			jsonErr(w, 500, err.Error())
+			return
+		}
+		j.ID = hex.EncodeToString(b)
+		d.Jobs = append(d.Jobs, j)
+	}
+	if err := s.dsSave(d); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	if err := s.dsSyncCron(r.Context(), d); err != nil {
+		jsonErr(w, 500, "saved, but writing "+dockerCronFile+" failed: "+err.Error())
+		return
+	}
+	log.Printf("docker schedule %q saved by %s (%s, %q, %s)", j.Name, clientIP(r), j.Stack, j.Cron, map[bool]string{true: "pull and deploy", false: "pull only"}[j.Apply])
+	jsonOK(w, 200, j)
+}
+
+func (s *Server) dsDelete(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	d := s.dsLoad()
+	kept := d.Jobs[:0]
+	found := false
+	for _, j := range d.Jobs {
+		if j.ID == id {
+			found = true
+			continue
+		}
+		kept = append(kept, j)
+	}
+	if !found {
+		jsonErr(w, 404, "no such schedule")
+		return
+	}
+	d.Jobs = kept
+	if err := s.dsSave(d); err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	if err := s.dsSyncCron(r.Context(), d); err != nil {
+		jsonErr(w, 500, "removed, but writing "+dockerCronFile+" failed: "+err.Error())
+		return
+	}
+	storage.ForgetRun(s.cfg.DataDir(), "docker-"+id)
+	log.Printf("docker schedule %s removed by %s", id, clientIP(r))
+	w.WriteHeader(204)
+}
+
+func (s *Server) dsRun(w http.ResponseWriter, r *http.Request) {
+	id := r.PathValue("id")
+	var job UpdateJob
+	found := false
+	for _, j := range s.dsLoad().Jobs {
+		if j.ID == id {
+			job, found = j, true
+		}
+	}
+	if !found {
+		jsonErr(w, 404, "no such schedule")
+		return
+	}
+	// Outliving the request: a pull the browser gave up on still finishes, but not forever.
+	ctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), 30*time.Minute)
+	defer cancel()
+	out, err := RunUpdate(ctx, s.store, job, dockerTool())
+	rec := storage.RunRecord{At: time.Now(), Taken: job.Stack}
+	if err != nil {
+		rec.Error = err.Error()
+	}
+	_ = storage.WriteRun(s.cfg.DataDir(), "docker-"+id, rec)
+	stackUpdatesForget("/" + job.Stack)
+	if err != nil {
+		jsonErr(w, 500, err.Error()+"\n"+tail(out, 2000))
+		return
+	}
+	log.Printf("docker schedule %q run by %s", job.Name, clientIP(r))
+	jsonOK(w, 200, map[string]string{"output": tail(out, 4000)})
+}
+
+func tail(s string, n int) string {
+	if len(s) <= n {
+		return s
+	}
+	return "…" + s[len(s)-n:]
+}

internal/api/server.go

@@ -69,6 +69,7 @@ type Server struct {
 	storage     *storage.Manager
 	space       *space.Manager
 	update      updateState
+	events      eventLog
 	kvms        kvmProxies
 	fsLocal     files.FS
 	fsSudo      files.FS // nil without passwordless sudo
@@ -138,6 +139,7 @@ func New(cfg *config.Config, sessions *auth.Sessions, mgr *term.Manager, col *me
 		go s.keepFleetInStep()
 	}
 	go s.watchForUpdates(context.Background())
+	go s.watchDockerEvents(context.Background())
 
 	m := http.NewServeMux()
 	m.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(204) })
@@ -190,6 +192,12 @@ func New(cfg *config.Config, sessions *auth.Sessions, mgr *term.Manager, col *me
 	m.HandleFunc("POST /api/docker/stacks/{name}/{action}", s.auth(s.stackAction))
 	m.HandleFunc("GET /api/docker/stacks/{name}/diff", s.auth(s.stackDiff))
 	m.HandleFunc("GET /api/docker/stacks/{name}/updates", s.auth(s.stackUpdates))
+	m.HandleFunc("GET /api/docker/updates", s.auth(s.machineUpdates))
+	m.HandleFunc("GET /api/docker/schedules", s.auth(s.dsList))
+	m.HandleFunc("PUT /api/docker/schedules", s.auth(s.dsPut))
+	m.HandleFunc("DELETE /api/docker/schedules/{id}", s.auth(s.dsDelete))
+	m.HandleFunc("POST /api/docker/schedules/{id}/run", s.auth(s.dsRun))
+	m.HandleFunc("GET /api/nodes/{node}/docker/updates", s.auth(s.machineUpdates))
 	// The same, asked of a node through this hub: the hub answers for the stacks whose files
 	// it holds (deployed to the node over tcp) and passes everything else to the node.
 	m.HandleFunc("GET /api/nodes/{node}/docker/stacks", s.auth(s.stacksList))
@@ -206,6 +214,10 @@ func New(cfg *config.Config, sessions *auth.Sessions, mgr *term.Manager, col *me
 	m.HandleFunc("POST /api/docker/containers/{id}/{action}", s.auth(s.dockerAction))
 	m.HandleFunc("GET /api/docker/containers/{id}/logs", s.auth(s.dockerLogs))
 	m.HandleFunc("GET /api/docker/containers/{id}/inspect", s.auth(s.dockerInspect))
+	m.HandleFunc("GET /api/docker/containers/{id}/health", s.auth(s.dockerHealth))
+	m.HandleFunc("GET /api/docker/containers/{id}/events", s.auth(s.dockerEvents))
+	m.HandleFunc("POST /api/docker/containers/{id}/exec", s.auth(s.dockerExec))
+	m.HandleFunc("POST /api/docker/containers/{id}/limits", s.auth(s.dockerUpdate))
 	m.HandleFunc("GET /api/alerts", s.auth(s.alertsGet))
 	m.HandleFunc("PUT /api/alerts", s.auth(s.alertsPut))
 	m.HandleFunc("POST /api/alerts/test", s.auth(s.alertsTest))

internal/api/stacks.go

@@ -12,6 +12,12 @@ package api
 // relative paths, the .env beside them and the labels all stay exactly as they were
 // deployed — else on the host.
 //
+// Those mounts are also written down. An orchestrator can be removed, and when it is, the
+// containers it left behind still carry labels naming paths inside a filesystem that no
+// longer exists; without a record of where that filesystem was mounted, every stack on the
+// machine would become unfindable at once. A remembered mount resolves with no container to
+// run in, which is right: compose runs on the host against the host's copy of the file.
+//
 // A stack can also be deployed *from another machine*: Dockhand on the hub drives a node's
 // Docker over tcp, so the node runs the containers and the hub holds the files. The node
 // alone can only list such a stack read-only. The hub, asked about one of its nodes, fills
@@ -39,7 +45,9 @@ import (
 	"sync"
 	"time"
 
+	"hamlab/internal/compose"
 	"hamlab/internal/priv"
+	"hamlab/internal/store"
 )
 
 type stack struct {
@@ -89,6 +97,100 @@ func stackMounts(ctx context.Context) []mountMap {
 	return maps
 }
 
+// mountsFor is stackMounts plus memory. An orchestrator that deployed everything here and
+// has since been removed takes the only record of where its files really are with it: the
+// containers it left behind are labelled with paths inside *its* filesystem. So every map
+// seen is written down, and a path that no live container explains is looked for in what
+// was seen before. A remembered map resolves with no `via`, which is correct — the
+// container is gone, so compose runs on the host, against the host's copy of the file.
+func mountsFor(ctx context.Context, db *store.Store) []mountMap {
+	live := stackMounts(ctx)
+	rememberMounts(db, live)
+	return append(live, recallMounts(db, live)...)
+}
+
+// mountsSeen is the same without writing anything down. The daemon owns state.json; a cron
+// job that wrote to it would be clobbered by the next thing the daemon saved, so out there
+// the record is read and never added to.
+func mountsSeen(ctx context.Context, db *store.Store) []mountMap {
+	live := stackMounts(ctx)
+	return append(live, recallMounts(db, live)...)
+}
+
+// rememberMounts folds what is mounted now into what was mounted before, and forgets a map
+// whose source has gone from disk — that one is not coming back, and keeping it would make
+// a stack look editable when its file no longer exists.
+//
+// This is on the path of every stack listing, which the Docker tab polls, so it reads first
+// and only writes when the set actually changed.
+func rememberMounts(db *store.Store, live []mountMap) {
+	if len(live) == 0 {
+		return // docker is down or answered nothing; that is not evidence of anything
+	}
+	key := func(c, d string) string { return c + "\x00" + d }
+	var have []store.RememberedMount
+	db.View(func(st *store.State) { have = append([]store.RememberedMount(nil), st.Mounts...) })
+	at := map[string]int{}
+	for i, m := range have {
+		at[key(m.Container, m.Dest)] = i
+	}
+	now, add, moved := time.Now(), 0, false
+	for _, m := range live {
+		i, ok := at[key(m.container, m.dest)]
+		if !ok {
+			add++
+			continue
+		}
+		if have[i].Src != m.src {
+			moved = true
+		}
+	}
+	// Pruning is the expensive half — a stat, or a sudo test(1), per remembered map — so it
+	// only happens when something else has already made a write necessary.
+	if add == 0 && !moved {
+		return
+	}
+	_ = db.Update(func(st *store.State) {
+		at := map[string]int{}
+		for i, m := range st.Mounts {
+			at[key(m.Container, m.Dest)] = i
+		}
+		for _, m := range live {
+			if i, ok := at[key(m.container, m.dest)]; ok {
+				st.Mounts[i].Src, st.Mounts[i].Seen = m.src, now
+				continue
+			}
+			st.Mounts = append(st.Mounts, store.RememberedMount{Container: m.container, Dest: m.dest, Src: m.src, Seen: now})
+		}
+		kept := st.Mounts[:0]
+		for _, m := range st.Mounts {
+			if readable(m.Src) {
+				kept = append(kept, m)
+			}
+		}
+		st.Mounts = kept
+	})
+}
+
+// recallMounts is what was seen before and is not running now.
+func recallMounts(db *store.Store, live []mountMap) []mountMap {
+	var remembered []store.RememberedMount
+	db.View(func(st *store.State) { remembered = append([]store.RememberedMount(nil), st.Mounts...) })
+	now := map[string]bool{}
+	for _, m := range live {
+		now[m.container+"\x00"+m.dest] = true
+	}
+	var out []mountMap
+	for _, m := range remembered {
+		if now[m.Container+"\x00"+m.Dest] {
+			continue
+		}
+		out = append(out, mountMap{"", filepath.Clean(m.Dest), filepath.Clean(m.Src)}) // no container: run compose here
+	}
+	sort.Slice(out, func(i, j int) bool { return len(out[i].dest) > len(out[j].dest) })
+	return out
+}
+
 func readable(p string) bool {
 	if _, err := os.Stat(p); err == nil {
 		return true
@@ -172,7 +274,7 @@ func (s *Server) findStacks(ctx context.Context) ([]stack, error) {
 			}
 		}
 	}
-	maps := stackMounts(ctx)
+	maps := mountsFor(ctx, s.store)
 	stacks := []stack{}
 	for _, st := range by {
 		sort.Strings(st.Containers)
@@ -353,7 +455,7 @@ func (s *Server) hubStacks(ctx context.Context, node string) ([]stack, error) {
 			continue
 		}
 		if remote == "" {
-			remote, maps = s.nodeDocker(ctx, node), stackMounts(ctx)
+			remote, maps = s.nodeDocker(ctx, node), mountsFor(ctx, s.store)
 		}
 		if remote == "-" {
 			continue
@@ -511,24 +613,13 @@ func (s *Server) stackFile(w http.ResponseWriter, r *http.Request) {
 // composeCmd is `docker compose …` for this stack, run where its files make sense and
 // pointed at the Docker that runs its containers. files are given as compose will see them.
 func composeCmd(ctx context.Context, st stack, files []string, args ...string) *exec.Cmd {
-	tool := dockerTool()
+	return compose.Cmd(ctx, specOf(st, files), args...)
-	base := []string{"compose", "-p", st.Name}
+}
-	for _, f := range files {
+
-		base = append(base, "-f", f)
+// specOf is a stack as the compose runner needs it — and as a scheduled update stores it,
-	}
+// so the cron job and the console invoke compose identically.
-	if st.Via != "" {
+func specOf(st stack, files []string) compose.Spec {
-		pre := []string{"exec"}
+	return compose.Spec{Project: st.Name, Files: files, Via: st.Via, Remote: st.Remote, Tool: dockerTool()}
-		if st.Remote != "" {
-			pre = append(pre, "-e", "DOCKER_HOST="+st.Remote)
-		}
-		return exec.CommandContext(ctx, tool, append(append(append(pre, st.Via, "docker"), base...), args...)...)
-	}
-	cmd := exec.CommandContext(ctx, tool, append(base, args...)...)
-	cmd.Dir = filepath.Dir(files[0])
-	if st.Remote != "" {
-		cmd.Env = append(os.Environ(), "DOCKER_HOST="+st.Remote)
-	}
-	return cmd
 }
 
 // seenFiles is the stack's file list as compose runs it, with file i replaced by a candidate.
@@ -788,6 +879,103 @@ var (
 
 func stackUpdatesForget(key string) { updMu.Lock(); delete(updCache, key); updMu.Unlock() }
 
+// checkImages asks the registry whether each image is still the one we hold. remote is a
+// DOCKER_HOST for a machine whose images live elsewhere, "" for this one. Each image is one
+// registry request, which Docker Hub counts — so this is only ever called when asked for.
+func checkImages(ctx context.Context, images []string, remote string) []imageCheck {
+	local := func(args ...string) ([]byte, error) { // the Docker that holds the images
+		if remote != "" {
+			args = append([]string{"-H", remote}, args...)
+		}
+		return dockerRun(ctx, args...)
+	}
+	res := []imageCheck{}
+	for _, img := range images {
+		ic := imageCheck{Image: img, State: "unknown"}
+		out, err := local("image", "inspect", "-f", "{{json .RepoDigests}}", img)
+		var have []string
+		if err != nil || json.Unmarshal(bytes.TrimSpace(out), &have) != nil || len(have) == 0 {
+			ic.Detail = "built or loaded here, not pulled from a registry"
+			res = append(res, ic)
+			continue
+		}
+		rem, err := dockerRun(ctx, "buildx", "imagetools", "inspect", img, "--format", "{{.Manifest.Digest}}")
+		digest := strings.TrimSpace(string(rem))
+		if i := strings.LastIndex(digest, "\n"); i >= 0 {
+			digest = digest[i+1:]
+		}
+		if err != nil || !strings.HasPrefix(digest, "sha256:") {
+			ic.Detail = "the registry didn't answer"
+			if err != nil {
+				ic.Detail += ": " + err.Error()
+			}
+			res = append(res, ic)
+			continue
+		}
+		ic.State = "update"
+		for _, h := range have {
+			if strings.HasSuffix(h, "@"+digest) {
+				ic.State = "current"
+			}
+		}
+		res = append(res, ic)
+	}
+	return res
+}
+
+// machineUpdates checks every image on this machine in one pass, so the container list can
+// be coloured and the fleet can be asked "what is behind?" without walking stack by stack.
+func (s *Server) machineUpdates(w http.ResponseWriter, r *http.Request) {
+	node := r.PathValue("node")
+	ctx, cancel := context.WithTimeout(r.Context(), 10*time.Minute)
+	defer cancel()
+	stacks, err := s.findStacks(ctx)
+	if node != "" {
+		stacks, err = s.hubStacks(ctx, node)
+	}
+	if err != nil {
+		jsonErr(w, 500, err.Error())
+		return
+	}
+	// one entry per image, however many stacks use it
+	type want struct{ remote string }
+	imgs := map[string]want{}
+	byImage := map[string][]string{} // image -> stacks
+	for _, st := range stacks {
+		for _, img := range st.Images {
+			imgs[img] = want{remote: st.Remote}
+			byImage[img] = append(byImage[img], st.Name)
+		}
+	}
+	refresh := r.URL.Query().Get("refresh") == "1"
+	key := "machine:" + node
+	updMu.Lock()
+	c, ok := updCache[key]
+	updMu.Unlock()
+	if ok && time.Since(c.at) < 6*time.Hour && !refresh {
+		jsonOK(w, 200, map[string]any{"images": c.res, "checked": c.at.UnixMilli(), "stacks": byImage})
+		return
+	}
+	if !refresh {
+		jsonOK(w, 200, map[string]any{"images": []imageCheck{}, "checked": 0, "stacks": byImage})
+		return
+	}
+	var list []string
+	remote := ""
+	for img, wnt := range imgs {
+		list = append(list, img)
+		if wnt.remote != "" {
+			remote = wnt.remote
+		}
+	}
+	sort.Strings(list)
+	res := checkImages(ctx, list, remote)
+	updMu.Lock()
+	updCache[key] = updAnswer{time.Now(), res}
+	updMu.Unlock()
+	jsonOK(w, 200, map[string]any{"images": res, "checked": time.Now().UnixMilli(), "stacks": byImage})
+}
+
 // stackUpdates asks each image's registry for the digest its tag points at now and compares
 // it with the digest that was pulled. It is asked for, never polled: every answer is a
 // registry request, and Docker Hub counts those. Answers are kept for six hours.
@@ -816,43 +1004,7 @@ func (s *Server) stackUpdates(w http.ResponseWriter, r *http.Request) {
 		jsonOK(w, 200, map[string]any{"images": []imageCheck{}, "checked": 0})
 		return
 	}
-	local := func(args ...string) ([]byte, error) { // the Docker that holds the images
+	res := checkImages(ctx, st.Images, st.Remote)
-		if st.Remote != "" {
-			args = append([]string{"-H", st.Remote}, args...)
-		}
-		return dockerRun(ctx, args...)
-	}
-	res := []imageCheck{}
-	for _, img := range st.Images {
-		ic := imageCheck{Image: img, State: "unknown"}
-		out, err := local("image", "inspect", "-f", "{{json .RepoDigests}}", img)
-		var have []string
-		if err != nil || json.Unmarshal(bytes.TrimSpace(out), &have) != nil || len(have) == 0 {
-			ic.Detail = "built or loaded here, not pulled from a registry"
-			res = append(res, ic)
-			continue
-		}
-		remote, err := dockerRun(ctx, "buildx", "imagetools", "inspect", img, "--format", "{{.Manifest.Digest}}")
-		digest := strings.TrimSpace(string(remote))
-		if i := strings.LastIndex(digest, "\n"); i >= 0 {
-			digest = digest[i+1:]
-		}
-		if err != nil || !strings.HasPrefix(digest, "sha256:") {
-			ic.Detail = "the registry didn't answer"
-			if err != nil {
-				ic.Detail += ": " + err.Error()
-			}
-			res = append(res, ic)
-			continue
-		}
-		ic.State = "update"
-		for _, h := range have {
-			if strings.HasSuffix(h, "@"+digest) {
-				ic.State = "current"
-			}
-		}
-		res = append(res, ic)
-	}
 	updMu.Lock()
 	updCache[key] = updAnswer{time.Now(), res}
 	updMu.Unlock()

internal/api/stacks_test.go

@@ -0,0 +1,121 @@
+package api
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"hamlab/internal/compose"
+	"hamlab/internal/store"
+)
+
+// A stack deployed by an orchestrator running in a container records the path its compose
+// file had inside that container. Resolving it has to work both while the orchestrator is
+// running — compose is then run inside it — and after it has been removed, where the same
+// mount is only remembered and compose has to run on the host instead.
+func TestResolveThroughMounts(t *testing.T) {
+	root := t.TempDir()
+	dir := filepath.Join(root, "orch", "stacks", "app")
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	file := filepath.Join(dir, "compose.yaml")
+	if err := os.WriteFile(file, []byte("services: {}\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	inner := "/app/data/stacks/app/compose.yaml"
+
+	for _, tc := range []struct {
+		name, container, wantVia string
+	}{
+		{"orchestrator running", "orch", "orch"},
+		{"orchestrator removed", "", ""},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			st := stack{Name: "app", File: inner, Files: []string{inner}}
+			st.resolve([]mountMap{{tc.container, "/app/data", filepath.Join(root, "orch")}})
+			if !st.Editable {
+				t.Fatalf("not editable: %s", st.Note)
+			}
+			if st.HostFile != file {
+				t.Errorf("host file = %q, want %q", st.HostFile, file)
+			}
+			if st.Via != tc.wantVia {
+				t.Errorf("via = %q, want %q", st.Via, tc.wantVia)
+			}
+		})
+	}
+
+	// Nothing explains the path: the stack is listed, but read-only and says why.
+	st := stack{Name: "app", File: inner, Files: []string{inner}}
+	st.resolve(nil)
+	if st.Editable || st.Note == "" {
+		t.Errorf("unresolvable stack should be read-only with a note, got editable=%v note=%q", st.Editable, st.Note)
+	}
+}
+
+// Every file of a multi-file stack must resolve through the same place. Redeploying from
+// the first file alone would silently drop whatever the override set.
+func TestResolveRefusesSplitStack(t *testing.T) {
+	root := t.TempDir()
+	dir := filepath.Join(root, "orch", "stacks", "app")
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(filepath.Join(dir, "compose.yaml"), []byte("services: {}\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	st := stack{Name: "app", File: "/app/data/stacks/app/compose.yaml", Files: []string{
+		"/app/data/stacks/app/compose.yaml",
+		"/app/data/stacks/app/override.yaml", // never written: not on disk
+	}}
+	st.resolve([]mountMap{{"orch", "/app/data", filepath.Join(root, "orch")}})
+	if st.Editable {
+		t.Error("a stack with a file that cannot be found should not be editable")
+	}
+}
+
+// A schedule made while an orchestrator was running says "run compose inside that
+// container". When the container is gone the schedule must still work, against the same
+// files on the host — otherwise retiring the orchestrator silently breaks every scheduled
+// update made before it went.
+func TestRepointSurvivesAMissingContainer(t *testing.T) {
+	root := t.TempDir()
+	dir := filepath.Join(root, "orch", "stacks", "app")
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	host := filepath.Join(dir, "compose.yaml")
+	if err := os.WriteFile(host, []byte("services: {}\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	db, err := store.Open(filepath.Join(root, "state.json"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := db.Update(func(st *store.State) {
+		st.Mounts = []store.RememberedMount{{Container: "gone-orchestrator", Dest: "/app/data", Src: filepath.Join(root, "orch")}}
+	}); err != nil {
+		t.Fatal(err)
+	}
+
+	in := compose.Spec{Project: "app", Files: []string{"/app/data/stacks/app/compose.yaml"}, Via: "gone-orchestrator", Tool: "docker"}
+	got := repoint(context.Background(), db, in)
+	if got.Via != "" {
+		t.Errorf("via = %q, want it dropped", got.Via)
+	}
+	if len(got.Files) != 1 || got.Files[0] != host {
+		t.Errorf("files = %v, want [%s]", got.Files, host)
+	}
+
+	// Nothing remembered: the schedule is left exactly as it was rather than pointed at a
+	// guess. It will fail loudly, which is better than deploying the wrong file.
+	empty, err := store.Open(filepath.Join(root, "empty.json"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got := repoint(context.Background(), empty, in); got.Via != in.Via || len(got.Files) != 1 || got.Files[0] != in.Files[0] {
+		t.Errorf("unresolvable spec was changed: %+v", got)
+	}
+}

internal/compose/compose.go

@@ -0,0 +1,50 @@
+// Package compose runs docker compose the way a stack was deployed — which is not always
+// simply "on this host". A stack deployed by an orchestrator running inside its own
+// container records paths in that container's filesystem, so compose has to run in there;
+// a stack deployed to another machine over tcp needs DOCKER_HOST pointed at it. Both the
+// console and the scheduled updates it writes must invoke it identically, or a redeploy
+// from one would not match the other's labels.
+package compose
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"path/filepath"
+)
+
+// Spec is everything needed to run compose for one stack.
+type Spec struct {
+	Project string   // -p
+	Files   []string // -f, in order
+	Via     string   // a container to run compose inside; "" is this host
+	Remote  string   // a DOCKER_HOST; "" is this machine's Docker
+	Tool    string   // "docker" or "podman"
+}
+
+// Cmd builds the command. args are compose's own, e.g. "pull" or "up", "-d".
+func Cmd(ctx context.Context, s Spec, args ...string) *exec.Cmd {
+	tool := s.Tool
+	if tool == "" {
+		tool = "docker"
+	}
+	base := []string{"compose", "-p", s.Project}
+	for _, f := range s.Files {
+		base = append(base, "-f", f)
+	}
+	if s.Via != "" {
+		pre := []string{"exec"}
+		if s.Remote != "" {
+			pre = append(pre, "-e", "DOCKER_HOST="+s.Remote)
+		}
+		return exec.CommandContext(ctx, tool, append(append(append(pre, s.Via, "docker"), base...), args...)...)
+	}
+	cmd := exec.CommandContext(ctx, tool, append(base, args...)...)
+	if len(s.Files) > 0 {
+		cmd.Dir = filepath.Dir(s.Files[0])
+	}
+	if s.Remote != "" {
+		cmd.Env = append(os.Environ(), "DOCKER_HOST="+s.Remote)
+	}
+	return cmd
+}

internal/storage/snapshots.go

@@ -452,8 +452,25 @@ func (m *Manager) WriteCronFile(ctx context.Context, policies []Policy, user, ex
 	return m.writeRoot(ctx, CronFile, text)
 }
 
-// writeRoot puts text at a root-owned path, through the same privilege escalation every
+// WriteRootFile puts text at a root-owned path, through the same privilege escalation every
-// other write here uses.
+// other write here uses. Shared by the snapshot schedules and the image-update schedules,
+// which both own a file in /etc/cron.d.
+func (m *Manager) WriteRootFile(ctx context.Context, path, text string) error {
+	return m.writeRoot(ctx, path, text)
+}
+
+// RemoveRootFile deletes one, for when the last schedule of a kind goes away.
+func (m *Manager) RemoveRootFile(ctx context.Context, path string) error {
+	out, code, err := m.run(ctx, "rm", "-f", path)
+	if err != nil {
+		return err
+	}
+	if code != 0 {
+		return errors.New(strings.TrimSpace(string(out)))
+	}
+	return nil
+}
+
 func (m *Manager) writeRoot(ctx context.Context, path, text string) error {
 	args := []string{"tee", path}
 	if m.sudo {

internal/store/store.go

@@ -44,6 +44,16 @@ type UserState struct {
 	UI json.RawMessage `json:"ui,omitempty"` // theme, accents; opaque to the server
 }
 
+// RememberedMount is one bind mount of a container that was running when Ham Lab last
+// looked. A compose stack records the path its file had *inside* whatever deployed it, so
+// when that container is gone the path is the only way back to the file on disk.
+type RememberedMount struct {
+	Container string    `json:"container"`
+	Dest      string    `json:"dest"` // the path inside the container
+	Src       string    `json:"src"`  // the path on the machine
+	Seen      time.Time `json:"seen"`
+}
+
 type State struct {
 	Terminals []Terminal            `json:"terminals"`
 	Sessions  map[string]Session    `json:"sessions"` // token -> session
@@ -51,8 +61,13 @@ type State struct {
 	// UI is the pre-users machine-wide look; on first boot with users it moves to the first
 	// account. Kept so an old state file still reads.
 	UI         json.RawMessage `json:"ui,omitempty"`
-	Alerts     json.RawMessage `json:"alerts,omitempty"`
+	Alerts     json.RawMessage `json:"alerts,omitempty"`      // rules, channels and recent events (internal/api/alerts.go)
-	ZFS        json.RawMessage `json:"zfs,omitempty"`  // snapshot schedules (internal/api/zfs.go)      // rules, channels and recent events (internal/api/alerts.go)
+	ZFS        json.RawMessage `json:"zfs,omitempty"`         // snapshot schedules (internal/api/zfs.go)
+	DockerJobs json.RawMessage `json:"docker_jobs,omitempty"` // image-update schedules (internal/api/dockersched.go)
+	// Mounts remembers where a container's bind mounts pointed, so a stack deployed by an
+	// orchestrator that has since been removed can still be found on disk. See
+	// internal/api/stacks.go.
+	Mounts []RememberedMount `json:"mounts,omitempty"`
 	SamplerMs  int             `json:"sampler_ms,omitempty"`  // how often this machine samples; the hub keeps every machine on the console's refresh rate
 	HistorySec int             `json:"history_sec,omitempty"` // how far back the graphs reach; the hub keeps every machine on the same one
 }

internal/term/manager.go

@@ -84,10 +84,21 @@ func newID() string {
 
 // Create starts a new login shell. id may be empty.
 func (m *Manager) Create(id, name string) (*Session, error) {
+	return m.CreateCmd(id, name, nil)
+}
+
+// CreateCmd is Create with something other than a login shell: argv nil is the shell, and
+// anything else runs that instead — `docker exec -it …` for a container's console. Every
+// session is the same thing afterwards, so attach, resize, the scrollback ring and the tab
+// strip work without knowing which it is.
+func (m *Manager) CreateCmd(id, name string, argv []string) (*Session, error) {
 	if id == "" {
 		id = newID()
 	}
 	cmd := exec.Command(m.shell, "-l")
+	if len(argv) > 0 {
+		cmd = exec.Command(argv[0], argv[1:]...)
+	}
 	cmd.Dir = m.home
 	cmd.Env = append(os.Environ(), "TERM=xterm-256color", "COLORTERM=truecolor", "HAMLAB=1")
 	ptmx, err := pty.StartWithSize(cmd, &pty.Winsize{Cols: defaultCol, Rows: defaultRow})

web/src/Docker.tsx

@@ -1,16 +1,18 @@
 // Docker: containers grouped by the stack that deployed them, with start / stop / restart,
 // logs that follow, and inspect — then images, volumes and networks. Live CPU and memory
 // come from the same container sensors everything else uses.
-import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
+import { FormEvent, useCallback, useEffect, useMemo, useRef, useState } from 'react'
-import { api, DockerContainer, DockerInfo, ImageCheck, Stack, StackDiff } from './api'
+import { api, apiPath, DockerContainer, DockerEvent, DockerHealth, DockerInfo, ImageCheck, MachineStatus, MachineUpdates, Stack, StackDiff, UpdateJob, UpdateJobRow } from './api'
 import TaskOutput from './TaskOutput'
 import StackNew, { DupFrom } from './StackNew'
-import { Sensor, fmtValue, useSensorStream } from './sensors'
+import { Sensor, fetchHistories, fmtValue, useSensorStream } from './sensors'
 import { useHashSel } from './hashsel'
-import { useRefreshMs } from './prefs'
+import { fmtWindowSec, useHistorySec, useRefreshMs } from './prefs'
-import { askText, askYes, Btn, CloseBtn, Loading, setUnsaved, toast } from './ui'
+import Sparkline from './Sparkline'
+import { fmtBytes } from './fmt'
+import { askText, askYes, Btn, CloseBtn, Loading, setUnsaved, toast, useEscape } from './ui'
 
-type Tab = 'containers' | 'stacks' | 'images' | 'volumes' | 'networks'
+type Tab = 'containers' | 'stacks' | 'updates' | 'images' | 'volumes' | 'networks'
 const dot = (c: DockerContainer) => (c.state !== 'running' ? (c.state === 'exited' || c.state === 'dead' ? 'bad' : 'warn') : c.health === 'unhealthy' ? 'bad' : c.health === 'starting' ? 'warn' : 'ok')
 const ports = (p: string) => [...new Set(p.split(', ').map((x) => x.replace(/^\[?::\]?:/, '').replace(/^0\.0\.0\.0:/, '').replace(/^[\d.]+:/, '')).filter((x) => x.includes('->')))].join('  ')
 
@@ -50,18 +52,26 @@ function LogView({ id }: { id: string }) {
   )
 }
 
+// What a container is doing, and what can be done to it: its numbers over time, its health
+// checks, what has happened to it lately, a shell inside it, and its limits.
 function Detail({ c, live, onBack, onChanged }: { c: DockerContainer; live: Record<string, Sensor>; onBack: () => void; onChanged: () => void }) {
-  const [tab, setTab] = useState<'logs' | 'inspect'>('logs')
+  const [tab, setTab] = useState<'logs' | 'stats' | 'health' | 'events' | 'inspect'>('logs')
   const [info, setInfo] = useState<Record<string, unknown> | null>(null)
-  useEffect(() => { setInfo(null); if (tab === 'inspect') api.dockerInspect(c.name).then(setInfo).catch((e) => toast(e instanceof Error ? e.message : 'failed', 'err')) }, [tab, c.name])
+  const [limits, setLimits] = useState(false)
+  useEffect(() => { setInfo(null); if (tab === 'inspect' || limits) api.dockerInspect(c.name).then(setInfo).catch((e) => toast(e instanceof Error ? e.message : 'failed', 'err')) }, [tab, limits, c.name])
   const act = (a: 'start' | 'stop' | 'restart' | 'remove') => async () => { try { await api.dockerAction(c.name, a); toast(`${c.name}: ${a} ok`); onChanged() } catch (e) { toast(`${c.name}: ${e instanceof Error ? e.message : 'failed'}`, 'err') } }
   const running = c.state === 'running'
-  const s = (k: string) => live[`container.${c.name.toLowerCase().replace(/[^a-z0-9-]+/g, '_').replace(/^_|_$/g, '')}.${k}`]
+  const seg = c.name.toLowerCase().replace(/[^a-z0-9-]+/g, '_').replace(/^_|_$/g, '')
+  const s = (k: string) => live[`container.${seg}.${k}`]
   const mounts = (info?.mounts as { Source: string; Destination: string; RW: boolean }[] | undefined) ?? []
+  const shell = async () => {
+    try { const t = await api.dockerExec(c.name); toast(`shell open as “${t.name}” on the Terminal tab`); window.dispatchEvent(new CustomEvent('hamlab:terminal', { detail: t.id })) }
+    catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
+  }
   return (
     <div className="svc-detail">
       <CloseBtn onClick={onBack} />
-      <div className="svc-head"><button className="btn back" onClick={onBack}>‹ Back</button><span className={'sdot ' + dot(c)} /><span className="dname">{c.name}</span></div>
+      <div className="svc-head"><button className="btn back" onClick={onBack}>‹ Back</button><span className={'sdot ' + dot(c)} /><span className="dname">{c.name}</span>{c.health && c.health !== 'none' && <span className={'badge' + (c.health === 'healthy' ? '' : ' bad')}>{c.health}</span>}</div>
       <div className="props">
         <div><span className="k">Image</span><span className="mono small">{c.image}</span></div>
         <div><span className="k">Status</span><span>{c.status}</span></div>
@@ -72,11 +82,22 @@ function Detail({ c, live, onBack, onChanged }: { c: DockerContainer; live: Reco
       <div className="svc-actions">
         {!running && <Btn className="btn on" onClick={act('start')}>Start</Btn>}
         {!running && <Btn className="btn danger" confirm={`Remove the container ${c.name}? Its image, volumes and bind-mounted data stay; anything written inside the container itself is lost.${c.stack ? ` Deploying the ${c.stack} stack makes it again.` : ''}`} onClick={async () => { await act('remove')(); onBack() }}>Remove</Btn>}
+        {running && <Btn className="btn" onClick={shell}>Shell</Btn>}
         {running && <Btn className="btn" confirm={`Restart ${c.name}?`} onClick={act('restart')}>Restart</Btn>}
+        <Btn className="btn" onClick={() => setLimits(true)}>Limits</Btn>
         {running && <Btn className="btn danger" confirm={`Stop ${c.name}? It stays stopped until something starts it.`} onClick={act('stop')}>Stop</Btn>}
       </div>
-      <div className="tabs"><button className={tab === 'logs' ? 'on' : ''} onClick={() => setTab('logs')}>Logs</button><button className={tab === 'inspect' ? 'on' : ''} onClick={() => setTab('inspect')}>Inspect</button></div>
+      <div className="tabs">
+        <button className={tab === 'logs' ? 'on' : ''} onClick={() => setTab('logs')}>Logs</button>
+        <button className={tab === 'stats' ? 'on' : ''} onClick={() => setTab('stats')}>Stats</button>
+        <button className={tab === 'health' ? 'on' : ''} onClick={() => setTab('health')}>Health</button>
+        <button className={tab === 'events' ? 'on' : ''} onClick={() => setTab('events')}>Events</button>
+        <button className={tab === 'inspect' ? 'on' : ''} onClick={() => setTab('inspect')}>Inspect</button>
+      </div>
       {tab === 'logs' && <LogView id={c.name} />}
+      {tab === 'stats' && <Stats seg={seg} live={live} />}
+      {tab === 'health' && <Health id={c.name} />}
+      {tab === 'events' && <Events id={c.name} />}
       {tab === 'inspect' && (!info ? <Loading label="Inspecting…" /> : (
         <div className="props">
           <div><span className="k">Restarts</span><span>{String(info.restart || 'no')} policy · restarted {String(info.restarts ?? 0)}×</span></div>
@@ -88,11 +109,154 @@ function Detail({ c, live, onBack, onChanged }: { c: DockerContainer; live: Reco
           <div><span className="k">Env</span><span className="mono small muted">{((info.env_names as string[]) ?? []).join('  ')} <span className="muted">(names only)</span></span></div>
         </div>
       ))}
+      {limits && <LimitsDialog c={c} info={info} onClose={() => setLimits(false)} onSaved={() => { setLimits(false); onChanged() }} />}
+    </div>
+  )
+}
+
+// CPU and memory over time, from the container sensors the sampler already keeps.
+function Stats({ seg, live }: { seg: string; live: Record<string, Sensor> }) {
+  const [hist, setHist] = useState<{ id: string; t: number[]; v: number[] }[] | null>(null)
+  const historySec = useHistorySec()
+  useEffect(() => {
+    let alive = true
+    fetchHistories(`container.${seg}.cpu,container.${seg}.mem`)
+      .then((h) => { if (alive) setHist(h) })
+      .catch(() => { if (alive) setHist([]) })
+    return () => { alive = false }
+  }, [seg])
+  const css = (n: string) => getComputedStyle(document.documentElement).getPropertyValue(n).trim() || '#3987e5'
+  const of = (k: string) => hist?.find((h) => h.id === `container.${seg}.${k}`)?.v ?? []
+  const cpu = of('cpu'), mem = of('mem')
+  if (hist === null) return <Loading label="Reading the history…" />
+  if (cpu.length < 2 && mem.length < 2) return <div className="notice">No history yet for this container — it is sampled while it runs, so give it a moment.</div>
+  return (
+    <div className="cards">
+      <section className="card">
+        <div className="sub-head"><span className="dname">CPU</span><span className="spacer" /><span className="num">{fmtValue(live[`container.${seg}.cpu`]?.value, '%')} of a core</span></div>
+        <Sparkline series={[{ name: 'cpu', values: cpu, color: css('--s1') }]} format={(v) => `${v.toFixed(1)}%`} height={64} />
+        <div className="muted small">the last {fmtWindowSec(historySec)}</div>
+      </section>
+      <section className="card">
+        <div className="sub-head"><span className="dname">Memory</span><span className="spacer" /><span className="num">{fmtValue(live[`container.${seg}.mem`]?.value, 'B')}</span></div>
+        <Sparkline series={[{ name: 'mem', values: mem, color: css('--s2') }]} format={(v) => fmtBytes(v)} height={64} />
+        <div className="muted small">the last {fmtWindowSec(historySec)}</div>
+      </section>
+    </div>
+  )
+}
+
+// The health check's verdict, and what its last few probes actually said.
+function Health({ id }: { id: string }) {
+  const [h, setH] = useState<DockerHealth | null>(null)
+  const [err, setErr] = useState('')
+  useEffect(() => { setH(null); api.dockerHealth(id).then(setH).catch((e) => setErr(e instanceof Error ? e.message : 'failed')) }, [id])
+  if (err) return <div className="error">{err}</div>
+  if (!h) return <Loading label="Asking…" />
+  if (!h.configured) return <div className="notice">This image defines no health check, so Docker only knows whether the process is running — which it is ({h.state}). A health check is a <span className="mono">healthcheck:</span> block in the compose file.</div>
+  return (
+    <>
+      <div className="props">
+        <div><span className="k">Health</span><span className={h.status === 'healthy' ? '' : 'warn-text'}>{h.status}</span></div>
+        <div><span className="k">State</span><span>{h.state}</span></div>
+        {!!h.failing_streak && <div><span className="k">Failing</span><span className="warn-text">{h.failing_streak} probe{h.failing_streak === 1 ? '' : 's'} in a row</span></div>}
+      </div>
+      <div className="muted small" style={{ marginTop: 8 }}>The last {h.log?.length ?? 0} probes — Docker keeps five.</div>
+      {(h.log ?? []).slice().reverse().map((p, i) => (
+        <section className="card" key={i} style={{ marginTop: 6 }}>
+          <div className="sub-head">
+            <span className={'sdot ' + (p.ExitCode === 0 ? 'ok' : 'bad')} />
+            <span>{new Date(p.Start).toLocaleString()}</span>
+            <span className="spacer" />
+            <span className="muted small">exit {p.ExitCode} · {Math.max(0, new Date(p.End).getTime() - new Date(p.Start).getTime())} ms</span>
+          </div>
+          {p.Output.trim() && <pre className="run-out" style={{ maxHeight: 140 }}>{p.Output.trim()}</pre>}
+        </section>
+      ))}
+    </>
+  )
+}
+
+// What has happened to this container lately, asked of Docker rather than kept here.
+function Events({ id }: { id: string }) {
+  const [ev, setEv] = useState<DockerEvent[] | null>(null)
+  const [kept, setKept] = useState(0)
+  const [since, setSince] = useState('7d')
+  const [err, setErr] = useState('')
+  useEffect(() => { setEv(null); setErr(''); api.dockerEvents(id, since).then((r) => { setEv(r.events); setKept(r.kept) }).catch((e) => setErr(e instanceof Error ? e.message : 'failed')) }, [id, since])
+  const bad = (a: string, h?: string) => /die|kill|oom|destroy/.test(a) || (h !== undefined && h !== 'healthy')
+  return (
+    <>
+      <div className="chips" style={{ marginBottom: 8 }}>
+        {['24h', '7d', '30d'].map((x) => <button key={x} className={'chip' + (since === x ? ' on' : '')} onClick={() => setSince(x)}>{x}</button>)}
+      </div>
+      {err && <div className="error">{err}</div>}
+      {!ev && !err && <Loading label="Asking Docker…" />}
+      {ev && ev.length === 0 && <div className="notice">Nothing for this container in the last {since}. Ham Lab keeps its own record — Docker's own only reaches back a few minutes — so history starts from when this version was installed. {kept} events kept across every container.</div>}
+      {ev && ev.length > 0 && (
+        <div className="snap-list">
+          {ev.map((e, i) => (
+            <div className="snap" key={i}>
+              <span className={bad(e.action, e.health) ? 'warn-text' : ''}>{e.action}{e.health ? `: ${e.health}` : ''}</span>
+              <span className="muted">{new Date(e.t).toLocaleString()}</span>
+              <span className="muted small">{e.exit_code ? `exit ${e.exit_code}` : ''}{e.signal ? ` signal ${e.signal}` : ''}</span>
+              <span />
+              <span />
+            </div>
+          ))}
+        </div>
+      )}
+    </>
+  )
+}
+
+// Memory, CPU and restart policy, changed on the running container. Docker does this live,
+// which is the point — and also the catch, said plainly below.
+function LimitsDialog({ c, info, onClose, onSaved }: { c: DockerContainer; info: Record<string, unknown> | null; onClose: () => void; onSaved: () => void }) {
+  const curMem = Number(info?.memory_limit ?? 0)
+  const [mem, setMem] = useState(curMem > 0 ? String(Math.round(curMem / (1 << 20))) + 'm' : '')
+  const [cpus, setCpus] = useState('')
+  const [restart, setRestart] = useState(String(info?.restart ?? ''))
+  const [err, setErr] = useState('')
+  const [busy, setBusy] = useState(false)
+  useEscape(onClose)
+  const save = async (e: FormEvent) => {
+    e.preventDefault(); setErr(''); setBusy(true)
+    try { await api.dockerLimits(c.name, { memory: mem, cpus, restart }); toast(`${c.name} updated`); onSaved() }
+    catch (e) { setErr(e instanceof Error ? e.message : 'failed'); setBusy(false) }
+  }
+  return (
+    <div className="modal-back" onClick={onClose}>
+      <form className="modal" onClick={(e) => e.stopPropagation()} onSubmit={save}>
+        <h3>Limits for {c.name}</h3>
+        <label className="muted">Memory — 512m, 2g, or 0 for no limit{curMem > 0 ? ` (now ${fmtBytes(curMem)})` : ' (no limit now)'}
+          <input className="mono" value={mem} onChange={(e) => setMem(e.target.value)} placeholder="leave blank to keep" spellCheck={false} />
+        </label>
+        <label className="muted">CPUs — 1.5 means one and a half cores, 0 for no limit
+          <input className="mono" value={cpus} onChange={(e) => setCpus(e.target.value)} placeholder="leave blank to keep" spellCheck={false} />
+        </label>
+        <label className="muted">Restart policy
+          <select value={restart} onChange={(e) => setRestart(e.target.value)}>
+            <option value="">leave as it is</option>
+            {['no', 'on-failure', 'unless-stopped', 'always'].map((p) => <option key={p} value={p}>{p}</option>)}
+          </select>
+        </label>
+        <div className="notice attn">
+          This changes the running container now, without recreating it. It does <b>not</b> edit the
+          compose file{c.stack ? <> — so deploying <b>{c.stack}</b> puts back whatever the file says</> : ''}.
+          Edit the stack to make it stick.
+        </div>
+        {err && <div className="error">{err}</div>}
+        <div className="svc-actions">
+          <button type="submit" className="btn on" disabled={busy || (!mem && !cpus && !restart)}>Apply</button>
+          <span className="spacer" />
+          <button type="button" className="btn" onClick={onClose}>Cancel</button>
+        </div>
+      </form>
     </div>
   )
 }
 
-// A plain line diff (longest common subsequence): compose files are short, so the simple way is fine.
 function lineDiff(a: string, b: string): { s: ' ' | '+' | '-'; t: string }[] {
   const x = a.split('\n'), y = b.split('\n'); const n = x.length, m = y.length
   const L = Array.from({ length: n + 1 }, () => new Uint16Array(m + 1))
@@ -120,6 +284,7 @@ const maskEnv = (text: string) => text.split('\n').map((l) => { const i = l.inde
 
 // One stack: its compose file to edit, and the four things to do with it. Saving validates
 // with compose itself and keeps the previous file beside it; deploying is a separate, asked-for act.
+
 function StackDetail({ name, startTask, onBack, onChanged, onDuplicate }: { name: string; startTask?: string; onBack: () => void; onChanged: () => void; onDuplicate: (from: DupFrom) => void }) {
   const [st, setSt] = useState<Stack | null>(null)
   const [text, setText] = useState(''); const [orig, setOrig] = useState('')
@@ -203,6 +368,12 @@ export default function Docker() {
   // what a new stack could join, and what it could collide with
   const nets = useMemo(() => (d?.networks ?? []).map((n) => String(n.Name)).filter((n) => !['bridge', 'host', 'none'].includes(n)).sort(), [d])
   const taken = useMemo(() => { const ports: Record<string, string> = {}; for (const c of d?.containers ?? []) for (const m of (c.ports ?? '').matchAll(/:(\d+)->/g)) ports[m[1]] = c.name; return { names: (d?.containers ?? []).map((c) => c.name), ports } }, [d])
+  // What the registry last said about each image, so a container can be marked without
+  // asking again. Only ever read from the cache here — checking is a deliberate act.
+  const [upd, setUpd] = useState<MachineUpdates | null>(null)
+  useEffect(() => { api.machineUpdates().then(setUpd).catch(() => setUpd(null)) }, [])
+  const imgState = useMemo(() => Object.fromEntries((upd?.images ?? []).map((i) => [i.image, i.state])), [upd])
+  const behind = (upd?.images ?? []).filter((i) => i.state === 'update').length
   if (!d) return <div className="dash">{err ? <div className="notice">{err === 'not found' ? "This machine's Ham Lab is older than the Docker tab. Update it from Machines." : err}</div> : <Loading label="Asking Docker…" />}</div>
   if (!d.available) return <div className="dash"><div className="notice">{d.message}</div></div>
   const selStack = sel.startsWith('stack:') ? sel.slice(6) : ''
@@ -210,12 +381,12 @@ export default function Docker() {
   const shownTab: Tab = selStack ? 'stacks' : tab
   const up = d.containers.filter((c) => c.state === 'running').length
   const sid = (c: DockerContainer, k: string) => sensors[`container.${c.name.toLowerCase().replace(/[^a-z0-9-]+/g, '_').replace(/^_|_$/g, '')}.${k}`]
-  const cols: Record<Exclude<Tab, 'containers' | 'stacks'>, [string, string][]> = { images: [['Repository', 'Repository'], ['Tag', 'Tag'], ['Size', 'Size'], ['CreatedSince', 'Created']], volumes: [['Name', 'Name'], ['Driver', 'Driver'], ['Mountpoint', 'Mountpoint']], networks: [['Name', 'Name'], ['Driver', 'Driver'], ['Scope', 'Scope']] }
+  const cols: Record<Exclude<Tab, 'containers' | 'stacks' | 'updates'>, [string, string][]> = { images: [['Repository', 'Repository'], ['Tag', 'Tag'], ['Size', 'Size'], ['CreatedSince', 'Created']], volumes: [['Name', 'Name'], ['Driver', 'Driver'], ['Mountpoint', 'Mountpoint']], networks: [['Name', 'Name'], ['Driver', 'Driver'], ['Scope', 'Scope']] }
   return (
     <div className={'svc docker' + (((cur || sel) && shownTab === 'containers') || selStack ? ' has-sel' : '')}>
       <div className="svc-list">
         <div className="svc-tools">
-          <div className="chips">{(['containers', 'stacks', 'images', 'volumes', 'networks'] as Tab[]).map((t) => <button key={t} className={'chip' + (shownTab === t ? ' on' : '')} onClick={() => { setTab(t); if (sel) setSel('') }}>{t} · {t === 'containers' ? `${up}/${d.containers.length}` : t === 'stacks' ? (stacks?.length ?? '…') : d[t].length}</button>)}</div>
+          <div className="chips">{(['containers', 'stacks', 'updates', 'images', 'volumes', 'networks'] as Tab[]).map((t) => <button key={t} className={'chip' + (shownTab === t ? ' on' : '')} onClick={() => { setTab(t); if (sel) setSel('') }}>{t}{t === 'containers' ? ` · ${up}/${d.containers.length}` : t === 'stacks' ? ` · ${stacks?.length ?? '…'}` : t === 'updates' ? (behind > 0 ? ` · ${behind}` : '') : ` · ${d[t].length}`}</button>)}</div>
           {(shownTab === 'containers' || shownTab === 'stacks') && <div className="svc-actions"><button className="btn" onClick={() => setMaking(shownTab === 'stacks' ? 'stack' : 'container')}>{shownTab === 'stacks' ? '+ New stack' : '+ New container'}</button></div>}
           {shownTab === 'images' && <form className="chan-add" onSubmit={async (e) => { e.preventDefault(); try { setPullTask((await api.pullImage(pull.trim())).id) } catch (er) { toast(er instanceof Error ? er.message : 'failed', 'err') } }}>
             <input className="mono" value={pull} onChange={(e) => setPull(e.target.value)} placeholder="pull an image — nginx:alpine, ghcr.io/owner/name:tag" spellCheck={false} autoCapitalize="off" autoCorrect="off" />
@@ -242,7 +413,7 @@ export default function Docker() {
               {list.map((c) => (
                 <div key={c.id} className={'svc-row' + (c.name === sel ? ' active' : '')} onClick={() => setSel(c.name)}>
                   <span className={'sdot ' + dot(c)} />
-                  <div className="svc-name"><div className="n">{c.name}</div><div className="d">{c.image}</div></div>
+                  <div className="svc-name"><div className="n">{c.name}<UpdateDot state={imgState[c.image]} /></div><div className="d">{c.image}</div></div>
                   <div className="svc-state">{c.state === 'running' ? <>{fmtValue(sid(c, 'mem')?.value ?? c.mem, 'B')}<br />{fmtValue(sid(c, 'cpu')?.value, '%')}</> : c.state}</div>
                 </div>
               ))}
@@ -260,16 +431,212 @@ export default function Docker() {
                 return <tr key={name}><td className="mono small" title={String(row.Mountpoint ?? '')}>{anon ? <>{name.slice(0, 12)}… <span className="muted">anonymous</span></> : name}</td><td className="small">{users.length ? users.join(', ') : <span className="muted">nothing</span>}</td><td className="row-end">{users.length ? <span className="badge">in use</span> : <button className="icon-btn" title="delete this volume and its data" onClick={remove}>✕</button>}</td></tr> })}
                 {d.volumes.length === 0 && <tr><td colSpan={3} className="muted small">No volumes.</td></tr>}</tbody></table>
           )}
-          {shownTab !== 'containers' && shownTab !== 'stacks' && shownTab !== 'volumes' && (
+          {shownTab !== 'containers' && shownTab !== 'stacks' && shownTab !== 'volumes' && shownTab !== 'updates' && (
-            <table className="dtable"><thead><tr>{cols[shownTab as Exclude<Tab, 'containers' | 'stacks'>].map(([, h]) => <th key={h}>{h}</th>)}{shownTab === 'images' && <th />}</tr></thead>
+            <table className="dtable"><thead><tr>{cols[shownTab as Exclude<Tab, 'containers' | 'stacks' | 'volumes' | 'updates'>].map(([, h]) => <th key={h}>{h}</th>)}{shownTab === 'images' && <th />}</tr></thead>
-              <tbody>{d[shownTab as Exclude<Tab, 'containers' | 'stacks'>].map((row, i) => <tr key={i}>{cols[shownTab as Exclude<Tab, 'containers' | 'stacks'>].map(([k]) => <td key={k} className="mono small">{String(row[k] ?? '')}</td>)}{shownTab === 'images' && <td className="row-end">{row.in_use ? <span className="badge" title="a container uses this image">in use</span> : <Btn className="icon-btn" title="remove this image" confirm={`Remove the image ${row.Repository}:${row.Tag}? Docker refuses if a container still uses it.`} onClick={async () => { const ref = row.Tag && row.Tag !== '<none>' ? `${row.Repository}:${row.Tag}` : String(row.ID); try { await api.removeImage(ref); toast(`${ref} removed`); load() } catch (er) { toast(er instanceof Error ? er.message : 'failed', 'err') } }}>✕</Btn>}</td>}</tr>)}</tbody></table>
+              <tbody>{d[shownTab as Exclude<Tab, 'containers' | 'stacks' | 'volumes' | 'updates'>].map((row, i) => <tr key={i}>{cols[shownTab as Exclude<Tab, 'containers' | 'stacks' | 'volumes' | 'updates'>].map(([k]) => <td key={k} className="mono small">{String(row[k] ?? '')}</td>)}{shownTab === 'images' && <td className="row-end">{row.in_use ? <span className="badge" title="a container uses this image">in use</span> : <Btn className="icon-btn" title="remove this image" confirm={`Remove the image ${row.Repository}:${row.Tag}? Docker refuses if a container still uses it.`} onClick={async () => { const ref = row.Tag && row.Tag !== '<none>' ? `${row.Repository}:${row.Tag}` : String(row.ID); try { await api.removeImage(ref); toast(`${ref} removed`); load() } catch (er) { toast(er instanceof Error ? er.message : 'failed', 'err') } }}>✕</Btn>}</td>}</tr>)}</tbody></table>
           )}
         </div>
       </div>
       {shownTab === 'containers' && !cur && sel && !selStack && <div className="svc-detail"><CloseBtn onClick={() => setSel('')} /><button className="btn back" onClick={() => setSel('')}>‹ Back</button><div className="notice">There is no container called <b className="mono">{sel}</b> on this machine. It may have been removed, renamed or recreated under another name. <button className="link" onClick={() => setSel('')}>Show them all</button></div></div>}
       {shownTab === 'containers' && (cur ? <Detail c={cur} live={sensors} onBack={() => setSel('')} onChanged={load} /> : (sel && !selStack ? null : <div className="svc-empty muted">Select a container.</div>))}
       {making && <StackNew key={'new:' + (dup?.name ?? making)} mode={making} from={dup ?? undefined} networks={nets} taken={taken} onClose={() => { setMaking(''); setDup(null) }} onCreated={async (name, deploy) => { setMaking(''); setDup(null); setTab('stacks'); if (deploy) { try { setStartTask({ name, id: (await api.stackAction(name, 'deploy')).id }) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') } } else setStartTask(null); await load(); setSel('stack:' + name) }} />}
+      {shownTab === 'updates' && <Updates upd={upd} onChecked={setUpd} />}
       {shownTab === 'stacks' && (selStack ? <StackDetail key={selStack + (startTask?.name === selStack ? startTask.id : '')} name={selStack} startTask={startTask?.name === selStack ? startTask.id : undefined} onBack={() => setSel('')} onChanged={load} onDuplicate={(f) => { setDup(f); setMaking('stack') }} /> : <div className="svc-empty muted">Select a stack to edit its compose file.</div>)}
     </div>
   )
 }
+
+// A container's image against the registry, as a single mark: nothing when nobody has
+// asked, so an uncoloured list means "not checked" rather than "all current".
+function UpdateDot({ state }: { state?: string }) {
+  if (!state || state === 'current') return null
+  const known = state === 'update'
+  return <span className={'updot' + (known ? ' behind' : ' unknown')} title={known ? 'a newer image has been published' : 'the registry could not say'}>{known ? '↑' : '?'}</span>
+}
+
+// What is out of date here, and the schedules that keep it from happening again.
+function Updates({ upd, onChecked }: { upd: MachineUpdates | null; onChecked: (u: MachineUpdates) => void }) {
+  const onHub = apiPath('/api/x') === '/api/x'
+  const [busy, setBusy] = useState(false)
+  const [jobs, setJobs] = useState<UpdateJobRow[] | null>(null)
+  const [file, setFile] = useState('')
+  const [editing, setEditing] = useState<Partial<UpdateJob> | null>(null)
+  const [stacks, setStacks] = useState<string[]>([])
+  const loadJobs = useCallback(async () => {
+    try { const r = await api.updateJobs(); setJobs(r.jobs); setFile(r.file) } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err'); setJobs([]) }
+  }, [])
+  useEffect(() => { loadJobs(); api.stacks().then((r) => setStacks(r.stacks.map((s) => s.name))).catch(() => {}) }, [loadJobs])
+  const check = async () => {
+    setBusy(true)
+    try { onChecked(await api.machineUpdates(true)); toast('checked') } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') } finally { setBusy(false) }
+  }
+  const run = async (j: UpdateJob) => {
+    if (!await askYes(`Run “${j.name}” now? It pulls ${j.stack}'s images${j.apply ? ' and deploys them, which restarts its containers' : ', without deploying them'}.`, 'Run now')) return
+    try { await api.runUpdateJob(j.id); toast(`${j.name}: done`); loadJobs() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
+  }
+  const remove = async (j: UpdateJob) => {
+    if (!await askYes(`Remove the schedule “${j.name}”? Nothing is updated or rolled back — the schedule simply stops.`, 'Remove')) return
+    try { await api.deleteUpdateJob(j.id); toast('removed'); loadJobs() } catch (e) { toast(e instanceof Error ? e.message : 'failed', 'err') }
+  }
+  const imgs = upd?.images ?? []
+  const behind = imgs.filter((i) => i.state === 'update')
+  return (
+    <div className="svc-detail">
+      <div className="dash-bar"><h2 className="section">What is out of date</h2><span className="spacer" />
+        <span className="muted small">{upd?.checked ? `checked ${new Date(upd.checked).toLocaleString()}` : 'not checked yet'}</span>
+        <Btn className="btn" busy={busy} onClick={check}>Check now</Btn>
+      </div>
+      <div className="muted small">Each check is one request per image, which registries count — so it happens when asked, and the answer is kept for six hours.</div>
+      {imgs.length === 0 && <div className="notice" style={{ marginTop: 8 }}>Nothing checked yet on this machine.</div>}
+      {imgs.length > 0 && (
+        <section className="card" style={{ marginTop: 8 }}>
+          {behind.length === 0 && <div className="muted">Every image that could be checked is current.</div>}
+          {imgs.filter((i) => i.state !== 'current').map((i) => (
+            <div className="row" key={i.image}>
+              <span className="k mono small">{i.image}</span>
+              <span className={i.state === 'update' ? 'warn-text' : 'muted small'}>{i.state === 'update' ? 'a newer image is published' : i.detail || 'unknown'}</span>
+              <span className="muted small">{(upd?.stacks?.[i.image] ?? []).join(' ')}</span>
+            </div>
+          ))}
+          {behind.length === 0 && imgs.some((i) => i.state === 'current') && <div className="muted small" style={{ marginTop: 6 }}>{imgs.filter((i) => i.state === 'current').length} current.</div>}
+        </section>
+      )}
+
+      {onHub && <Fleet />}
+
+      <div className="dash-bar" style={{ marginTop: 14 }}><h2 className="section">Schedules</h2><span className="spacer" />
+        <Btn className="btn on" onClick={() => setEditing({ cron: '0 3 * * *', apply: true, enabled: true })}>+ New schedule</Btn>
+      </div>
+      <div className="muted small">Written to <span className="mono">{file || '/etc/cron.d/hamlab-docker'}</span>, which Ham Lab owns — so they keep running whether or not this console is up.</div>
+      {jobs === null && <Loading label="Reading…" />}
+      {jobs?.length === 0 && <div className="notice" style={{ marginTop: 8 }}>No schedule yet. Nothing on this machine updates itself.</div>}
+      <div className="cards" style={{ marginTop: 8 }}>
+        {(jobs ?? []).map(({ job, next, last }) => (
+          <section className="card" key={job.id}>
+            <div className="sub-head">
+              <span className={'dot ' + (last?.error ? 'bad' : job.enabled ? 'ok' : '')} />
+              <span className="dname">{job.name}</span>
+              {!job.enabled && <span className="badge">paused</span>}
+              <span className="spacer" />
+              <span className="muted small">{job.enabled && next ? new Date(next).toLocaleString() : 'not scheduled'}</span>
+            </div>
+            <div className="props" style={{ marginTop: 6 }}>
+              <div><span className="k">Stack</span><span>{job.stack}{job.services?.length ? ` · ${job.services.join(' ')}` : ''}</span></div>
+              <div><span className="k">Cron</span><span className="mono small">{job.cron}</span></div>
+              <div><span className="k">Does</span><span>{job.apply ? 'pulls and deploys' : 'pulls only — you deploy'}</span></div>
+              <div><span className="k">Last run</span><span>{last?.at ? new Date(last.at).toLocaleString() : 'never'}</span></div>
+            </div>
+            {last?.error && <div className="error" style={{ marginTop: 6 }}>{last.error}</div>}
+            <div className="svc-actions">
+              <Btn className="btn on" onClick={() => run(job)}>Run now</Btn>
+              <button className="btn" onClick={() => setEditing(job)}>Edit</button>
+              <Btn className="btn" onClick={async () => { await api.saveUpdateJob({ ...job, enabled: !job.enabled }); loadJobs() }}>{job.enabled ? 'Pause' : 'Resume'}</Btn>
+              <Btn className="btn danger" onClick={() => remove(job)}>Remove</Btn>
+            </div>
+          </section>
+        ))}
+      </div>
+      {editing && <JobDialog j={editing} stacks={stacks} onClose={() => setEditing(null)} onSaved={() => { setEditing(null); loadJobs() }} />}
+    </div>
+  )
+}
+
+// Every other machine at once. Only shown on the hub, because only the hub can reach them,
+// and each answer is that machine's own six-hour cache — nothing is checked here without
+// being asked for.
+function Fleet() {
+  const [rows, setRows] = useState<{ name: string; online: boolean; upd?: MachineUpdates; err?: string }[] | null>(null)
+  const [busy, setBusy] = useState('')
+  const load = useCallback(async (refresh?: string) => {
+    let nodes: MachineStatus[] = []
+    try { nodes = (await api.machines()).nodes } catch { setRows([]); return }
+    const out = await Promise.all(nodes.map(async (n) => {
+      if (!n.online) return { name: n.name, online: false }
+      try { return { name: n.name, online: true, upd: await api.nodeUpdates(n.name, refresh === n.name) } }
+      catch (e) { return { name: n.name, online: true, err: e instanceof Error ? e.message : 'failed' } }
+    }))
+    setRows(out)
+  }, [])
+  useEffect(() => { load() }, [load])
+  if (rows === null) return <div style={{ marginTop: 14 }}><Loading label="Asking the other machines…" /></div>
+  if (rows.length === 0) return null
+  return (
+    <>
+      <h2 className="section" style={{ marginTop: 14 }}>The other machines</h2>
+      <section className="card">
+        {rows.map((r) => {
+          const behind = (r.upd?.images ?? []).filter((i) => i.state === 'update')
+          return (
+            <div className="row" key={r.name}>
+              <span className="k">{r.name}</span>
+              <span>
+                {!r.online ? <span className="muted">offline</span>
+                  : r.err ? <span className="muted small">{r.err}</span>
+                  : !r.upd?.checked ? <span className="muted">not checked</span>
+                  : behind.length === 0 ? <span className="muted">all current</span>
+                  : <span className="warn-text" title={behind.map((i) => i.image).join('\n')}>{behind.length} behind</span>}
+              </span>
+              <span className="row-end">{r.online && !r.err && <Btn className="link" busy={busy === r.name} onClick={async () => { setBusy(r.name); try { await load(r.name) } finally { setBusy('') } }}>check</Btn>}</span>
+            </div>
+          )
+        })}
+      </section>
+      <div className="muted small">Open a machine from the switcher to schedule anything on it — a schedule lives on the machine that runs it.</div>
+    </>
+  )
+}
+
+function JobDialog({ j, stacks, onClose, onSaved }: { j: Partial<UpdateJob>; stacks: string[]; onClose: () => void; onSaved: () => void }) {
+  const [f, setF] = useState<Partial<UpdateJob>>(j)
+  const [svc, setSvc] = useState((j.services ?? []).join(' '))
+  const [err, setErr] = useState('')
+  const [busy, setBusy] = useState(false)
+  useEscape(onClose)
+  const set = (k: keyof UpdateJob, v: unknown) => setF((x) => ({ ...x, [k]: v }))
+  const save = async (e: FormEvent) => {
+    e.preventDefault(); setErr(''); setBusy(true)
+    try { await api.saveUpdateJob({ ...f, services: svc.split(/\s+/).filter(Boolean) }); onSaved() }
+    catch (e) { setErr(e instanceof Error ? e.message : 'failed'); setBusy(false) }
+  }
+  return (
+    <div className="modal-back" onClick={onClose}>
+      <form className="modal" onClick={(e) => e.stopPropagation()} onSubmit={save}>
+        <h3>{f.id ? 'Edit schedule' : 'Keep a stack up to date'}</h3>
+        <label className="muted">Stack
+          <select value={f.stack ?? ''} onChange={(e) => set('stack', e.target.value)} required>
+            <option value="">choose…</option>
+            {stacks.map((s) => <option key={s} value={s}>{s}</option>)}
+          </select>
+        </label>
+        <label className="muted">Name — optional
+          <input value={f.name ?? ''} onChange={(e) => set('name', e.target.value)} placeholder={f.stack ?? 'nightly'} maxLength={64} />
+        </label>
+        <label className="muted">Only these services — optional, space separated; blank means the whole stack
+          <input className="mono" value={svc} onChange={(e) => setSvc(e.target.value)} placeholder="all of them" spellCheck={false} />
+        </label>
+        <label className="muted">When — a cron expression
+          <input className="mono" value={f.cron ?? ''} onChange={(e) => set('cron', e.target.value)} placeholder="0 3 * * *" spellCheck={false} required />
+        </label>
+        <label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
+          <input type="checkbox" checked={!!f.apply} onChange={(e) => set('apply', e.target.checked)} style={{ width: 'auto' }} />
+          Deploy the new image, not just fetch it
+        </label>
+        <label className="muted" style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
+          <input type="checkbox" checked={!!f.enabled} onChange={(e) => set('enabled', e.target.checked)} style={{ width: 'auto' }} />
+          Enabled
+        </label>
+        <div className="notice">
+          {f.apply
+            ? 'Deploying restarts the stack’s containers. On something that carries DNS or ingress, pick an hour you are asleep.'
+            : 'Pulling only means the new image is on the machine and the list shows it as available — nothing restarts until you deploy.'}
+        </div>
+        {err && <div className="error">{err}</div>}
+        <div className="svc-actions">
+          <button type="submit" className="btn on" disabled={busy || !f.stack || !f.cron}>Save</button>
+          <span className="spacer" />
+          <button type="button" className="btn" onClick={onClose}>Cancel</button>
+        </div>
+      </form>
+    </div>
+  )
+}

web/src/KVM.tsx

@@ -22,11 +22,35 @@ import { Btn, Loading, askYes, toast, useEscape } from './ui'
 type KeyboardLock = { lock?: (keys?: string[]) => Promise<void>; unlock?: () => void }
 const keyboardAPI = (): KeyboardLock | undefined => (navigator as unknown as { keyboard?: KeyboardLock }).keyboard
 
+// Fullscreen, across the three ways browsers spell it. Safari on a Mac wants the webkit
+// prefix; Safari on iOS has no fullscreen for an ordinary element at all — only for a
+// <video> — so there the page expands inside the viewport instead, which is the same thing
+// minus the browser's own chrome.
+type FSElement = HTMLElement & { webkitRequestFullscreen?: () => Promise<void> | void }
+type FSDocument = Document & { webkitFullscreenElement?: Element | null; webkitExitFullscreen?: () => Promise<void> | void }
+const nativeFullscreen = () => {
+  if (typeof document === 'undefined') return false
+  const el = document.documentElement as FSElement
+  return typeof el.requestFullscreen === 'function' || typeof el.webkitRequestFullscreen === 'function'
+}
+const fullscreenElement = () => (document as FSDocument).fullscreenElement ?? (document as FSDocument).webkitFullscreenElement ?? null
+async function enterFullscreen(el: FSElement) {
+  if (typeof el.requestFullscreen === 'function') return el.requestFullscreen()
+  if (typeof el.webkitRequestFullscreen === 'function') return el.webkitRequestFullscreen()
+  throw new Error('this browser has no fullscreen for a page element')
+}
+function exitFullscreen() {
+  const d = document as FSDocument
+  if (typeof d.exitFullscreen === 'function') return d.exitFullscreen()
+  if (typeof d.webkitExitFullscreen === 'function') return d.webkitExitFullscreen()
+}
+
 export default function KVM({ machine, title }: { machine: string | null; title: string }) {
   const [kvm, setKvm] = useState<Device | null | undefined>(undefined)
   const [adding, setAdding] = useState(false)
   const [err, setErr] = useState('')
-  const [full, setFull] = useState(false)
+  const [full, setFull] = useState(false)   // native fullscreen is on
+  const [expanded, setExpanded] = useState(false) // the fallback: filling the viewport in-page
   const box = useRef<HTMLDivElement>(null)
   const frame = useRef<HTMLIFrameElement>(null)
   const key = machine ?? '' // the hub is a machine too, and its key is the empty one
@@ -41,21 +65,33 @@ export default function KVM({ machine, title }: { machine: string | null; title:
   // than the browser. Both are undone together when fullscreen ends, however it ends.
   useEffect(() => {
     const onChange = () => {
-      const on = !!document.fullscreenElement
+      const on = !!fullscreenElement()
       setFull(on)
       if (!on) keyboardAPI()?.unlock?.()
     }
     document.addEventListener('fullscreenchange', onChange)
-    return () => { document.removeEventListener('fullscreenchange', onChange); keyboardAPI()?.unlock?.() }
+    document.addEventListener('webkitfullscreenchange', onChange)
+    return () => {
+      document.removeEventListener('fullscreenchange', onChange)
+      document.removeEventListener('webkitfullscreenchange', onChange)
+      keyboardAPI()?.unlock?.()
+    }
   }, [])
+  // Expanding is in-page, so Escape is ours to handle.
+  useEscape(() => setExpanded(false), expanded)
 
-  const goFullscreen = async () => {
+  const goBig = async () => {
+    if (!nativeFullscreen()) { setExpanded(true); frame.current?.focus(); return }
     try {
-      await box.current?.requestFullscreen()
+      if (box.current) await enterFullscreen(box.current)
       await keyboardAPI()?.lock?.()
       frame.current?.focus()
-    } catch (e) { toast(e instanceof Error ? e.message : 'fullscreen refused', 'err') }
+    } catch {
+      setExpanded(true) // refused (an iframe policy, a permissions policy): fill the page instead
+      frame.current?.focus()
     }
+  }
+  const goSmall = () => { setExpanded(false); if (fullscreenElement()) exitFullscreen() }
 
   const remove = async () => {
     if (!kvm || !await askYes(`Forget ${title}'s KVM? The device itself is not touched — this only removes its address.`, 'Remove')) return
@@ -66,6 +102,7 @@ export default function KVM({ machine, title }: { machine: string | null; title:
   if (kvm === undefined) return <div className="dash"><Loading label="Looking…" /></div>
 
   const locks = !!keyboardAPI()?.lock
+  const big = full || expanded
 
   return (
     <div className="kvm">
@@ -76,7 +113,7 @@ export default function KVM({ machine, title }: { machine: string | null; title:
           : <span className="muted small">no KVM</span>}
         <span className="spacer" />
         {kvm && <>
-          <Btn className="btn" onClick={goFullscreen} title={locks ? 'fullscreen, and the browser stops keeping Escape, Tab and Ctrl-W for itself' : 'fullscreen — this browser will still keep some keys for itself'}>Fullscreen</Btn>
+          <Btn className="btn" onClick={goBig} title={!nativeFullscreen() ? 'fill the screen — this browser has no true fullscreen for a page element' : locks ? 'fullscreen, and the browser stops keeping Escape, Tab and Ctrl-W for itself' : 'fullscreen — this browser will still keep some keys for itself'}>{nativeFullscreen() ? 'Fullscreen' : 'Expand'}</Btn>
           <a className="btn" href={kvm.url} target="_blank" rel="noreferrer" title="the device directly, outside Ham Lab — only works on the LAN">Open ↗</a>
           <Btn className="btn" onClick={() => setAdding(true)}>Change</Btn>
           <Btn className="btn danger" onClick={remove}>Remove</Btn>
@@ -85,9 +122,10 @@ export default function KVM({ machine, title }: { machine: string | null; title:
       </div>
 
       {kvm ? (
-        <div className={'kvm-frame' + (full ? ' full' : '')} ref={box}>
+        <div className={'kvm-frame' + (full ? ' full' : '') + (expanded ? ' expanded' : '')} ref={box}>
           <iframe ref={frame} key={kvm.id} src={`/kvm/${kvm.id}/`} title={kvm.name}
             allow="fullscreen; clipboard-read; clipboard-write" />
+          {expanded && <button className="btn kvm-shrink" onClick={goSmall} title="back to the console">Done</button>}
         </div>
       ) : (
         <div className="svc-empty muted kvm-none">
@@ -98,7 +136,7 @@ export default function KVM({ machine, title }: { machine: string | null; title:
           </div>
         </div>
       )}
-      {kvm && !full && (
+      {kvm && !big && (
         <div className="kvm-hint muted small">
           Click the picture, then type — the keys go to {title}.
           {locks ? ' Fullscreen also takes Escape, Tab and Ctrl-W back from the browser.' : ' This browser keeps Escape, Tab and Ctrl-W for itself; use the device’s own on-screen keys for those, or a Chromium browser.'}

web/src/Shell.tsx

@@ -55,18 +55,18 @@ const GROUPS: Group[] = [
   { id: 'procs', label: 'Processes', icon: '▚' },
   { id: 'terminal', label: 'Terminal', icon: '>_' },
   { id: 'services', label: 'Services', icon: '◈' },
-  { id: 'docker', label: 'Docker', icon: DockerIcon },
   { id: 'jobs', label: 'Jobs', icon: ClockIcon },
-  { id: 'updates', label: 'Updates', icon: '⇪' },
   { id: 'disks', label: 'Disks', icon: '▤' },
-  { id: 'zfs', label: 'ZFS', icon: '⛁' },
-  { id: 'usage', label: 'Usage', icon: '▧' },
   { id: 'files', label: 'Files', icon: FolderIcon },
-  { id: 'alerts', label: 'Alerts', icon: BellIcon },
+  { id: 'usage', label: 'Usage', icon: '▧' },
+  { id: 'zfs', label: 'ZFS', icon: '⛁' },
+  { id: 'docker', label: 'Docker', icon: DockerIcon },
   { id: 'kvm', label: 'KVM', icon: '⌨' },
+  { id: 'updates', label: 'Updates', icon: '⇪' },
+  { id: 'alerts', label: 'Alerts', icon: BellIcon },
   { id: 'machines', label: 'Machines', icon: '⌂' },
-  { id: 'users', label: 'Users', icon: '⚇' },
   { id: 'widgets', label: 'Widgets', icon: '▦' },
+  { id: 'users', label: 'Users', icon: '⚇' },
   { id: 'settings', label: 'Settings', icon: '⚙' },
 ]
 
@@ -125,6 +125,20 @@ export default function Shell({ me: hubMe, onLoggedOut }: { me: Me; onLoggedOut:
 
   useEffect(() => { reload() }, [reload])
 
+  // A shell opened inside a container from the Docker tab is a terminal like any other:
+  // pick it up, show the Terminal tab, and make it the active one.
+  useEffect(() => {
+    const h = async (e: Event) => {
+      const id = (e as CustomEvent<string>).detail
+      if (!id) return
+      await reload()
+      setActive(id)
+      setView('terminal')
+    }
+    window.addEventListener('hamlab:terminal', h)
+    return () => window.removeEventListener('hamlab:terminal', h)
+  }, [reload])
+
   // Machine list for the switcher (hub-local), refreshed while the console is open.
   const loadMachines = useCallback(async () => { try { setMachines((await api.machines()).nodes) } catch { /* not a hub or offline */ } }, [])
   useEffect(() => { loadMachines(); const t = setInterval(loadMachines, listRate); return () => clearInterval(t) }, [loadMachines, listRate])

web/src/api.ts

@@ -109,6 +109,14 @@ export interface StorageDevice { name: string; kind: string; model?: string; ven
 export interface SmartAttr { id: number; name: string; value: number; worst: number; thresh: number; raw: string; failed?: string; warn: boolean }
 export interface SmartInfo { device: string; available: boolean; supported: boolean; standby: boolean; passed: boolean | null; temp: number; power_on_hours: number; power_cycles: number; model?: string; serial?: string; firmware?: string; attrs: SmartAttr[]; nvme?: { critical_warning: number; percentage_used: number; available_spare: number; spare_threshold: number; media_errors: number; error_log_entries: number; data_read_bytes: number; data_written_bytes: number; unsafe_shutdowns: number }; mmc?: { life_a: number; life_b: number; pre_eol: number }; self_test?: string; self_test_remaining?: number; self_test_supported: boolean; self_test_note?: string; warnings: string[]; error?: string; fetched_at: number }
 export interface DockerContainer { id: string; name: string; image: string; state: string; status: string; health: string; ports: string; stack: string; service: string; created: string; mem: number }
+export interface UpdateJob { id: string; name: string; stack: string; machine?: string; services?: string[]; cron: string; apply: boolean; enabled: boolean }
+export interface UpdateJobRow { job: UpdateJob; next?: string; last?: { at?: string; taken?: string; error?: string } }
+export interface MachineUpdates { images: ImageCheck[]; checked: number; stacks: Record<string, string[]> }
+
+export interface DockerHealthProbe { Start: string; End: string; ExitCode: number; Output: string }
+export interface DockerHealth { configured: boolean; state: string; status?: string; failing_streak?: number; log?: DockerHealthProbe[] }
+export interface DockerEvent { t: number; name: string; action: string; image?: string; exit_code?: string; signal?: string; health?: string }
+
 export interface DockerInfo { available: boolean; message?: string; tool?: string; containers: DockerContainer[]; images: Record<string, unknown>[]; volumes: Record<string, unknown>[]; networks: Record<string, unknown>[] }
 export interface Stack { name: string; containers: string[]; running: number; file: string; host_file: string; files: string[]; host_files: string[] | null; images: string[] | null; via: string; remote: string; held_by: string; editable: boolean; note?: string; has_env: boolean }
 export interface StackHome { dir: string; inner: string; via: string; remote: string; held_by: string; stacks: number; default: boolean }
@@ -248,6 +256,17 @@ export const api = {
   dockerAction: (id: string, action: 'start' | 'stop' | 'restart' | 'remove') => req<void>('POST', `/api/docker/containers/${encodeURIComponent(id)}/${action}`),
   dockerLogs: (id: string, since = '') => req<{ lines: { t: string; m: string }[] }>('GET', `/api/docker/containers/${encodeURIComponent(id)}/logs${since ? `?since=${encodeURIComponent(since)}` : '?n=300'}`),
   dockerInspect: (id: string) => req<Record<string, unknown>>('GET', `/api/docker/containers/${encodeURIComponent(id)}/inspect`),
+  machineUpdates: (refresh = false) => req<MachineUpdates>('GET', `/api/docker/updates${refresh ? '?refresh=1' : ''}`),
+  // One named machine, asked of the hub — the fleet view, which is on the hub by definition.
+  nodeUpdates: (node: string, refresh = false) => req<MachineUpdates>('GET', `/api/nodes/${encodeURIComponent(node)}/docker/updates${refresh ? '?refresh=1' : ''}`, undefined, true),
+  updateJobs: () => req<{ jobs: UpdateJobRow[]; file: string }>('GET', '/api/docker/schedules'),
+  saveUpdateJob: (j: Partial<UpdateJob>) => req<UpdateJob>('PUT', '/api/docker/schedules', j),
+  deleteUpdateJob: (id: string) => req<void>('DELETE', `/api/docker/schedules/${encodeURIComponent(id)}`),
+  runUpdateJob: (id: string) => req<{ output: string }>('POST', `/api/docker/schedules/${encodeURIComponent(id)}/run`),
+  dockerHealth: (id: string) => req<DockerHealth>('GET', `/api/docker/containers/${encodeURIComponent(id)}/health`),
+  dockerEvents: (id: string, since = '7d') => req<{ events: DockerEvent[]; since: string; kept: number; capacity: number }>('GET', `/api/docker/containers/${encodeURIComponent(id)}/events?since=${since}`),
+  dockerExec: (id: string) => req<{ id: string; name: string }>('POST', `/api/docker/containers/${encodeURIComponent(id)}/exec`, {}),
+  dockerLimits: (id: string, p: { memory?: string; cpus?: string; restart?: string }) => req<void>('POST', `/api/docker/containers/${encodeURIComponent(id)}/limits`, p),
   stacks: () => req<{ stacks: Stack[] }>('GET', '/api/docker/stacks'),
   stackHomes: () => req<{ homes: StackHome[] }>('GET', '/api/docker/stacks/homes'),
   createStack: (p: { name: string; text: string; env: string; home: string; env_from?: string }) => req<{ stack: Stack }>('POST', '/api/docker/stacks', p),

web/src/styles.css

@@ -751,3 +751,11 @@ pre.diff .add { color: var(--ok); background: color-mix(in srgb, var(--ok) 12%,
 .kvm-hint { padding: 6px 12px; border-top: 1px solid var(--border); }
 @media (max-width: 800px) { .kvm-bar .kvm-url { display: none; } }
 .kvm-none { text-align: center; } .kvm-none p { margin: 4px 0; }
+/* No true fullscreen for a page element (Safari on iOS): fill the viewport in-page instead. */
+.kvm-frame.expanded { position: fixed; inset: 0; z-index: 60; }
+.kvm-frame.expanded iframe { height: 100%; }
+.kvm-shrink { position: absolute; right: max(10px, env(safe-area-inset-right)); bottom: max(10px, env(safe-area-inset-bottom)); z-index: 61; background: var(--bg2); border: 1px solid var(--border); opacity: .75; }
+.kvm-shrink:hover, .kvm-shrink:active { opacity: 1; }
+.updot { margin-left: 6px; font-size: 11px; font-weight: 700; }
+.updot.behind { color: var(--s2); } .updot.unknown { color: var(--muted); }
+.card > .row > .row-end { text-align: right; white-space: nowrap; }